Victsm commented on a change in pull request #25907: [SPARK-29206][SHUFFLE]
Make number of shuffle server threads a multiple of number of chunk fetch
handler threads.
URL: https://github.com/apache/spark/pull/25907#discussion_r328297534
##########
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -111,8 +111,30 @@ public int numConnectionsPerPeer() {
/** Requested maximum length of the queue of incoming connections. Default
is 64. */
public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, 64);
}
- /** Number of threads used in the server thread pool. Default to 0, which is
2x#cores. */
- public int serverThreads() { return
conf.getInt(SPARK_NETWORK_IO_SERVERTHREADS_KEY, 0); }
+ /**
+ * The configured ratio between number of server threads and number of chunk
fetch handler
Review comment:
The thing this patches attempts to do is to guarantee the # server threads
is a multiple of the # chunk fetch handler threads. Using the percent approach,
it's tricky to configure things correctly, as discussed in the jira ticket.
The ratio parameter could probably use a better name. However, to make the
chunk fetch handler EventLoopGroup truly eliminate the RPC message timeout
issue, the condition of # server_threads = # chunk_fetch_handler_threads *
int_ratio needs to be satisfied.
This would mean that we cannot directly use the user configured # server
threads if it cannot be divided by the int_ratio. That's why the change to
serverThreads() is introduced below.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
