tgravescs commented on a change in pull request #25907: [SPARK-29206][SHUFFLE]
Make number of shuffle server threads a multiple of number of chunk fetch
handler threads.
URL: https://github.com/apache/spark/pull/25907#discussion_r328347987
##########
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -111,8 +111,30 @@ public int numConnectionsPerPeer() {
/** Requested maximum length of the queue of incoming connections. Default
is 64. */
public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, 64);
}
- /** Number of threads used in the server thread pool. Default to 0, which is
2x#cores. */
- public int serverThreads() { return
conf.getInt(SPARK_NETWORK_IO_SERVERTHREADS_KEY, 0); }
+ /**
+ * The configured ratio between number of server threads and number of chunk
fetch handler
Review comment:
I think we need to be more clear on the comment. I realize you have the
jira linked, but I think we should say here that this is the divisor or
multiplier (depending on phrasing) and that we change the number server threads
to make it a multiple based on this config. The user may find it weird that
they explicitly set server threads = 9 and then we change that to 10 based on
this multiple.
So I think the comment needs to be very clear on the results they get.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
