Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216068689
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred",
Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process
ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process
ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying
channel to send
+ * response back to client, the I/O on the channel is still being
handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default
EventLoopGroup
+ * that's registered with the Channel, by waiting inside the
ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to
put a limit on
+ * the max number of threads from TransportServer's default
EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which
are I/O intensive
+ * and could take long time to process due to disk contentions. By
configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some
threads for
+ * handling other RPC messages, thus making the Client less likely to
experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which
is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of
io.serverThreads
--- End diff --
No based on how many threads required for other rpc calls, i have not
tested them, but the whole point would be to reduce the dependency how much
time the chunkFetchedRequests will be spending doing disk I/O
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
