xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398340491
##########
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
In this PR, we make the separate event loop group configurable by checking
the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
> What do you mean by the config must be set
Here the function `chunkFetchHandlerThreads` is only called while the config
is set.
https://github.com/apache/spark/blob/0fe203e7032a326ff0c78f6660ab1b09409fe09d/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java#L124-L129
> What value do you expect by default? Apparently, this seems to revert
SPARK-25641 together without mentioning SPARK-25641.
Yes, this PR makes the feature disabled by default, let me also mention
SPARK-25641 in PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
