Ngone51 commented on pull request #30139: URL: https://github.com/apache/spark/pull/30139#issuecomment-722461726
> IMO, When the data shuffle partition is large, I don't think the channel size will be small https://github.com/apache/spark/blob/fa5dc0a45a414c34b31c5d7efe396aa04f1e66e3/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java#L106-L109 The default concurrent connections between the peer are 1. So the total connection number of a single server should depend on the number of executors, right? If there are more than 5k or 10k executors, the problem does still exists. @wangyum May I ask what's the scale of the cluster in your case? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
