Deegue opened a new pull request #25057: [SPARK-28239][SHUFFLE] Allow TCP connections created by shuffle service auto close on YARN NodeManagers URL: https://github.com/apache/spark/pull/25057 ## What changes were proposed in this pull request? Make it possible to close TCP connections on YARN NodeManager automatically. It will fix connections piling up on NodeManagers and NodeManagers getting slow. ## How was this patch tested? Manually tested. Build Spark and deploy as [Configuring the External Shuffle Service](http://spark.apache.org/docs/latest/running-on-yarn.html#configuring-the-external-shuffle-service) Set spark.shuffle.io.enableTcpAutoClose to true. Then execute command `lsof -i:7337 |wc -l` on NodeManagers. The results will like :  It keeps in a range and won't increase continuously. Environment: Hadoop2.6.0-CDH5.8.3(netty3) Spark2.4.0(netty4) Configs: spark.dynamicAllocation.enabled=true spark.shuffle.service.enabled=true
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
