beliefer commented on issue #23355: [SPARK-26418][SHUFFLE] Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService URL: https://github.com/apache/spark/pull/23355#issuecomment-448960858 If ExternalShuffleService only received a OpenBlocks message, not received any ChunkFetchRequest message, associate StreamState is hold by streams. You thought this method that build relationships between Channel and StreamState is valid. But when a Channel is terminated of failed,TransportClientFactory will create another one, please see the method TransportClientFactory.createClient. So I propose to add a parameter to control some timeout of Channel that sended OpenBlocks.When the time expired,call the method connectionTerminated.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
