wangshuo128 commented on issue #23355: [SPARK-26418][SHUFFLE] Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService URL: https://github.com/apache/spark/pull/23355#issuecomment-454310459 @Ngone51 Thanks a lot for comment and review! > A minor concern is: > Since we always register the channel when we receive the `FetchChunkRequest` in the past, so the associated channel with the stream will aways be correct. And for this change, if user call `sendRpc(OpenBlocks)` and `fetchChunk` with different client, then, we may record the wrong channel(which is the `OpenBlocks` chanel, rather than `fetchChunk` channel) with the stream which is being fetched, which may fall into the same issue as this pr fixed. So, I think we should leave some comment above `fetchChunk` to notify user that this method _must_ use the same client with the `OpenBlocks`, otherwise, there may be potential memory leak. I had the same concern too. I added some comment at TransportClient.fetchChunk .
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
