tgravescs edited a comment on pull request #31942: URL: https://github.com/apache/spark/pull/31942#issuecomment-810267053
it would be nice to have more description in this pr and jira. When is this possible, what are the ramifications of this, was this actually hit in real application, etc. > And in the case of ChunkFetchSuccess, I wonder we may release the buffer here too early since the listener.onSuccess(...) is executed asynchronously: That shouldn't happen as the caller is supposed to retain or copy it like you referenced, but yeah in the shuffle fetcher at first glance it doesn't look like its cleaned up in the good case after calling retain on it. I'll have to look a bit deeper. It would really be nice to have a test for this, even if its manual once doing heap dumps or changing code to see if we have some actual leaks or if there is a path we are missing here. This comes back to my original request of having more information in the jira. Also in the very least I would like to see a fairly long lived real application ran other then unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
