linehrr commented on issue #24461: [SPARK-27434][CORE] Fix mem leak due to hadoop fs caching mechanism when eventLog is enabled URL: https://github.com/apache/spark/pull/24461#issuecomment-488024150 @vanzin enable eventLog itself along does not cause problem. like long running streaming jobs, they are fine. problem only shows up when spark context gets open and closed repeatedly. which likely to create file system object every time instead of reusing the one that's cached previously. the point where I don't understand is how this is happening, I checked the `Key` in the cache map, it should be the same compared to the previous one, and the `equals` method got overridden properly as well, `hashcode` method override is ok as well. therefore I can't explain why cache is not reused for fs object, only thing I know is it's not reused.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
