[GitHub] [spark] srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak due to hadoop fs caching mechanism when eventLog is enabled

srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak due to hadoop fs caching mechanism when eventLog is enabled URL: https://github.com/apache/spark/pull/24461#issuecomment-488030569 Hm, I wonder if that's more at the root of the problem: you'd generally not close the

[GitHub] [spark] srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak

srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak URL: https://github.com/apache/spark/pull/24461#issuecomment-487657526 Let's see what tests say so far. BTW you'll need to improve the title of the PR and JIRA.

[GitHub] [spark] srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak

srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak URL: https://github.com/apache/spark/pull/24461#issuecomment-487422243 Hm your heap dump is convincing, although this `FileSystem` is just obtained from Hadoop and its cache. Generally that's not safe to close, but maybe

[GitHub] [spark] srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak

srowen commented on issue #24461: [SPARK-27434][CORE] Fix mem leak URL: https://github.com/apache/spark/pull/24461#issuecomment-487390580 I don't think you can close `FileSystem` objects. They're shared. Why is it a memory leak?