Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1387#issuecomment-49901334
@pwendell I don't think that's the goal here; the executor is already dead,
so this is not targeted at freeing up executor memory through a very indirect
way. This is targeted at cleanup: looking at the posted logs (also see
SPARK-2491, although that's just the logs that are also posted here), there are
exceptions that would be caused by code holding files open when the VM shuts
down.
I stand by my earlier comment that relying on the gc for this is hacky: the
code that is making these references unreachable should be cleaning them up.
BTW, this also means that, in my opinion, ContextCleaner.scala is similarly
hacky, and should be replaced with code that does the right thing without
relying on the gc. (Also, because ContextCleaner can be disabled through a conf
option...) Regardless of whether one JVM implementation consistently triggers
the gc when you call `System.gc()`, others might not, and that still doesn't
mean everything that we'd like to be collected actually is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
