Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/6220#issuecomment-102950055
This is tricky actually. The problem with reference-tracking based
mechanism that we have is not bullet-proof as it depends on GC behavior in the
driver. This is a problem that I have seen in Spark Streaming programs as well.
For driver with large heaps, some RDD that is not in scope may not got
dereferenced for a long time, until a full GC occurs. And in the mean time,
nothing will get cleaned. The solutions to that is to call System.gc() at some
interval, say one hour.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
