Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4021#issuecomment-75495372
I just realized that ContextCleaner won't manage cleanup of accumulators in
the thread-local accumulator registries in executors, but I don't think this is
a big deal: we're more concerned about the driver OOMing in a long-running app
than the executors, and it would be prohibitively expensive to add a bunch of
driver -> executor RPCs just to reclaim one map entry.
Therefore, this looks good to me. There's one minor import-ordering nit,
but I'll just fix that up myself on merge. Thanks for working on this, esp.
since this code is a bit tricky to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
