GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/4835
[SPARK-6075] Fix bug in that caused lost accumulator updates: do not store
WeakReferences in localAccums map
This fixes a non-deterministic bug introduced in #4021 that could cause
tasks' accumulator updates to be lost. The problem is that `localAccums`
should not hold weak references: after the task finishes running there won't be
any strong references to these local accumulators, so they can get
garbage-collected before the executor reads the `localAccums` map. We don't
need weak references here anyways, since this map is cleared at the end of each
task.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-6075
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4835.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4835
----
commit 120c7b085008e708e5586169eb33fb01b3a733d1
Author: Josh Rosen <[email protected]>
Date: 2015-02-28T20:19:34Z
[SPARK-6075] Do not store WeakReferences in localAccums map
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
