Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8090#issuecomment-129983636
@squito ah sorry I missed your comment. Just so I understand your point,
are you saying that when a stage is submitted with all missing partitions, none
of the tasks have completed in the past so there should not be existing
accumulator values right? That sounds possibly reasonable.
However, are there cases where we resubmit the whole stage even if a
fraction of the tasks have completed? What if we lost the shuffle files of
those completed map tasks? Will we resubmit the stage using the same `Stage`
object? The check is there to avoid double counting of accumulator values, and
I'm not 100% sure if it's safe to remove.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
