Github user jasonmoore2k commented on the pull request:
https://github.com/apache/spark/pull/12228#issuecomment-214628237
@andrewor14 @srowen
I'm starting to think this change wasn't great to make. I'm now seeing
speculated tasks sometimes getting into a cycle of running, being
(legitimately) denied from committing their tasks, retrying, being denied,
retrying, etc. And with the change I made in this PR, this retrying now has no
limit. Which is really bad (jobs potentially running forever). Whereas
previously they would eventually fail (still not great).
I think the ideal behavior would be to have the speculated task instead be
considered a success if the commit is denied legitimately (because another task
for the same partition has already completed), rather than considered a failure
and retried without limit (to only fail again). Any thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
