Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/17297
>> I don't think its true that it relaunches all tasks that hadn't
completed when the fetch failure occurred. it relaunches all the tasks haven't
completed, by the time the stage gets resubmitted. More tasks can complete in
between the time of the first failure, and the time the stage is resubmitted.
Actually, I realized that it's not true. If you looked at the code
(https://github.com/sitalkedia/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1419),
when the stage fails because of fetch failure, we remove the stage from the
output commiter. So if any task completes between the time of first fetch
failure and the time stage is resubmitted, will be denied to commit the output
and so the scheduler re-launches all tasks in the stage with the fetch failure
that hadn't completed when the fetch failure occurred.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
