Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/17297
cc - @kayousterhout - Addressed your earlier comment about
https://github.com/apache/spark/pull/12436 ignoring fetch failure from stale
map output. I have addressed this issue by adding epoch for each map output
registered, that way if the task's epoch is smaller than the epoch of the map
output, we can ignore the fetch failure. This also takes care of epoch changes
which will be triggered due to executor loss for a shuffle task when its
shuffle map task executor is gone as pointed out by @mridulm.
Let me know what you think of the approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
