Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/17088
@tgravescs - I agree this might cause additional work in situations where
shuffle fetch is transient like you mentioned above. But in those cases, IMO,
users should tune the shuffle retry configurations to make sure we do not see
any fetch failure. Current model assumes file lost in case of fetch failure and
in case external shuffle service is enabled, we should clean all the files on a
host as opposed to cleaning only for a specific executor.
@kayousterhout - Filed a JIRA for the flaky build - SPARK-19803
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
