Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22112
Update, according to the discussion in
https://github.com/apache/spark/pull/9214 , the current behavior of shuffle
writing is: "first write wins". We can't simply change it to "last write wins",
as we may have concurrent read-write issues. To switch to "last write wins", we
have to finish https://github.com/apache/spark/pull/6648 first.
Since it's not realistic to complete
https://github.com/apache/spark/pull/6648 before Spark 2.4, in this PR I fail
the job directly if we hit a fetch failure and the preceding map stage is
indeterminate. The error message asks users to do checkpoint to avoid this
issue.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
