Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21698
Thanks @cloud-fan your summary above is super useful, and I think it's
clear enough.
> So when we see fetch failure and rerun map tasks, we should track which
reducers have its shuffle blocks being rewritten, and rerun them.
IIUC, patterns like `rdd.map(...).groupBy()` shall always be under risk if
we can generate non-determine output in `map()` right?
> Simply inserting a sort before shuffle doesn't help. The fix for
dataframe is adding a sort before round-robin, to make it deterministic. If we
add the sort after round-robin and before shuffle, the problem still exists.
Does this means, if we can generate non-determine output, then we can still
loss some data even add a local sort before shuffle, because the reduce tasks
may have already finished (or even have committed)?
> be more conservative when handling fetch failure and rerun more reduce
tasks. We can provide an internal API to tag a RDD as deterministic (very
common in Spark SQL) and then we can safely be optimistic when handling fetch
failure.
This is somehow like what I proposed yesterday, one issue we can't resolve
is that some ResultTasks may have committed, in that case it seems the best
effort we can make is just fail the job.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
