Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21131
hmm, will we have a problem for shuffle here? Assuming a shuffle stage has
2 tasksets, one is zombie, one is normal. Both of them have running tasks.
if a task in zombie taskset finishes, it sends a task completion event to
dag scheduler, which will be ignored later as the stage attempt id is not the
latest. However, when the corresponding task in the normal taskset finishes, it
will not send event to dag scheduler because this task is already marked as
finished in this taskset. Then the shuffle stage never finishes.
cc @JoshRosen @vanzin @zsxwing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
