Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21558
@tgravescs, that's exactly what we're seeing. I think it might just be
misleading to have a stage-local attempt ID although it is more friendly for
users and matches what MR does.
@jiangxb1987, we see SPARK-24492 occasionally (it has gotten better with
later fixes to the coordinator) and haven't tracked down the cause yet. If this
were the underlying cause that would be great, but I'm not sure how it could be
the cause. If the same attempt number is reused, then two tasks in different
stage attempts may both be authorized to commit. That wouldn't cause the
retries because it wouldn't cause extra commit denials.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
