Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/22371
@squito , thanks for the review. I intend to using `ConcurrentHashMap[Int,
AtomicReferenceArray]` previously.
After re-think the code, I can know the lock here is used to prevent the
same task with different attempt to commit the shuffle writer result at the
same time. The task has a different attempt can be caused by follows:
1. Failed task or stage. In this case, the previous task attempt should
already finish(failed or killed) or the result is not used anymore.
2. `Speculative task`. In this case, the speculative task can't be
scheduled to the same executor as other attempts.
So, what's real value for the lock. Maybe I'm wrong, hopeful some answers.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
