Venkata krishnan Sowrirajan created SPARK-35549:
---------------------------------------------------
Summary: Register merge status even after shuffle dependency is
merge finalized
Key: SPARK-35549
URL: https://issues.apache.org/jira/browse/SPARK-35549
Project: Spark
Issue Type: Sub-task
Components: Shuffle, Spark Core
Affects Versions: 3.1.0
Reporter: Venkata krishnan Sowrirajan
Currently the merge statuses which arrive late from the external shuffle
services (or shuffle mergers) won't get registered once the shuffle dependency
merge is finalized.
This needs to be carefully done as there are lot of corner cases like:
a) executor/node loss causing re-computation due to fetch failure and if the
merge statuses gets registered very late then that can cause inconsistencies.
b) similar such cases
cc [~mridulm80] [~mshen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
