weixiuli created SPARK-33747:
--------------------------------
Summary: Avoid calling unregisterMapOutput when the map stage is
being rerunning.
Key: SPARK-33747
URL: https://issues.apache.org/jira/browse/SPARK-33747
Project: Spark
Issue Type: Bug
Components: Block Manager
Affects Versions: 3.0.1, 2.4.5
Reporter: weixiuli
When a fetch failure happened, DAGScheduler will try to unregister the
corresponding map output. The current logic has a race condition that the new
map stage attempt is running while the old reduce stage attempt returns another
fetch failure. In this case, if the map output is always unregistered, it may
actually unregister the map output from the new map stage attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
