[ https://issues.apache.org/jira/browse/SPARK-33747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247592#comment-17247592 ]
Apache Spark commented on SPARK-33747: -------------------------------------- User 'weixiuli' has created a pull request for this issue: https://github.com/apache/spark/pull/30716 > Avoid calling unregisterMapOutput when the map stage is being rerunning. > ------------------------------------------------------------------------ > > Key: SPARK-33747 > URL: https://issues.apache.org/jira/browse/SPARK-33747 > Project: Spark > Issue Type: Bug > Components: Block Manager > Affects Versions: 2.4.5, 3.0.1 > Reporter: weixiuli > Priority: Major > Fix For: 2.4.5, 3.0.1 > > > When a fetch failure happened, DAGScheduler will try to unregister the > corresponding map output. The current logic has a race condition that the new > map stage attempt is running while the old reduce stage attempt returns > another fetch failure. In this case, if the map output is always > unregistered, it may actually unregister the map output from the new map > stage attempt. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org