wuyi created SPARK-53898:
----------------------------
Summary: MapOutputTrackerMaster.shufflestatuses is mistakenly
cleaned by Shuffle cleanup
Key: SPARK-53898
URL: https://issues.apache.org/jira/browse/SPARK-53898
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.1, 4.0.0
Reporter: wuyi
MapOutputTrackerMaster.shufflestatuses can be mistakenly cleaned by Shuffle
Cleanup feature, leading to SparkException (crashing the SparkContext) by the
subsequent access to that already removed shuffle metadata. A real case is the
ongoing subquery could access the shuffle metadata which has been already
cleanedup after the main query completes. See the detailed discussion at:
https://github.com/apache/spark/pull/52213#discussion_r2415632474.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
