Zhongwei Zhu created SPARK-44345:
------------------------------------
Summary: Only log unknown shuffle map output as error when shuffle
migration disabled
Key: SPARK-44345
URL: https://issues.apache.org/jira/browse/SPARK-44345
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.4.1
Reporter: Zhongwei Zhu
When decommission and shuffle migration is enabled, there're lots of error
message like `Asked to update map output for unknown shuffle` .
As shuffle clean and unregister is done by ContextCleaner in an async way, when
target block manager received the shuffle block from decommissioned block
manager, then update shuffle location to map output tracker, but at that time,
shuffle might have been unregistered. This should not consider as error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
