BoYang created SPARK-33114:
------------------------------
Summary: Add metadata in MapStatus to support custom shuffle
manager
Key: SPARK-33114
URL: https://issues.apache.org/jira/browse/SPARK-33114
Project: Spark
Issue Type: Improvement
Components: Shuffle
Affects Versions: 3.0.1
Reporter: BoYang
Current MapStatus class is tightly bound with local (sort merge) shuffle which
uses BlockManagerId to store the shuffle data location. It could not support
other custom shuffle manager implementation.
We could add "metadata" to MapStatus and allow different shuffle manager
implementation to store information related to them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
