mahesh kumar behera created SPARK-47141:
-------------------------------------------
Summary: Support shuffle migration to external storage
Key: SPARK-47141
URL: https://issues.apache.org/jira/browse/SPARK-47141
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: mahesh kumar behera
Fix For: 4.0.0
Currently Spark supports migration of shuffle data to peer nodes during node
decommissioning. If peer nodes are not accessible, then Spark falls back to
external storage. User needs to provide the storage location path. There are
scenarios where user may want to migrate to external storage instead of peer
nodes. This may be because of unstable nodes or due to the need of aggressive
scale down. So user should be able to configure to migrate the shuffle data
directly to external storage if the use case permits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
