Venkata krishnan Sowrirajan created SPARK-59138:
---------------------------------------------------
Summary: Preserve reliably-stored shuffle map outputs on executor
loss
Key: SPARK-59138
URL: https://issues.apache.org/jira/browse/SPARK-59138
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.3
Reporter: Venkata krishnan Sowrirajan
On executor loss, DAGScheduler unregisters all map outputs on the lost executor
and forces the map stage to recompute. When a shuffle's output is reliably
stored off-executor (e.g. a remote shuffle service such as Celeborn), the data
survives the executor, so this recompute is wasteful.
Today reliability is expressed only through
ShuffleDriverComponents.supportsReliableStorage(), a single application-wide
flag. That cannot represent a mixed setup where one shuffle lives on the remote
service while another falls back to local disk on the same executor (e.g.
Celeborn AUTO fallback). As a result Spark either recomputes reliable shuffles
unnecessarily, or would skip recompute for local-disk fallback shuffles whose
output was genuinely lost.
Make reliability per-shuffle:
- Add ShuffleHandle.isReliablyStored (default false), overridable by a
ShuffleManager that routes a shuffle to reliable storage.
- Store the bit per shuffle in MapOutputTracker and skip only reliably-stored
shuffles when unregistering outputs on executor/worker loss. Genuine fetch
failures continue to unregister everything.
- Honor the per-shuffle bit in TaskSetManager's re-run gate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
