[
https://issues.apache.org/jira/browse/SPARK-59138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59138:
-----------------------------------
Labels: pull-request-available (was: )
> Preserve reliably-stored shuffle map outputs on executor loss
> -------------------------------------------------------------
>
> Key: SPARK-59138
> URL: https://issues.apache.org/jira/browse/SPARK-59138
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.3
> Reporter: Venkata krishnan Sowrirajan
> Priority: Major
> Labels: pull-request-available
>
> On executor loss, DAGScheduler unregisters all map outputs on the lost
> executor and forces the map stage to recompute. When a shuffle's output is
> reliably stored off-executor (e.g. a remote shuffle service such as
> Celeborn), the data survives the executor, so this recompute is wasteful.
> Today reliability is expressed only through
> ShuffleDriverComponents.supportsReliableStorage(), a single application-wide
> flag. That cannot represent a mixed setup where one shuffle lives on the
> remote service while another falls back to local disk on the same executor
> (e.g. Celeborn AUTO fallback). As a result Spark either recomputes reliable
> shuffles unnecessarily, or would skip recompute for local-disk fallback
> shuffles whose output was genuinely lost.
> Make reliability per-shuffle:
> - Add ShuffleHandle.isReliablyStored (default false), overridable by a
> ShuffleManager that routes a shuffle to reliable storage.
> - Store the bit per shuffle in MapOutputTracker and skip only
> reliably-stored shuffles when unregistering outputs on executor/worker loss.
> Genuine fetch failures continue to unregister everything.
> - Honor the per-shuffle bit in TaskSetManager's re-run gate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
