Venkata krishnan Sowrirajan created SPARK-59807:
---------------------------------------------------
Summary: Make executor hold storage-aware for reliably-stored
shuffles
Key: SPARK-59807
URL: https://issues.apache.org/jira/browse/SPARK-59807
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.2.0
Reporter: Venkata krishnan Sowrirajan
When deciding whether an executor may be released (the executor-hold /
retirement path), Spark consults the app-global
ShuffleDriverComponents.supportsReliableStorage(). In a mixed configuration
this holds executors that only produced reliably-stored shuffle output, whose
data survives their loss, and conversely cannot represent an executor holding a
local-fallback shuffle when the global flag says storage is reliable.
SPARK-59138 added the aggregate primitive
MapOutputTrackerMaster.hasUnreliablyStoredShuffle, an application-wide query
over the active ordinary-shuffle registry, but no consumer. This ticket wires
the executor-hold decision to that aggregate instead of the global flag alone,
so an executor is held only while at least one shuffle it is needed for is not
reliably stored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
