venkata91 opened a new pull request, #3835: URL: https://github.com/apache/celeborn/pull/3835
### What changes were proposed in this pull request? - Adds a per-shuffle reliable-storage signal so Spark does not recompute a lost executor's map stages for shuffles whose data lives on Celeborn workers. - On executor loss, Spark's `DAGScheduler` unregisters the executor's map outputs and recomputes the map stage. A `CelebornShuffleHandle` is only created for shuffles stored on Celeborn workers (fallback shuffles use Spark's `SortShuffleManager`), so their output survives executor loss and the recompute is wasteful. - Overrides `ShuffleHandle.isReliablyStored` to return `true` on `CelebornShuffleHandle`; fallback shuffles keep Spark's `BaseShuffleHandle` (default `false`), so the mixed case is handled per shuffle. - The override is committed commented out: `ShuffleHandle.isReliablyStored` does not exist in any released Spark, so enabling it would break compilation against the Spark versions Celeborn builds against. - Blocked on the Spark-side PR (apache/spark#58437, SPARK-XXXXX). Do not merge until that lands, ships in a release, and Celeborn's Spark dependency is bumped; then uncomment the override and update the JIRA IDs. - Follow-up to CELEBORN-2451, which ties `supportsReliableStorage` to the `NEVER` fallback policy to stay DRA-safe. This PR lets AUTO fallback avoid the recompute penalty per shuffle. ### Why are the changes needed? - Under DRA with AUTO fallback, losing an executor needlessly recomputes map stages whose output is safely stored on Celeborn workers. - The app-global `supportsReliableStorage()` flag cannot express "this shuffle is on Celeborn, that one fell back to local disk," so it forces `NEVER` to be safe. Per-shuffle reliability removes that constraint. ### Does this PR resolve a correctness bug? - [ ] Yes ### Does this PR introduce _any_ user-facing change? - [ ] Yes ### How was this patch tested? - Not yet exercised end-to-end: the override is commented out pending the Spark dependency. - Once unblocked, validated against a Spark build carrying SPARK-XXXXX with a DRA + AUTO-fallback job that loses an executor, asserting the Celeborn-backed shuffle's map stage is not recomputed while a fallback shuffle's is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
