venkata91 opened a new pull request, #3834:
URL: https://github.com/apache/celeborn/pull/3834
### What changes were proposed in this pull request?
- Tie `supportsReliableStorage()` to the fallback policy: `true` only for
`NEVER`, `false` for `AUTO`/`ALWAYS` (was keyed on deprecated
`forceFallback`, returning `true` under AUTO).
- `SparkShuffleManager`: policy-aware DRA guidance; log the `registerShuffle`
fallback at `error` only when shuffle tracking is also off (else `warn`).
### Why are the changes needed?
- Fallback is per-shuffle, but `supportsReliableStorage()` is one global
flag.
Under AUTO a shuffle can fall back to local-disk `SortShuffleManager`; that
output dies with a DRA-reclaimed executor, risking `FetchFailedException`.
Today nothing prevents the unsafe config.
- Reporting `false` under AUTO/ALWAYS lets Spark's
`ExecutorAllocationManager`
fail fast unless a shuffle service, shuffle tracking, or decommissioning is
on, making AUTO safe with `shuffleTracking.enabled=true`; `NEVER` keeps the
fast path. Trade-off: under AUTO, Spark may recompute some map stages on
executor loss; `NEVER` avoids it.
### Does this PR resolve a correctness bug?
- [x] Yes
### Does this PR introduce _any_ user-facing change?
- [x] Yes: with AUTO (default), DRA without shuffle service and without
shuffle tracking now fails fast at startup. Enable tracking or set `NEVER`.
### How was this patch tested?
- New `testReliableStorageFollowsFallbackPolicy`;
`-Dtest=SparkShuffleManagerSuite` passes (10/10), spotless clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
