wang-haihua opened a new pull request, #3779: URL: https://github.com/apache/celeborn/pull/3779
### What changes were proposed in this pull request? Flip the default of `celeborn.client.spark.fetch.cleanFailedShuffle` from `false` to `true`. Only two lines change: - `common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala` — `createWithDefault(false)` → `createWithDefault(true)` on `CLIENT_FETCH_CLEAN_FAILED_SHUFFLE`. - `docs/configuration/client.md` — the golden config table is updated in the same commit so `ConfigurationSuite` stays green. ### Why are the changes needed? `celeborn.client.spark.fetch.cleanFailedShuffle` was introduced in 0.6.0 by [CELEBORN-1896](https://issues.apache.org/jira/browse/CELEBORN-1896) (PR #3109) and hardened in 0.7.0 by [CELEBORN-2088](https://issues.apache.org/jira/browse/CELEBORN-2088) (PR #3401). When enabled: 1. The driver records shuffleIds that hit `FetchFailed`. 2. When the retried stage registers a new shuffleId, the previously failed shuffleId is enqueued for cleanup by `FailedShuffleCleaner`. 3. `FailedShuffleCleaner` reuses the existing `UnregisterShuffle` path to reclaim the stale shuffle data on workers. Since 0.4.0 `celeborn.client.spark.stageRerun.enabled` has defaulted to `true`, so `FetchFailed` stage rerun is already the out-of-the-box behavior. However `cleanFailedShuffle` still defaults to `false`, so every retried stage silently leaves its old shuffle data on the workers until the app exits or the periodic expired-shuffle cleanup catches it. The two defaults contradict each other. A recent production incident on our cluster: - A single stage retried three times. - Retry #1 wrote ~143 TiB of shuffle data before failing. - Retry #2 wrote another ~143 TiB before failing. - Retry #3 was still running, ~61 TiB written so far. - **300+ TiB of stale shuffle bytes were sitting on the workers simultaneously**, none of which the job would ever fetch again. On a shared cluster this drives workers into disk-full / high-usage exclusion, cascading into more `FetchFailed` events for unrelated applications and destabilizing the cluster until the offending app exits. Flipping the default removes this failure mode without adding any new deletion path. The proposal was raised on dev@ and supported there; see the [discussion thread](https://lists.apache.org/[email protected]) for context. ### Does this PR introduce _any_ user-facing change? Yes — behavior change with an explicit escape hatch. - **Before**: after a `FetchFailed`-driven stage rerun, the failed shuffleId's data lingered on the workers until the application exited (or `celeborn.worker.shuffle.expired.timeout` caught it). - **After**: the failed shuffleId's data is reclaimed as soon as the retried stage registers, via the existing `UnregisterShuffle` path. - **Rollback**: users who need the previous behavior can set `celeborn.client.spark.fetch.cleanFailedShuffle=false` in their Spark configuration. The release notes should call out both the new default and the rollback flag. ### How was this patch tested? - `ConfigurationSuite` — golden `docs/configuration/client.md` regenerated to keep the doc test green. - `CelebornFetchFailureDiskCleanSuite` — the end-to-end integration test for this path already sets `spark.celeborn.client.spark.fetch.cleanFailedShuffle=true` explicitly and continues to pass; it exercises the exact code path affected by this default flip. - No new deletion route or code path is introduced; the mechanism is the existing `FailedShuffleCleaner` → `UnregisterShuffle` flow that has shipped since 0.6.0. Related JIRA: [CELEBORN-2401](https://issues.apache.org/jira/browse/CELEBORN-2401). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
