Hi community,
I'd like to start a discussion around the default value of
`celeborn.client.spark.fetch.cleanFailedShuffle`, introduced in 0.6.0.
Background
----------
The config gates a cleanup path that reclaims disk space held by
shuffle files which can no longer be fetched. The mechanism itself
looks safe:
* When a FetchFailed is captured on the driver, the failed shuffleId
is recorded.
* When the retried stage registers a new shuffleId, the previously
failed shuffleId is enqueued for cleanup.
* FailedShuffleCleaner then reuses the existing UnregisterShuffle
path to remove the stale shuffle data on the workers.
Since 0.4.0, `celeborn.client.spark.stageRerun.enabled` defaults to
`true`, so FetchFailed stage rerun is already the out-of-the-box
behavior. However `cleanFailedShuffle` still defaults to `false`,
which means every retried stage silently leaves its old shuffle data
on the workers until the app exits or the periodic expired-shuffle
cleanup catches it.
Why this matters in production
------------------------------
We've hit the disk-pressure side of this on large shuffle jobs where
a stage retries multiple times. A recent case from our cluster:
* A single stage retried three times.
* Retry #1 wrote ~143 TiB of shuffle data before failing.
* Retry #2 wrote another ~143 TiB before failing.
* Retry #3 was still running, ~61 TiB written so far.
With `cleanFailedShuffle=false`, all three generations of shuffle data
sit on the workers simultaneously — well over 300 TiB of stale bytes
that the job will never fetch again. On a shared cluster this drives
workers into disk-full / high-usage exclusion, cascading into more
FetchFailed for unrelated apps and destabilizing the cluster until
the app finally exits. Turning `cleanFailedShuffle` on effectively
removes this failure mode for us.
Questions for the community
---------------------------
1. What was the rationale for keeping `cleanFailedShuffle` disabled
by default when it was introduced?
2. Would the community be open to flipping the default to `true`
in a future release?
3. Are there scenarios where enabling this flag could cause a
late-arriving fetch to hit a "shuffle not registered" error
instead of the current benign leak?
Happy to follow up with a JIRA / doc PR / default-flip PR
once the direction is clear.
Thanks,
Haihua Wang (Randy)
