buska88 opened a new pull request, #3824:
URL: https://github.com/apache/celeborn/pull/3824
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX]
Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a
faster review.
-->
### What changes were proposed in this pull request?
Introduce a `drainIncompleteFrame` mechanism to `MemoryManager` /
`ChannelsLimiter` /
`TransportFrameDecoder` that lets a worker recover from a backpressure
deadlock caused
by stuck half-frames in Netty's pipeline:
- `TransportFrameDecoder#hasLikelyLargeIncompleteFrame()` heuristically
detects
channels stuck with a large (> 64KB) buffered-but-undecoded frame.
- `TransportFrameDecoder#enableFrameDrain()` puts a channel into
single-frame "drain"
mode: once exactly one frame is decoded, it fires `FrameDrainCompleted` so
the
channel can be re-paused immediately without resuming full traffic.
- `MemoryManager#checkAndTriggerDrainIncompleteFrame()` periodically checks,
while
backpressure is active, whether application-layer active memory (sort
memory + disk
buffer + memory file storage + pending replicate bytes) is at/below a
configurable
watermark, and if so triggers a drain tick at a configurable interval.
- `ChannelsLimiter#drainIncompleteFrame(ratio)` scans paused channels for
likely-stuck
half-frames and resumes a configurable ratio of them in drain mode to
release the
stuck memory.
New configs added to `CelebornConf`:
- `celeborn.worker.monitor.drainIncompleteFrame.enabled` (default `true`)
- `celeborn.worker.monitor.drainIncompleteFrame.interval` (default `5ms`)
- `celeborn.worker.monitor.drainIncompleteFrame.ratio` (default `0.05`)
- `celeborn.worker.monitor.drainIncompleteFrame.watermark.ratio` (default
`0.1`)
### Why are the changes needed?
Worker monitors direct memory usage and pauses reading from all business
channels
(`autoRead=false`) once usage crosses a threshold, waiting for buffered data
to be
consumed before resuming. However, if `autoRead=false` takes effect exactly
when a
channel is mid-way through an incomplete frame, the buffered bytes stay
stuck inside
`TransportFrameDecoder` — never dispatched, never released, and invisible to
every
application-layer memory counter. This can produce a deadlock:
application-layer
counters report near-zero usage while the underlying direct memory doesn't
drop below
the resume threshold, because it's still holding onto these stuck
half-frames — and
since reads stay globally paused, the stuck half-frame never receives the
remaining
bytes needed to complete decoding. The worker becomes stuck and requires a
manual
restart to recover.
This is not a rare corner case in practice: with the common frame size
around a few
hundred KB (e.g. `celeborn.client.push.buffer.max.size=256KB`) and Netty's
`AdaptiveRecvByteBufAllocator` capping a single `channelRead` at ~64KB in
steady state,
a frame typically needs 4~5 reads to complete, making a stuck half-frame at
the moment
of pause fairly likely under sustained high throughput.
This issue was first discovered in production: some workers got stuck in
backpressure
and never recovered even though application-layer memory counters had
dropped to near
zero. Using Arthas thread/stack inspection together with internal memory
metrics, we
confirmed the stuck memory was sitting inside the Netty pipeline
(`TransportFrameDecoder`)
rather than any application-layer counter, matching the root cause above.
See [CELEBORN-2441](https://issues.apache.org/jira/browse/CELEBORN-2441) for
more details.
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed
automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [x] Yes
New configs are added under `celeborn.worker.monitor.drainIncompleteFrame.*`
(documented in `docs/configuration/worker.md`), all with safe defaults;
behavior is
unchanged from prior versions when
`celeborn.worker.monitor.drainIncompleteFrame.enabled=false`.
### How was this patch tested?
- Added unit tests:
- `TransportFrameDecoderSuiteJ` covers `hasLikelyLargeIncompleteFrame()`
and frame
drain behavior.
- `ChannelsLimiterSuiteJ` covers candidate selection and drain ratio
behavior.
- `MemoryManagerSuite` covers watermark-based triggering and pending
replicate
bytes accounting.
- Verified in production on a fleet of ~500 worker machines: after enabling
`drainIncompleteFrame`, workers that previously got stuck in backpressure
recovered
automatically, with no observed CPU regression during normal operation.
1. `celeborn.PausePushData.Delta` metric before/after rollout — a
backpressure spike
that used to stay stuck is now resolved within the same minute:
<img width="853" height="525" alt="image"
src="https://github.com/user-attachments/assets/1fbfa269-0291-4770-9e92-05908854180f";
/>
2. Worker log showing `drainIncompleteFrame` actively resuming stuck
channels and
the serving state transitioning back to `NONE_PAUSED` right after:
<img width="874" height="554" alt="3e3c6bdc-8619-439c-9e1b-3cb7ac927d79"
src="https://github.com/user-attachments/assets/b5c2f165-4bd4-4a23-87af-ad323c07bb73";
/>
3. `celeborn.PausePushDataTime.Delta` metric dropping back to `0` once
`drainIncompleteFrame` takes effect, meaning the worker is no longer in
backpressure for that minute:
<img width="1722" height="672" alt="image"
src="https://github.com/user-attachments/assets/a73c34ce-bebf-4f50-a703-931925ef543b";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
