buska88 commented on code in PR #3824:
URL: https://github.com/apache/celeborn/pull/3824#discussion_r3849777307
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/memory/MemoryManager.java:
##########
@@ -411,6 +443,41 @@ public void switchServingState() {
}
}
+ /**
+ * Fires a drainIncompleteFrame tick when backpressure is active and
appActiveMemory is at/below
+ * the watermark.
+ */
+ private void checkAndTriggerDrainIncompleteFrame() {
+ if (!drainIncompleteFrameEnabled) return;
+
+ long appActiveMemory =
+ sortMemoryCounter.get()
+ + diskBufferCounter.get()
+ + memoryFileStorageCounter.sum()
+ + pendingReplicateBytesCounter.get();
Review Comment:
Thanks for pointing this out! I agree this can create the deadlock you
described.
One caveat: it's not a *permanent* deadlock — `failExpiredPushRequest()`
runs on
an independent scheduled thread that doesn't depend on the paused channel's
readability, so once `shufflePushDataTimeout` elapses, `onFailure` fires and
the
counter is released regardless. That said, recovery then falls back to
timeout-based retry (tens of seconds+) instead of the millisecond-level drain
this PR aims for, and repeats each cycle if backpressure persists — so I
still
think it's worth fixing.
On the three options: I'm hesitant about the first two (release on
write-complete,
or exclude the counter from the drain gate), since `pendingReplicateBytes`
reflects real retained memory — relaxing it could let drain resume more push
channels while replicate data is still genuinely occupying memory, risking
OOM.
I'd lean towards option 3 (exempting response/ack channels from the pause),
which
keeps the counter accurate while unblocking acks. Did you have a specific
approach in mind for identifying response channels, or is a fresh
registration
flag/separate limiter the way to go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
