Hi Nicholas, Keyong,

Thanks for taking a look. Keyong, sounds great. With both of you aligned,
we'll keep the contract and plugins to own their I/O timeouts.

Nicholas, agreed! Here's my plan.

1. Lifecycle contract. You're right, getReason() runs before the trigger
today, so a blocked getReason() can hold back a check that already returned
true. I'll trigger first and log the reason after. The CIP-22 now clarifies
the contract for all three methods. shouldDecommission(), getReason() and
close() must return quickly and put their own timeouts on I/O. close() can
overlap a check that ignored the interrupt and outlived the 5s wait, so it
has to be thread-safe against that check and must not wait on it.

2. Timing. I hadn't separated entering InDecommission from rejecting
writes, and they're a few steps apart. The hook sends
ReportWorkerDecommission synchronously and only then sets shutdown. With
the defaults (askTimeout 60s, maxRetries 15), a hung master can stretch
that to minutes. The CIP now clarifies each step, says the 80s/105s numbers
assume a fast check and a responsive master, and adds an unavailable-master
test. REST and master-driven decommission share this ordering, so I've
filed CELEBORN-2538 to fix it for all triggers rather than inside CIP-22.
Does that work?

3. Exit race. Agreed, the first accepted exit should decide both the mode
and the timeout. I filed CELEBORN-2537 and linked it from the CIP. Will add
regression test (graceful accepted first, a later decommission rejected,
timeout unchanged) along with the fix.

4. Validation. I'll add unit tests for a checker that ignores interrupts
and for blocked getReason() and close(). And you're right that WorkerSuite
can't prove the shutdown path, since it stubs out System.exit. The test
plan now has cluster cases with real worker processes: SIGTERM during
polling, the hook and file cleanup, and an unavailable master.

CIP doc is now updated, and will address the changes in the PR shortly.

Regards
Venkata krishnan


On Sun, Nov 1, 2026 at 11:59 PM Keyong Zhou via dev <[email protected]>
wrote:

> Hi Venkata,
>
> Thanks for the proposal! The design looks good to me. For the open question
> in the document, I'm also inclined to keep the contract.
>
> Regards,
> Keyong Zhou.
>
> Venkatakrishnan Sowrirajan <[email protected]> 于2026年11月2日周一 14:56写道:
>
> > Hi Celeborn Devs,
> >
> > I would like to propose adding support for a pluggable worker
> > self-decommission mechanism based on an external signal which is
> especially
> > useful in the context of cloud environments with spot instances, K8s
> > environments with planned maintenance etc.
> >
> > Here is the proposal doc/design of CIP-22:
> >
> >
> https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing
> >
> > Please share your review comments or questions in the CIP-22 doc.
> >
> > Regards
> > Venkata krishnan
> >
>

Reply via email to