Hi Venkata,

Yes, that works for me. Thanks for the detailed follow-up and for updating 
CIP-22.

Keeping I/O timeouts within the plugins sounds reasonable, with the 
prompt-return requirements and the possible overlap between close() and an 
outstanding check explicitly documented. Triggering decommission before 
retrieving the reason also addresses that concern.

Separating the shared shutdown-ordering issue and the exit race into 
CELEBORN-2538 and CELEBORN-2537 makes sense, since both affect existing 
triggers as well. Thanks for keeping them linked from the CIP and making the 
timing assumptions explicit.

The additional blocked-method tests and real-worker-process cases cover the 
validation gaps we discussed. In particular, the unavailable-master case should 
help verify the distinction between accepting decommission and actually 
rejecting writes.

This addresses my concerns at the design level. I’ll take another look once the 
corresponding PR changes and tests are ready.

Best regards,
Nicholas Jiang

On 2026/11/04 18:39:03 Venkatakrishnan Sowrirajan wrote:
> Gentle ping @Nicholas Jiang <[email protected]> and @Keyong Zhou
> <[email protected]> for further reviews! If you think it looks good, I can
> start a VOTE thread. Thanks!
> 
> Regards
> Venkata krishnan
> 
> 
> On Mon, Nov 2, 2026 at 11:07 AM Venkatakrishnan Sowrirajan <[email protected]>
> wrote:
> 
> > Hi Nicholas, Keyong,
> >
> > Thanks for taking a look. Keyong, sounds great. With both of you aligned,
> > we'll keep the contract and plugins to own their I/O timeouts.
> >
> > Nicholas, agreed! Here's my plan.
> >
> > 1. Lifecycle contract. You're right, getReason() runs before the trigger
> > today, so a blocked getReason() can hold back a check that already returned
> > true. I'll trigger first and log the reason after. The CIP-22 now clarifies
> > the contract for all three methods. shouldDecommission(), getReason() and
> > close() must return quickly and put their own timeouts on I/O. close() can
> > overlap a check that ignored the interrupt and outlived the 5s wait, so it
> > has to be thread-safe against that check and must not wait on it.
> >
> > 2. Timing. I hadn't separated entering InDecommission from rejecting
> > writes, and they're a few steps apart. The hook sends
> > ReportWorkerDecommission synchronously and only then sets shutdown. With
> > the defaults (askTimeout 60s, maxRetries 15), a hung master can stretch
> > that to minutes. The CIP now clarifies each step, says the 80s/105s
> > numbers assume a fast check and a responsive master, and adds an
> > unavailable-master test. REST and master-driven decommission share this
> > ordering, so I've filed CELEBORN-2538 to fix it for all triggers rather
> > than inside CIP-22. Does that work?
> >
> > 3. Exit race. Agreed, the first accepted exit should decide both the mode
> > and the timeout. I filed CELEBORN-2537 and linked it from the CIP. Will
> > add regression test (graceful accepted first, a later decommission
> > rejected, timeout unchanged) along with the fix.
> >
> > 4. Validation. I'll add unit tests for a checker that ignores interrupts
> > and for blocked getReason() and close(). And you're right that WorkerSuite
> > can't prove the shutdown path, since it stubs out System.exit. The test
> > plan now has cluster cases with real worker processes: SIGTERM during
> > polling, the hook and file cleanup, and an unavailable master.
> >
> > CIP doc is now updated, and will address the changes in the PR shortly.
> >
> > Regards
> > Venkata krishnan
> >
> >
> > On Sun, Nov 1, 2026 at 11:59 PM Keyong Zhou via dev <
> > [email protected]> wrote:
> >
> >> Hi Venkata,
> >>
> >> Thanks for the proposal! The design looks good to me. For the open
> >> question
> >> in the document, I'm also inclined to keep the contract.
> >>
> >> Regards,
> >> Keyong Zhou.
> >>
> >> Venkatakrishnan Sowrirajan <[email protected]> 于2026年11月2日周一 14:56写道:
> >>
> >> > Hi Celeborn Devs,
> >> >
> >> > I would like to propose adding support for a pluggable worker
> >> > self-decommission mechanism based on an external signal which is
> >> especially
> >> > useful in the context of cloud environments with spot instances, K8s
> >> > environments with planned maintenance etc.
> >> >
> >> > Here is the proposal doc/design of CIP-22:
> >> >
> >> >
> >> https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing
> >> >
> >> > Please share your review comments or questions in the CIP-22 doc.
> >> >
> >> > Regards
> >> > Venkata krishnan
> >> >
> >>
> >
> 

Reply via email to