Thanks Nicholas for the quick response. That makes sense, let me start a
discussion with the CIP.

Regards
Venkata krishnan


On Sun, Nov 1, 2026 at 6:41 PM Nicholas Jiang via dev <
[email protected]> wrote:

> Hi Venkata,
>
> Thanks for proposing this. +1 on the direction. An opt-in checker that
> reuses the existing decommission path sounds useful for environments where
> an external notification arrives before worker shutdown.
>
> I agree this is complementary to the proposal in #3698: that addresses
> decommission when SIGTERM initiates shutdown, while this allows the worker
> to start earlier.
>
> Regarding the CIP question, I would suggest a short CIP, since this
> introduces a public extension interface and new configuration options. It
> would help us agree on the contract for custom implementations and their
> compatibility expectations.
>
> A few points would be useful to cover:
> - The checker lifecycle and failure behavior, including slow or blocked
> checks. I see the draft already covers initialization failures and polling
> exceptions.
> - How repeated signals and concurrent REST, master-driven, or shutdown
> events interact, so they consistently follow the existing worker state
> transitions.
> - How the polling interval and decommission timeout relate to an external
> termination deadline. An earlier trigger provides more time to drain, but
> cannot guarantee completion before a forced termination.
> The no-op default and file-based example seem like a reasonable initial
> scope. The CIP can stay focused on this extension point and its integration
> with the existing lifecycle.
>
> Best regards,
> Nicholas Jiang
>
> On 2026-11-01 20:32 UTC Venkatakrishnan Sowrirajan wrote:
> > Hi Celeborn devs,
> >
> > I'd like to propose letting a Celeborn worker decommission itself when an
> > external signal fires, through a small pluggable checker.
> >
> > Today that only happens when someone calls the REST API. But in cloud and
> > autoscaled setups, the first heads-up usually comes from somewhere else,
> > like a spot preemption notice or a node drain. So basically every
> > deployment ends up writing a sidecar that watches for it and calls the
> API.
> >
> > The idea is a small pluggable checker in the worker. The worker polls it,
> > and when it says yes, the worker goes through the existing DECOMMISSION
> > path (same as the REST call). The default is a no-op, so nothing changes
> > unless you configure one. I've put up a draft PR with a simple file-based
> > checker as an example.
> >
> > This is related to https://github.com/apache/celeborn/pull/3698
> > (decommission on SIGTERM). I think the two are complementary... that one
> > handles the worker already being killed, this one acts on the external
> > signal before that.
> >
> > Does this warrant a CIP?
> >
> > JIRA: https://issues.apache.org/jira/browse/CELEBORN-2536
> > PR: https://github.com/apache/celeborn/pull/3939
> >
> > Regards
> > Venkata krishnan
> >
>

Reply via email to