Hi Venkata,

Thanks for proposing this. +1 on the direction. An opt-in checker that reuses 
the existing decommission path sounds useful for environments where an external 
notification arrives before worker shutdown.

I agree this is complementary to the proposal in #3698: that addresses 
decommission when SIGTERM initiates shutdown, while this allows the worker to 
start earlier.

Regarding the CIP question, I would suggest a short CIP, since this introduces 
a public extension interface and new configuration options. It would help us 
agree on the contract for custom implementations and their compatibility 
expectations.

A few points would be useful to cover:
- The checker lifecycle and failure behavior, including slow or blocked checks. 
I see the draft already covers initialization failures and polling exceptions.
- How repeated signals and concurrent REST, master-driven, or shutdown events 
interact, so they consistently follow the existing worker state transitions.
- How the polling interval and decommission timeout relate to an external 
termination deadline. An earlier trigger provides more time to drain, but 
cannot guarantee completion before a forced termination.
The no-op default and file-based example seem like a reasonable initial scope. 
The CIP can stay focused on this extension point and its integration with the 
existing lifecycle.

Best regards,
Nicholas Jiang

On 2026-11-01 20:32 UTC Venkatakrishnan Sowrirajan wrote:
> Hi Celeborn devs,
> 
> I'd like to propose letting a Celeborn worker decommission itself when an
> external signal fires, through a small pluggable checker.
> 
> Today that only happens when someone calls the REST API. But in cloud and
> autoscaled setups, the first heads-up usually comes from somewhere else,
> like a spot preemption notice or a node drain. So basically every
> deployment ends up writing a sidecar that watches for it and calls the API.
> 
> The idea is a small pluggable checker in the worker. The worker polls it,
> and when it says yes, the worker goes through the existing DECOMMISSION
> path (same as the REST call). The default is a no-op, so nothing changes
> unless you configure one. I've put up a draft PR with a simple file-based
> checker as an example.
> 
> This is related to https://github.com/apache/celeborn/pull/3698
> (decommission on SIGTERM). I think the two are complementary... that one
> handles the worker already being killed, this one acts on the external
> signal before that.
> 
> Does this warrant a CIP?
> 
> JIRA: https://issues.apache.org/jira/browse/CELEBORN-2536
> PR: https://github.com/apache/celeborn/pull/3939
> 
> Regards
> Venkata krishnan
> 

Reply via email to