Hi Venkata, Thanks for proposing this. +1 on the direction. An opt-in checker that reuses the existing decommission path sounds useful for environments where an external notification arrives before worker shutdown.
I agree this is complementary to the proposal in #3698: that addresses decommission when SIGTERM initiates shutdown, while this allows the worker to start earlier. Regarding the CIP question, I would suggest a short CIP, since this introduces a public extension interface and new configuration options. It would help us agree on the contract for custom implementations and their compatibility expectations. A few points would be useful to cover: - The checker lifecycle and failure behavior, including slow or blocked checks. I see the draft already covers initialization failures and polling exceptions. - How repeated signals and concurrent REST, master-driven, or shutdown events interact, so they consistently follow the existing worker state transitions. - How the polling interval and decommission timeout relate to an external termination deadline. An earlier trigger provides more time to drain, but cannot guarantee completion before a forced termination. The no-op default and file-based example seem like a reasonable initial scope. The CIP can stay focused on this extension point and its integration with the existing lifecycle. Best regards, Nicholas Jiang On 2026-11-01 20:32 UTC Venkatakrishnan Sowrirajan wrote: > Hi Celeborn devs, > > I'd like to propose letting a Celeborn worker decommission itself when an > external signal fires, through a small pluggable checker. > > Today that only happens when someone calls the REST API. But in cloud and > autoscaled setups, the first heads-up usually comes from somewhere else, > like a spot preemption notice or a node drain. So basically every > deployment ends up writing a sidecar that watches for it and calls the API. > > The idea is a small pluggable checker in the worker. The worker polls it, > and when it says yes, the worker goes through the existing DECOMMISSION > path (same as the REST call). The default is a no-op, so nothing changes > unless you configure one. I've put up a draft PR with a simple file-based > checker as an example. > > This is related to https://github.com/apache/celeborn/pull/3698 > (decommission on SIGTERM). I think the two are complementary... that one > handles the worker already being killed, this one acts on the external > signal before that. > > Does this warrant a CIP? > > JIRA: https://issues.apache.org/jira/browse/CELEBORN-2536 > PR: https://github.com/apache/celeborn/pull/3939 > > Regards > Venkata krishnan >
