Thanks Nicholas for the quick response. That makes sense, let me start a discussion with the CIP.
Regards Venkata krishnan On Sun, Nov 1, 2026 at 6:41 PM Nicholas Jiang via dev < [email protected]> wrote: > Hi Venkata, > > Thanks for proposing this. +1 on the direction. An opt-in checker that > reuses the existing decommission path sounds useful for environments where > an external notification arrives before worker shutdown. > > I agree this is complementary to the proposal in #3698: that addresses > decommission when SIGTERM initiates shutdown, while this allows the worker > to start earlier. > > Regarding the CIP question, I would suggest a short CIP, since this > introduces a public extension interface and new configuration options. It > would help us agree on the contract for custom implementations and their > compatibility expectations. > > A few points would be useful to cover: > - The checker lifecycle and failure behavior, including slow or blocked > checks. I see the draft already covers initialization failures and polling > exceptions. > - How repeated signals and concurrent REST, master-driven, or shutdown > events interact, so they consistently follow the existing worker state > transitions. > - How the polling interval and decommission timeout relate to an external > termination deadline. An earlier trigger provides more time to drain, but > cannot guarantee completion before a forced termination. > The no-op default and file-based example seem like a reasonable initial > scope. The CIP can stay focused on this extension point and its integration > with the existing lifecycle. > > Best regards, > Nicholas Jiang > > On 2026-11-01 20:32 UTC Venkatakrishnan Sowrirajan wrote: > > Hi Celeborn devs, > > > > I'd like to propose letting a Celeborn worker decommission itself when an > > external signal fires, through a small pluggable checker. > > > > Today that only happens when someone calls the REST API. But in cloud and > > autoscaled setups, the first heads-up usually comes from somewhere else, > > like a spot preemption notice or a node drain. So basically every > > deployment ends up writing a sidecar that watches for it and calls the > API. > > > > The idea is a small pluggable checker in the worker. The worker polls it, > > and when it says yes, the worker goes through the existing DECOMMISSION > > path (same as the REST call). The default is a no-op, so nothing changes > > unless you configure one. I've put up a draft PR with a simple file-based > > checker as an example. > > > > This is related to https://github.com/apache/celeborn/pull/3698 > > (decommission on SIGTERM). I think the two are complementary... that one > > handles the worker already being killed, this one acts on the external > > signal before that. > > > > Does this warrant a CIP? > > > > JIRA: https://issues.apache.org/jira/browse/CELEBORN-2536 > > PR: https://github.com/apache/celeborn/pull/3939 > > > > Regards > > Venkata krishnan > > >
