Hi Venkata, Thanks for the proposal. I support adding an opt-in, worker-side extension point that reuses the existing decommission path. Keeping cloud-specific implementations and shuffle-data migration outside this CIP seems reasonable.
After reviewing the updated proposal and PR #3939, I have a few comments: 1. Checker lifecycle and bounded execution. Could we extend the bounded-execution contract to getReason() and close()? Currently, getReason() runs before the decommission call, so a blocked diagnostic method can prevent an already-detected signal from taking effect. Also, the five-second shutdown wait only bounds the in-flight poll; the subsequent synchronous close() can still block worker cleanup. Please specify how implementations should safely handle close() overlapping an unfinished check. 2. Timing assumptions. Could the timing section distinguish entering InDecommission from actually rejecting new writes? In the existing path, sendWorkerDecommissionToMaster() runs synchronously before shutdown is set, so master RPC delays and retries can consume additional time. The 80s/105s examples therefore assume fast checks and a responsive master. Documenting those assumptions and testing an unavailable master would make the operational guidance more reliable. 3. Concurrent exit requests. Thanks for explicitly identifying the shutdown-hook timeout race. Handling it in a separate fix is reasonable, but could we link a tracking issue and add regression coverage? “The earlier decision wins” should cover both the exit mode and its timeout; a rejected decommission request should not change an accepted graceful shutdown’s timeout. 4. End-to-end validation. The proposed cluster test is useful. Please also cover SIGTERM arriving during polling, an interruption-ignoring checker, and blocked getReason()/`close()` callbacks. The current WorkerSuite test replaces System.exit, so a forked-JVM or cluster test is needed to validate the actual shutdown-hook and cleanup behavior. Regarding the per-check timeout question, I am comfortable starting with plugin-managed I/O timeouts, provided the full lifecycle contract is explicit. A generic executor timeout alone cannot stop non-interruptible plugin code. Regards, Nicholas Jiang On 2026-11-02 06:56 UTC Venkatakrishnan Sowrirajan wrote: > Hi Celeborn Devs, > > I would like to propose adding support for a pluggable worker > self-decommission mechanism based on an external signal which is especially > useful in the context of cloud environments with spot instances, K8s > environments with planned maintenance etc. > > Here is the proposal doc/design of CIP-22: > https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing > > Please share your review comments or questions in the CIP-22 doc. > > Regards > Venkata krishnan >
