Hi Celeborn devs, I'd like to propose letting a Celeborn worker decommission itself when an external signal fires, through a small pluggable checker.
Today that only happens when someone calls the REST API. But in cloud and autoscaled setups, the first heads-up usually comes from somewhere else, like a spot preemption notice or a node drain. So basically every deployment ends up writing a sidecar that watches for it and calls the API. The idea is a small pluggable checker in the worker. The worker polls it, and when it says yes, the worker goes through the existing DECOMMISSION path (same as the REST call). The default is a no-op, so nothing changes unless you configure one. I've put up a draft PR with a simple file-based checker as an example. This is related to https://github.com/apache/celeborn/pull/3698 (decommission on SIGTERM). I think the two are complementary... that one handles the worker already being killed, this one acts on the external signal before that. Does this warrant a CIP? JIRA: https://issues.apache.org/jira/browse/CELEBORN-2536 PR: https://github.com/apache/celeborn/pull/3939 Regards Venkata krishnan
