shivaam opened a new issue, #75533:
URL: https://github.com/apache/airflow/issues/75533
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
main (b43705cccee)
### What happened and how to reproduce it?
When the supervisor stops a task that a language SDK runs (TypeScript
through `NodeCoordinator`, and the same for `JavaCoordinator` and
`ExecutableCoordinator`), it signals only the runtime process. Processes the
task started keep running.
The common case is clearing a running task. The heartbeat gets a 409, the
supervisor sends SIGTERM, and the try ends with `server_terminated`, so the
task state is handled correctly. But a CLI the task started (for example with
`child_process.spawn`) is re-parented to PID 1 and keeps running alongside the
next try. While that child is alive it also keeps Node's event loop open, so
once the handler has returned, the runtime exits only when the SIGKILL
escalation fires after `[core] killed_task_cleanup_time` (60 s by default).
A TypeScript handler that shows it:
```ts
import { spawn } from "node:child_process";
import { Bundle, TaskHandler, getContext } from "apache-airflow-ts-sdk";
async function runCli() {
const { signal } = getContext();
spawn("sleep", ["300"], { stdio: "ignore" }); // stands in for any CLI the
task runs
await new Promise((_, reject) => signal.addEventListener("abort", () =>
reject(signal.reason)));
}
await new Bundle(new TaskHandler("my_dag", "run_cli", runCli)).serve();
```
Reproduced on `main` with a local test. It packs this handler with
`airflow-ts-pack` and runs it through `NodeCoordinator.execute_task` with the
real Node runtime. The Execution API is mocked and answers the heartbeat with
409 once the handler is running, as the API server does after the task is
cleared. The supervisor reported only `server_terminated`. But `sleep 300` was
still running with PPID 1, and the runtime was ended by SIGKILL (exit code -9)
after the escalation delay. This was not tried in a full deployment.
Python tasks have not had this problem since #65738 (for #65505): the task
process runs in its own process group, and the supervisor signals the group
with `os.killpg`. `_PopenActivitySubprocess.start` overrides
`ActivitySubprocess.start`. It calls `subprocess.Popen` without a process group
and does not set `new_process_group`, so `_signal_subprocess` signals the
runtime's pid alone:
-
https://github.com/apache/airflow/blob/b43705cccee66d6ef54c919ed1740efd3154d7ed/task-sdk/src/airflow/sdk/coordinators/_subprocess.py#L359-L393
-
https://github.com/apache/airflow/blob/b43705cccee66d6ef54c919ed1740efd3154d7ed/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1263-L1285
The same applies to the other ways the supervisor stops a language SDK task:
`execution_timeout`, repeated heartbeat failures and task-success overtime.
### What you think should happen instead?
Stopping a language SDK task should not leave the processes it started
running. Two broad directions, each worth researching:
1. **In the supervisor:** run the language runtime in its own process group
and signal the whole group, as is done for Python tasks. Emulating this in the
same test stopped the child.
2. **In the language runtime or the task:** have the SDK, or the task
author, stop the child processes when the task is aborted (for example, the TS
SDK's `ctx.signal` can be passed to `spawn`).
### Operating System
Reproduced on macOS; the launch code is the same on any POSIX worker.
### Deployment
Not applicable: reproduced with a test against `main`.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @shivaam before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
