namanjain24-sudo opened a new pull request, #75545:
URL: https://github.com/apache/airflow/pull/75545
The coordinators start a language runtime with `subprocess.Popen` and leave
it in the supervisor's own
process group, so stopping the task signals the runtime alone. Anything the
task started — a CLI spawned
from a TypeScript handler, for instance — is re-parented to init and keeps
running alongside the next try,
and holds the runtime's event loop open until the SIGKILL escalation fires
after `killed_task_cleanup_time`.
Python tasks stopped leaking this way in #65505: the task process runs in
its own process group so the
supervisor can signal the whole group.
`WatchedSubprocess._signal_subprocess` already does that with
`os.killpg` whenever `new_process_group` was recorded at start time, so the
runtime only needs the same
treatment — `process_group=0` on the `Popen`, and the flag set on the
subprocess object.
This is direction 1 of the two the issue lists. It fixes every path that
stops the task, since they all go
through `_signal_subprocess`: clearing a running task, `execution_timeout`,
repeated heartbeat failures and
task-success overtime.
`process_group` is POSIX-only, which matches the rest of this machinery —
the Python path already calls
`os.posix_spawn(..., setpgroup=0)` and `os.fork()` + `os.setpgid`
unconditionally.
closes: #75533
### Testing
Checked the real behaviour rather than only the call, with a runtime that
spawns a child the way the
issue's handler does:
```
WITHOUT process_group (old): own_pgid=False grandchild_alive_after_stop=True
WITH process_group=0 (fix): own_pgid=True
grandchild_alive_after_stop=False
```
- New test asserts the runtime is started with its own process group and
that the subprocess records it, so
`_signal_subprocess` takes the `killpg` path. It fails without the change
(`KeyError: 'process_group'`).
- `task-sdk/tests/task_sdk/coordinators/` — 475 passed.
- `task-sdk/tests/task_sdk/execution_time/test_supervisor.py` — 374 passed,
1 skipped.
- `breeze run mypy task-sdk/src/airflow/sdk/coordinators/_subprocess.py` —
clean.
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
