GitHub user CaptainAni187 added a comment to the discussion: DAGs with external
python task are not killed when marked as Failed.
This is a known bug in the 3.2 line, and it's fixed in Airflow 3.3.1.
When you mark the task as failed, the next heartbeat from the task gets a 409
and the supervisor kills the task process. That's the "Server indicated the
task shouldn't be running anymore" line in your log. In 3.2.2 the signal only
goes to the task-runner process itself. Your `@task.external_python` function
runs in a separate Python process that the task-runner starts with
`subprocess.Popen`, so it never receives the signal. Once the task-runner is
killed, that child gets reparented and keeps running, which is why your log
keeps printing `alive` after the termination message.
It was reported in #65505 and fixed by #65738. The fix puts the task-runner in
its own process group and sends the signal to the whole group with `os.killpg`,
so venv and external Python children get it too. I checked the release tags,
and that code is in 3.3.1 and 3.3.2 but not in 3.2.2 or 3.3.0, so upgrading to
3.3.1 or later should fix it.
If you can't upgrade yet, one workaround is to have the long-running function
notice that its parent is gone and exit on its own:
```python
import os
import sys
import time
parent = os.getppid()
while True:
if os.getppid() != parent:
sys.exit(1)
print("alive", flush=True)
time.sleep(1)
```
That only helps for code that loops like this, but it stops the orphaned
process from running forever.
GitHub link:
https://github.com/apache/airflow/discussions/74298#discussioncomment-18776309
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
