hooiv commented on PR #70163: URL: https://github.com/apache/airflow/pull/70163#issuecomment-5070456409
I've pushed a completely new approach that fixes the actual root cause! In #70117, @Andrushika correctly pointed out that `dumb-init` couldn't be reaping the process while the supervisor was alive, but proved *something* was stealing the exit status. It turns out it was **Celery's `billiard` library**. Because the supervisor runs inside the Celery worker process, it inherits `billiard`'s aggressive global `SIGCHLD` handler. When the Airflow task subprocess exits, the `billiard` handler intercepts the signal and calls `waitpid(-1, WNOHANG)`, silently eating the exit status before our `psutil.wait()` can see it! I've force-pushed a fix that simply resets `SIGCHLD` to `SIG_DFL` inside `celery_executor_utils.py` right before the supervisor takes over. This prevents `billiard` from interfering and perfectly resolves the memory leak. The PR description has been updated . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
