seanmuth opened a new pull request, #72035: URL: https://github.com/apache/airflow/pull/72035
A forked task child could hang forever if any library registered an atexit handler that isn't fork-safe -- one that holds a native lock or thread that doesn't survive `fork()`. Task-SDK's custom exit path in `_fork_main` ran these handlers via `atexit._run_exitfuncs()` before calling `os._exit()`, so a single fork-unsafe handler could block that final `os._exit()` from ever being reached, leaving the child -- and the pod hosting it -- stuck until its grace period expired. This was hit in production via pyarrow's S3 client finalizer deadlocking in its AWS-CRT teardown after `pyiceberg` registered it post-fork. The same failure mode applies to any fork-unsafe atexit handler in any library, so this skips atexit entirely in the forked child rather than trying to special-case known offenders -- there's no way to tell a safe handler from an unsafe one in advance. The prior post-fork-atexit behavior was a deliberate choice (see discussion on the internal tracking ticket) to let task code tidy up after the process finishes. This PR trades that intentional cleanup opportunity for guaranteed exit -- happy to discuss alternatives (e.g. running atexit with a timeout) if that tradeoff needs more thought. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes -- Claude Code (Sonnet 5) Generated-by: Claude Code (Sonnet 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
