seanmuth opened a new pull request, #73503: URL: https://github.com/apache/airflow/pull/73503
`execute_tasks_new_python_interpreter` (#72164) and the macOS-forced exec path both still call `os.fork()` before `execv()`. CPython's `os.fork()` runs every `os.register_at_fork(after_in_child=...)` callback synchronously, inside the `fork()` call itself, before any Python-level code — including the planned `execv()` — gets control back. A third-party library's own fork handler that blocks there hangs the child before exec is ever reached, regardless of how soon the caller tries to exec — confirmed live: a deployment's task process hung inside `datadog`'s dogstatsd client, which registers exactly such a handler by default. This isn't just Airflow's own known OpenSSL provider-store case (#71707) — it's any library that registers an at-fork handler that isn't async-signal-safe, and fork+exec can't protect against that structurally, no matter how the call sites are ordered. `os.posix_spawn()` doesn't have this gap: CPython's binding never calls `PyOS_AfterFork_Child()`, and glibc's own `posix_spawn` (2.24+) uses `clone(CLONE_VM|CLONE_VFORK)` rather than `fork()`, so registered `os.register_at_fork()`/`pthread_atfork()` handlers are structurally unreachable, not just less likely to hang. It's also not a new cost on top of the existing exec path — benchmarked against a real Airflow import, `posix_spawn` is measurably not more expensive than the fork+exec it replaces (slightly cheaper, from skipping `fork()`'s own copy-on-write setup before the exec). No new config surface: wherever `use_exec` was already `True` (the platform gate or `execute_tasks_new_python_interpreter`), the spawn mechanism underneath is now always `posix_spawn`. That decision is already made by existing config; this only changes how "give me a fresh interpreter" is implemented once it's been decided. related: #71707, #72164, #72493 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Sonnet 5 Generated-by: Claude Sonnet 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
