gang-zh opened a new pull request, #72164: URL: https://github.com/apache/airflow/pull/72164
Resolves #71707. Supersedes #71723 — same approach and platform gate, credited to @pulaksaha143 as co-author; that PR has been idle since Aug 17 with static checks failing, so this picks it up and finishes it. ## Why On Linux the supervisor bare-forks the task process. If any supervisor thread (OTel exporter, google-auth refresh, …) holds a C-level lock like OpenSSL's at the instant of the fork, the child inherits it permanently locked and deadlocks at its first TLS call — typically the DAG bundle download — stuck in RUNNING forever. `execution_timeout` never fires because the hang precedes `execute()` (details in #71707). `exec()`-ing a fresh interpreter right after fork — what `_FORK_EXEC_PLATFORMS` already forces on macOS — makes inherited-lock state unreachable. This extends the existing `[core] execute_tasks_new_python_interpreter` config to opt into that path on any platform. It stays opt-in because the fresh interpreter re-imports the SDK and plugins: we measured **~+9s per task** on a plugin-heavy production image (it lands in task duration, not queued duration — the TI is marked running before the new interpreter boots). ## Production validation We've run this exact behavior (image-patched) on Airflow 3.2.2 / KubernetesExecutor in production: a TLS-stress probe deadlocked **16 of 13,930 bare-forked children** (`pthread_rwlock_wrlock` under `SSL_CTX_new_ex`) vs **0 of 5,195 with exec**; four production deployments since, thousands of task starts, zero hangs, deferral resume / retries / XCom / log shipping all normal. Full data: https://github.com/apache/airflow/pull/71723#issuecomment-5432889600 ## Changes vs #71723 - Test patches the config through `AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER` so it reaches the **task-sdk** conf object the supervisor reads (`conf_vars` patches airflow-core's conf, which the supervisor never consults) — and imports at top level per review feedback there. - Static checks pass (ruff format + lint clean). - `config.yml` description documents the new effect of the flag. A backport to v3-2/v3-3 would be very welcome — every KubernetesExecutor user on those lines forks cold per task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
