Shoryamishra61 opened a new pull request, #74577: URL: https://github.com/apache/airflow/pull/74577
### Description In Airflow 3, manual DAG run and task instance state transitions trigger OpenLineage listener hooks in the API server (`patch_dag_run_state` and `_emit_state_listener_hooks`). Previously, `OpenLineageListener.executor` lazily constructed a `ProcessPoolExecutor(initializer=_executor_initializer)` unconditionally. In the multithreaded asynchronous API server process (FastAPI / Uvicorn / AnyIO), calling `os.fork()` duplicates only the calling thread, leaving mutexes held by concurrent threads permanently locked in the child process. This caused child pool processes to deadlock in `futex_wait` and silently drop events. Furthermore, the API server's FastAPI lifespan never notified listener shutdown hooks (`before_stopping`), resulting in leaked orphan child processes when Gunicorn reloaded or retired API workers. This change: 1. Wires listener `on_starting` and `before_stopping` lifecycle hooks into the FastAPI app `lifespan` in `airflow.api_fastapi.app`. 2. Dynamically routes OpenLineage event dispatch through a bounded in-process `ThreadPoolExecutor` when running in the API server, eliminating `os.fork()` from the API server entirely while preserving the scheduler's `ProcessPoolExecutor` isolation and DB connection initializer. 3. Adds PID tracking (`_check_pid`) to reset executor references on post-fork PID change so parent executors are never inherited. 4. Adds double-checked locking for thread-safe executor creation under concurrent API requests. 5. Shuts down the thread pool with `wait=False, cancel_futures=True` on API worker exit so slow HTTP transports do not delay worker recycling. closes: #74544 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Antigravity (Gemini 3.8 Flash High) Generated-by: Antigravity (Gemini 3.8 Flash High) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
