I-am-Uchenna opened a new pull request, #67901:
URL: https://github.com/apache/airflow/pull/67901
Closes #47160
### Problem
On Python 3.12+, the OpenLineage listener emits a `DeprecationWarning` when
running in a multi-threaded process:
```
DeprecationWarning: This process (pid=...) is multi-threaded, use of fork()
may lead to deadlocks in the child.
```
This warning comes from two sources in `listener.py`:
1. `_fork_execute` calls `os.fork()` directly to run OL event emission in a
child process
2. The `ProcessPoolExecutor` in the `executor` property defaults to the
`fork` start method on Linux
### Fix
**ProcessPoolExecutor**: Changed to use `forkserver` multiprocessing context
via `mp_context=multiprocessing.get_context("forkserver")`. The forkserver
starts a clean, single-threaded server process that handles subsequent forks
safely. All callables and arguments passed through `submit_callable` are
already picklable (module-level functions and basic types), so this is a
drop-in change.
**_fork_execute**: Added a `warnings.catch_warnings()` context around the
`os.fork()` call to suppress the specific `DeprecationWarning`. The fork here
is intentional by design: the child runs user-provided extractors that may not
be picklable, ruling out spawn/forkserver for this code path. The existing code
already takes proper precautions for safe forking, including ORM
reconfiguration with disabled connection pool, `os._exit(0)` to bypass atexit
handlers, and logging shutdown before exit.
^ Add meaningful description above
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_id}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
