jason810496 opened a new pull request, #68875: URL: https://github.com/apache/airflow/pull/68875
## Why Python 3.14 changed the Unix (non-macOS) default `multiprocessing` start method from `fork` to `forkserver` ([python/cpython#84559](https://github.com/python/cpython/issues/84559)). - `forkserver`/`spawn` children re-import Airflow instead of sharing it copy-on-write, so the scheduler's `LocalExecutor` workers and the triggerer each pay for a private copy. - Benchmarking 3.13 -> 3.14 measured steady-state PSS (Proportional Set Size, the copy-on-write-aware resident memory metric) up **+40-77%** for the scheduler and **+74-108%** for the triggerer. This restores control over the start method so deployments can recover the pre-3.14 footprint. It is opt-in: unset keeps the platform default, so there is no behaviour change unless configured. ## What - Add `[core] mp_start_method` (overridable per component via `[scheduler]`, `[triggerer]`, `[dag_processor]`, `[api]`) and `[core] mp_forkserver_preload`. - Add `resolve_mp_start_method` / `set_component_mp_start_method` to `process_utils`; the latter applies the configured method and forkserver preload modules, and is a safe no-op when unset or when the method is unavailable on the platform. - Call `set_component_mp_start_method` from the scheduler, triggerer, dag-processor, and api-server CLI entrypoints, before any `multiprocessing` work. - Resolve `LocalExecutor.is_mp_using_fork` per instance instead of once at import, so it honours a start method set by the CLI before the executor is created. Note: `[core] mp_start_method` existed historically but was removed (functional part in #48817, config remnants in #61150) and only ever governed the pre-3.0 Dag-processing manager, never `LocalExecutor`. The version restored here is wired into the executor and component start paths so it actually controls the workers. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
