ignacioparicio opened a new pull request, #70901: URL: https://github.com/apache/airflow/pull/70901
`[core] execute_tasks_new_python_interpreter` is still documented as a live option, but since Airflow 3.0 CeleryExecutor ignores it. The code that read it now sits behind `if not AIRFLOW_V_3_0_PLUS:` in the Celery provider, and the Task SDK path that replaced it never carried the setting forward. That happened in https://github.com/apache/airflow/pull/46265 ("Swap CeleryExecutor over to use TaskSDK for execution", commit 984c61d287). That PR was about moving Celery onto the Task SDK. The old `execute_command` task — the only thing that ever read this setting — was kept behind a version guard so the provider would keep working on Airflow 2, and the new `execute_workload` path simply never reimplemented it. Losing the option looks like a side effect rather than a decision: the PR body does not mention it, and there was no newsfragment or deprecation. What is left is inconsistent: - `config.yml` documents the option under `[core]` with no deprecation marker. - `plugins.rst` still tells users to set it to pick up plugin changes without restarting the worker. - The Edge worker does honour it on Airflow 3, added in #65943 with a dedicated test. - CeleryExecutor and LocalExecutor silently ignore it. - `airflow config lint`, the designated 2 to 3 upgrade check, does not mention it, so an upgrade gives no signal at all. We hit this upgrading a CeleryExecutor deployment from 2.11 to 3.3. We had the option set to `True`, so every task ran in a fresh interpreter. On 3.3 tasks are forked from the Celery prefork worker instead, and any task that starts a `multiprocessing` child now inherits the worker's daemon flag and dies with `daemonic processes are not allowed to have children`. Nothing in the upgrade told us the option had stopped working; we found it by reading the provider source. This PR changes no behaviour. It makes the current state discoverable: - `config.yml`: note that only the Edge executor honours the option. - `plugins.rst`: drop the advice to set it, since it no longer works for Celery or Local. - `airflow config lint`: flag it when set to `True`. Restoring a per-task fresh interpreter is a separate discussion, and #70280 looks like the right place for it. Two things I deliberately left out, happy to add either if you disagree: - No newsfragment. No behaviour change, and the existing precedents for adding a `ConfigChange` entry did not add one. - No startup warning from the Celery worker. It would catch deployments that never run `config lint`, but it is a provider change with its own changelog, so it felt like a separate PR. One known limitation: `is_invalid_if` compares the raw config string, so the lint rule matches `True` but not `true` or `1`. Making it boolean-aware means touching `ConfigChange.message`, which every entry in that list runs through, so I kept this PR to the one entry. Happy to generalise it here or in a follow-up. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Used to map the blast radius: which code paths, docs, and CLI checks still reference the option, and which executors still honour it. Findings verified against the source by hand; the wording and the scope of the change are mine. Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
