divaltor opened a new pull request, #77020: URL: https://github.com/apache/airflow/pull/77020
After the supervisor bare-forks the task process, a full GC in either process writes to the headers of the objects the task process inherited. That splits the copy-on-write pages, so every KubernetesExecutor pod holds two copies of the supervisor's heap, including the task runner that `_import_task_runner_before_fork` loads. ```python gc_frozen = target is _subprocess_main and not use_exec ``` The freeze uses the same gate as the task-runner pre-import, so a spawned task process (macOS, `execute_tasks_new_python_interpreter`) is unaffected. `wait()` unfreezes after the task process exits. LocalExecutor workers and Celery children run `supervise()` once per task in one long-lived process, and staying frozen like the triggerer (#76016) would leak there. | DAG file | Pod PSS before | Pod PSS after | |---|---|---| | One `EmptyOperator`, no other imports | 178-200 MB | **151 MB** | | Same, plus top-level `import pandas` | 225-234 MB | 198 MB | <details><summary>Measurements</summary> `apache/airflow:slim-3.4.0-python3.12`, linux/arm64, three runs each. The script copies the supervisor's import and bare-fork order, the child parses the DAG with `DagBag`, both processes run `gc.collect()`, and PSS comes from `/proc/<pid>/smaps_rollup`. With the freeze, the result was the same on every run. Without it, it depends on when the GC runs, so long tasks lose more over time. Same fix for other forkers: #58365 (LocalExecutor), #60505 (Dag processor), #62212 (Celery worker), #76016 (triggerer). </details> --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Amp following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
