github-actions[bot] opened a new pull request, #76029: URL: https://github.com/apache/airflow/pull/76029
* Apply gc.freeze before forking serve_logs in the triggerer The triggerer forks serve_logs from its main process. Under the fork start method the child shares the parent's pages copy-on-write; a later full GC writes to every object's header and splits those shared pages into private copies, inflating anonymous memory with no new allocation. Freeze the GC before the fork and unfreeze right after, mirroring the executors (#58365), the Dag processor manager (#60505), and the Celery worker (#62212). No-op under forkserver/spawn. * Trim the gc.freeze comment to the non-obvious point * Keep the GC frozen after forking serve_logs (per review) Unfreezing right after the fork let the main process's next full GC split the COW-shared pages, cancelling the benefit. Freeze and stay frozen under the fork start method only (forks happen once, like the Dag processor manager); skip the freeze entirely for forkserver/spawn. * Move the gc.freeze to before the trigger-runner fork Freezing before the serve_logs fork was too early -- the heavy object graph is built afterward, so the pages shared with the later trigger-runner fork were still split by the next full GC and memory still stepped (~246->431 MB PSS). Freeze right before the runner's bare fork instead, gated on not _should_use_exec() (no COW sharing when the child execs), and stay frozen. Reverts the serve_logs change; measured flat (~245->246 MB PSS). * Freeze the GC before the serve_logs fork too, not only the runner fork Ephraim's 4-way PSS measurement on 3.4.0a4 (fork start method, GC forced in all three triggerer processes) showed the runner-fork freeze alone still steps (245 -> 341 MB): serve_logs is forked earlier, shares the heap, and its own full GC copies it. Freezing before the serve_logs fork keeps it flat and the runner inherits the frozen generation. Keep the runner-fork freeze for --skip-serve-logs and forkserver, where nothing is frozen beforehand. Co-Authored-By: Claude <[email protected]> --------- (cherry picked from commit 5af4c5bf12da93f975d3329c92deb660578462e9) Co-authored-by: Rahul Vats <[email protected]> Co-authored-by: Claude <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
