ephraimbuddy commented on code in PR #76016:
URL: https://github.com/apache/airflow/pull/76016#discussion_r4422811413
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -566,6 +567,12 @@ def start(
**kwargs,
):
proc_id = job.id if job is not None else uuid4()
+ if not supervisor._should_use_exec():
+ # Freeze before the bare fork and stay frozen: unfreezing would
let this process's next
+ # full GC split the pages the runner child shares copy-on-write (a
memory jump with no
+ # new allocation). Forked once, like the Dag processor manager.
Skipped when the child
+ # execs, which discards the inherited pages.
+ gc.freeze()
Review Comment:
Nit: the Dag processor manager forks per file; it just freezes once.
`_should_use_exec()` could also be computed once and passed to `use_exec=`
below.
```suggestion
if not supervisor._should_use_exec():
# The runner is forked once, so stay frozen: unfreezing would
let this process's next
# full GC copy the pages the runner shares. An exec'd child
shares nothing.
gc.freeze()
```
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @ephraimbuddy before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
