Nik-Tenzo opened a new pull request, #76298:
URL: https://github.com/apache/airflow/pull/76298
#62212 freezes the garbage collector in the Celery main process before the
pool forks, then unfreezes it on
`worker_ready`. The first pool processes keep their inherited memory shared,
but any pool process forked after
`worker_ready` inherits unfrozen objects: one replaced at
`worker_max_tasks_per_child` or
`worker_max_memory_per_child`, or added by `--autoscale`. Within a few
hundred tasks it rewrites much of what it
inherited, and those pages become private copies.
This removes the `worker_ready` handler, so the preloaded objects stay
frozen for the worker's lifetime. They are
import-time objects that live as long as the process, so keeping them frozen
costs nothing; the dag-processor
already freezes once and never unfreezes (#60505).
Measured on ECS Fargate (8 GB workers, concurrency 15–20,
`worker_max_tasks_per_child=1000`, Python 3.14, Airflow
3.3.2, provider 3.23.1), logging each pool process's USS every 50 tasks:
- Pool processes forked before `worker_ready`: ~52–56 MiB from task 50 to
1,000.
- Pool processes forked after it: start at ~51 MiB, then jump in one step to
~325 MiB after 150–250 tasks (5 of 5
sampled).
- Container memory climbed 1–2 GB an hour as the pool was replaced; with 20
slots the workers ran out of memory every
2.5–3 hours.
- Refreezing after `worker_ready` (our workaround): 30 replacement processes
past 250 tasks stayed at ~51–54 MiB,
container memory flat, and the main process at ~170 MiB instead of ~300
MiB.
The #62212 benchmark most likely only measured the first fork: the provider
doesn't set
`worker_max_tasks_per_child`, and Celery's default is no limit, so no pool
process was replaced.
related: #62212
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
