The GitHub Actions job "Tests (AMD)" on airflow.git/fix/celery-executor-app-cache-memory-leak has succeeded. Run started by GitHub user arkadiuszbach (triggered by arkadiuszbach).
Head commit for run: b1dab4722bb0aa1f4dba4cdc4ac53f4a8ac4f0f1 / bach.ab <[email protected]> fix(celery): cache Celery apps to stop per-send memory leak create_celery_app() rebuilt a brand-new Celery app on every workload send. When sends run in-process (a single task per heartbeat, or sync_parallelism == 1) this happens in the long-lived scheduler/executor process, so each rebuilt app — and its broker/result-backend connection pools — accumulates and is never released, leaking memory over time. PR #60675 assumed sends always run in throwaway ProcessPoolExecutor subprocesses, where a fresh app per send is harmless. The in-process hot path in _send_workloads_to_celery() breaks that assumption. Cache apps by their resolved (team-aware) name so each distinct app is built once per process and reused. Config is static per app-name per process, so this is safe for both call sites: subprocesses still build their own app once, and the in-process path stops recreating it on every send. Add regression tests for app caching/reuse and per-team app isolation. Report URL: https://github.com/apache/airflow/actions/runs/28039265237 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
