aeroyorch opened a new issue, #69001: URL: https://github.com/apache/airflow/issues/69001
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.2.2 ### What happened and how to reproduce it? The scheduler memory grows until the process is OOMKilled and restarted. I attach a Grafana graph with the growth over time. <img width="1600" height="499" alt="Image" src="https://github.com/user-attachments/assets/bd398687-bfdd-4df1-ba24-b23ef67fed06"; /> We profiled the scheduler with `memray`, following the [Airflow memory profiling guide](https://airflow.apache.org/docs/apache-airflow/stable/howto/memory-profiling.html). The part that keeps growing is the `DBDagBag` cache of deserialized DAGs, which in the scheduler is a plain dict with no eviction. Two memray snapshots from one run, with the memory retained under `get_dag_for_run` / `_get_dag`: | Snapshot | Peak heap | `_get_dag` (DBDagBag cache) | |----------|-----------|-----------------------------| | snapshot 1 | 1.38 GB | 464.9 MB | | snapshot 2 | 2.18 GB | 669.5 MB | The snapshots also show another large part, the task instances loaded during each scheduling pass. That one seems normal. The cause seems to be in `SchedulerJobRunner.__init__`: `self.scheduler_dag_bag = DBDagBag(load_op_links=False)`. With no `cache_size`, `DBDagBag._dags` has no size or TTL limit, so it keeps one deserialized DAG per `dag_version_id` and does not drop old versions. The bag lives for the whole scheduler process, which by default runs indefinitely (`[scheduler] num_runs = -1`). As the number of DAG versions grows over time, this becomes unbounded. To reproduce: DAGs that keep producing new versions over time, a standalone dag-processor, and a scheduler that runs for a long time. Watch `len(scheduler_dag_bag._dags)` and the process resident memory (RSS). Both grow continuously and only go down after a restart. ### What you think should happen instead? The scheduler `DBDagBag` should use a bounded LRU+TTL cache, like the API server since #60804 (that PR limited the change to the API server only). Two settings, `[scheduler] dag_cache_size` and `dag_cache_ttl`, like the `[api]` ones, would keep it bounded, with a size for the active set of DAG versions and a TTL to drop the old ones. A value of 0 keeps today's unbounded behavior. ### Operating System _No response_ ### Deployment Official Apache Airflow Helm Chart ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers _No response_ ### Official Helm Chart version 1.21.0 ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
