aeroyorch commented on code in PR #69007:
URL: https://github.com/apache/airflow/pull/69007#discussion_r3479121651
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -347,8 +347,24 @@ def __init__(
if log:
self._log = log
+ dag_cache_size = conf.getint("scheduler", "dag_cache_size",
fallback=1024)
+ dag_cache_ttl_config = conf.getint("scheduler", "dag_cache_ttl",
fallback=3600)
- self.scheduler_dag_bag = DBDagBag(load_op_links=False)
+ if dag_cache_size < 0:
+ self.log.warning("scheduler dag_cache_size must be >= 0, using
unbounded dict")
+ dag_cache_size = 0
+
+ if dag_cache_ttl_config < 0:
+ self.log.warning("scheduler dag_cache_ttl must be >= 0, disabling
TTL")
+ dag_cache_ttl_config = 0
+
+ dag_cache_ttl = dag_cache_ttl_config if dag_cache_ttl_config > 0 else
None
+
+ self.scheduler_dag_bag = DBDagBag(
+ load_op_links=False,
+ cache_size=dag_cache_size,
+ cache_ttl=dag_cache_ttl,
Review Comment:
What about the metrics produced internally by `DBDagBag`? Right now, the
label of the API server is hardcoded but the stats label or prefix should be
provided as part of the constructor too (e.g.
`stats_prefix="scheduler.dag_bag"`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
