ferruzzi commented on code in PR #53722:
URL: https://github.com/apache/airflow/pull/53722#discussion_r2495605286
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2108,18 +2136,43 @@ def _emit_pool_metrics(self, session: Session =
NEW_SESSION) -> None:
with DebugTrace.start_span(span_name="emit_pool_metrics",
component="SchedulerJobRunner") as span:
pools = Pool.slots_stats(session=session)
for pool_name, slot_stats in pools.items():
- Stats.gauge(f"pool.open_slots.{pool_name}", slot_stats["open"])
- Stats.gauge(f"pool.queued_slots.{pool_name}",
slot_stats["queued"])
- Stats.gauge(f"pool.running_slots.{pool_name}",
slot_stats["running"])
- Stats.gauge(f"pool.deferred_slots.{pool_name}",
slot_stats["deferred"])
- Stats.gauge(f"pool.scheduled_slots.{pool_name}",
slot_stats["scheduled"])
-
- # Same metrics with tagging
- Stats.gauge("pool.open_slots", slot_stats["open"],
tags={"pool_name": pool_name})
- Stats.gauge("pool.queued_slots", slot_stats["queued"],
tags={"pool_name": pool_name})
- Stats.gauge("pool.running_slots", slot_stats["running"],
tags={"pool_name": pool_name})
- Stats.gauge("pool.deferred_slots", slot_stats["deferred"],
tags={"pool_name": pool_name})
- Stats.gauge("pool.scheduled_slots", slot_stats["scheduled"],
tags={"pool_name": pool_name})
+ # If enabled on the config, publish metrics twice,
+ # once with backward compatible name, and then with tags.
+ DualStatsManager.gauge(
+ f"pool.open_slots.{pool_name}",
+ "pool.open_slots",
+ slot_stats["open"],
+ tags={},
+ extra_tags={"pool_name": pool_name},
+ )
Review Comment:
Sorry I missed you previous comment from last week. The way you mentioned
you were going to try is exactly my thought as well.
> The failing tests on the CI have to do with the new validations. It seems
that some metrics didn't exist in the old docs
That's exactly what I was hoping to catch and fix with the registry idea.
There needs to be a check the other way as well; just because it is in the docs
doesn't mean it still exists in the code... but we can do that part later if
you want. That might just be an initial manual cleanup plus a pre-commit hook
that checks the commit for changes, but that may be a side project on its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
