dstandish opened a new issue, #72885:
URL: https://github.com/apache/airflow/issues/72885
### Description
`airflow.sdk.observability.stats.Stats` (and the OTel-backed stats logger
under the hood) only
exposes synchronous, push-style metric recording: `Stats.gauge(name, value)`
records whatever
value the caller passes *at the moment it's called*. There's no way to
register a callback that
the metrics SDK invokes on its own export/collection cadence.
This is a real gap for "current state" gauges — e.g. a thread-pool occupancy
count, a
connection-pool checkout count, or a boolean saturation flag. With only a
synchronous API,
producers have to hand-roll a polling loop (a `while` loop on a timer that
reads current state
and calls `Stats.gauge(...)`) to get periodic samples. That has two failure
modes in practice:
1. If the value is only pushed on a state *transition* (e.g. "just became
saturated") rather than
on a fixed interval, the metric goes silent for as long as the state
doesn't change — which
most gauge-consuming backends (Prometheus/OTel scrape-based pipelines
included) treat the same
as "no data," defeating the point of an alertable gauge.
2. Even when done correctly with a polling loop, every producer
re-implements the same
"read current value on a timer, catch exceptions so a metrics failure
doesn't crash the loop"
boilerplate that the OTel spec's `ObservableGauge` instrument (a callback
registered once,
invoked by the SDK itself at export time) already solves generically.
OpenTelemetry's Metrics API has supported asynchronous/observable instruments
(`ObservableCounter`, `ObservableUpDownCounter`, `ObservableGauge`) for a
long time. Since
Airflow's OTel-backed stats path is already built on the OTel SDK, it seems
feasible to expose
an equivalent on the `Stats` façade — e.g. `Stats.gauge(name, callback=...)`
or a dedicated
`Stats.observable_gauge(name, callback)` registration API — so callers with
a canonical "current
value" getter don't need to own their own timer loop, and the emitted
samples are guaranteed
fresh on every collection cycle rather than only on edges.
### Use case (concrete example)
Surfaced from an internal package building on Airflow's `Stats` interface
for scheduling-latency
observability: a saturation flag ("are we at `max_active_dagruns` right
now?") is currently only
loggable, not gauge-able, without a hand-rolled polling loop, precisely
because the only available
primitive is the synchronous push `Stats.gauge()`.
### Use case
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
