This is an automated email from the ASF dual-hosted git repository.
Miretpl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 9b43d6abc0f Document service.name and service.instance.id for
OpenTelemetry metrics (#71564)
9b43d6abc0f is described below
commit 9b43d6abc0fc0766b1b1a886657bb6befc33ee1d
Author: Stefan Wang <[email protected]>
AuthorDate: Wed Aug 19 13:38:37 2026 -0700
Document service.name and service.instance.id for OpenTelemetry metrics
(#71564)
* Document how to separate metrics from replicated Airflow components
Every Airflow process reports the same service.name, so in HA the
schedulers publish the same gauge series and the backend keeps whichever
export landed last. Setting service.instance.id per replica already fixes
this, but nothing in the metrics docs says so.
Signed-off-by: 1fanwang <[email protected]>
* Cover both service.name and service.instance.id, nest under OpenTelemetry
Frame the section around identifying a metric's source rather than around
the
collision that motivated it: service.name says which component reports, and
service.instance.id says which instance of it, so both are needed and each
has
its own environment variable.
Make it a subsection of Setup - OpenTelemetry, since it only applies there,
and
move the StatsD caveat to the StatsD section where a StatsD user will meet
it.
Name the job label alongside instance, and drop the claim that one backend
implements the Prometheus compatibility spec.
Signed-off-by: 1fanwang <[email protected]>
* Describe processes rather than replicas, and state the defaults
Airflow processes running the same component are independent, with no
leader-follower relationship, so calling them replicas describes something
the
deployment does not have. Talk about components and their processes instead.
Say what each attribute defaults to and what that costs: service.name is
airflow for every process, so a scheduler, a triggerer and a worker arrive
under one name, and service.instance.id is unset, so processes running one
component cannot be told apart. Distinguishing components is useful on its
own,
independently of running several of any one of them.
Signed-off-by: 1fanwang <[email protected]>
* Cover scheduler HA in the StatsD note and tighten the wording
Every scheduler emits pool gauges under the same StatsD series, so with
more than one the server keeps whichever value arrived last. Say so in
the StatsD note rather than only pointing at OpenTelemetry.
Signed-off-by: 1fanwang <[email protected]>
* Use the same 'when' phrasing in the StatsD note
Signed-off-by: 1fanwang <[email protected]>
---------
Signed-off-by: 1fanwang <[email protected]>
---
.../logging-monitoring/metrics.rst | 53 ++++++++++++++++++++++
1 file changed, 53 insertions(+)
diff --git
a/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
b/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
index af0fa6e300b..d3b9e2da3d4 100644
---
a/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
+++
b/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
@@ -53,6 +53,14 @@ custom StatsD client. This module must be available on your
:envvar:`PYTHONPATH`
See :doc:`../modules_management` for details on how Python and Airflow manage
modules.
+.. note::
+
+ StatsD has no resource concept, so metrics cannot be attributed to the
process that
+ produced them. When several processes run the same component, such as
schedulers in high
+ availability, each exports the same series and the server keeps whichever
value arrived last.
+ Use OpenTelemetry to tell them apart, as described in
+ :ref:`identifying-components-and-their-instances`.
+
Setup - OpenTelemetry
---------------------
@@ -98,6 +106,51 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two
resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every
Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per
component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by
default, so processes
+ running the same component (e.g. 2+ schedulers) are indistinguishable. Set
it per process
+ to attribute a metric to one of them.
+
+Airflow reads ``service.name`` from ``OTEL_SERVICE_NAME``, and every other
resource attribute from
+``OTEL_RESOURCE_ATTRIBUTES``:
+
+.. code-block:: bash
+
+ # on one of the schedulers
+ export OTEL_SERVICE_NAME="airflow-scheduler"
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Processes that share a resource also share a series, and the backend keeps
whichever export
+arrived last. When several processes run the same component, data are lost
instead of aggregated:
+each scheduler samples the metadata database on its own loop, so a gauge such
as
+``pool.open_slots`` reports an arbitrary scheduler's sample rather than a
value derived from all
+of them.
+
+Once each process is identified, its samples form their own series and can be
combined
+deliberately — for example, the lowest number of open slots any scheduler
observed:
+
+.. code-block:: text
+
+ min by (pool_name) (airflow_pool_open_slots)
+
+How the attributes surface depends on the backend. Those implementing the
OpenTelemetry
+`Prometheus compatibility
<https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/>`_
+spec expose them as the ``job`` and ``instance`` labels.
+
+
Enable Https
-----------------
