dstandish opened a new pull request, #71855:
URL: https://github.com/apache/airflow/pull/71855
The OTel metrics pipeline can end up with two writers on one cumulative
series — one climbing,
one frozen — in two independent ways. This fixes both.
**A second `get_otel_logger()` call in the same process leaves the first
pipeline running.** Every
`MeterProvider` owns a `PeriodicExportingMetricReader` whose constructor
starts a daemon export
thread, and `shutdown_on_exit=False` means nothing reaps one that gets
replaced. Its instruments
stop being recorded to, so it republishes frozen totals under a different
`start_time_unix_nano`
for the life of the process. The scheduler reaches this on every start:
`BaseExecutor.__init__` and
`SchedulerJobRunner._execute` each call `stats.initialize()`, with a
`stats.incr("schedulerjob_start")` in between that materialises the first
pipeline. Fixed by
building the pipeline once per process, so there is no second one to reap
and no cumulative reset
partway through startup.
**A forked child re-exports the pipeline it inherited.** `fork()` copies the
parent's provider, and
the SDK registers `register_at_fork(after_in_child=...)` for every
`PeriodicExportingMetricReader`,
so the child restarts the export thread behind it. Nothing in the child
records to that pipeline,
so it republishes the totals the parent held at the instant of the fork,
once per export interval,
for as long as the child lives. Airflow forks constantly and the long-lived
children make it
permanent: LocalExecutor pool workers, the OpenLineage dag-state-change
`ProcessPoolExecutor`, and
the scheduler's log and health-check servers all fork from a scheduler whose
pipeline is already
live. Stopping the reader is not enough on its own — the revived ticker does
one final collect on
its way out — so the collect callback is dropped too.
A `MeterProvider` the SDK built for the deployment, from `OTEL_CONFIG_FILE`
or an
`opentelemetry-instrument` agent, reaches the child the same way, carrying
the atexit shutdown that
`shutdown_on_exit=True` registered for it. Its readers are left running,
since on that path they
are the only pipeline the child has, but the inherited copy of that hook is
dropped so the child
cannot dump the parent's state on the way out. Only a provider's own
`_metric_readers` are ever
stopped, never the class-level `_all_metric_readers` every provider shares,
so an agent's pipeline
keeps working in forked children.
related: #71800 — that PR fixes the first defect above by shutting down the
replaced provider
instead of never building a second one. The two are alternatives for that
half; shutting down and
replacing force-flushes the old pipeline and restarts the series under a new
`start_time_unix_nano`, which building once avoids.
supersedes: #71804, which carried an earlier version of this work.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5), for this description and the verification
runs
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
