frank-ye opened a new pull request, #71800:
URL: https://github.com/apache/airflow/pull/71800
Re-initialising the OTel metrics pipeline inside a long-lived process leaves
the previous
pipeline running and exporting.
#64703 resets the SDK's `Once()` guard so that `set_meter_provider()`
genuinely succeeds on a
second call. That fixed metrics being silently dropped in forked children
(#64690). What it does
not do is shut down the provider it replaces.
The outgoing provider owns a `PeriodicExportingMetricReader`, and that
reader's constructor has
already started a daemon thread which exports on its own interval. The
provider is built with
`shutdown_on_exit=False`, so nothing ever reaps it. The abandoned reader
therefore keeps
exporting, but its instruments are no longer recorded to, so its cumulative
totals freeze. It
republishes those frozen totals indefinitely, under the same resource, with
a different
`start_time_unix_nano`.
A consumer then receives two conflicting cumulative streams for one series:
one climbing
correctly, one stuck at whatever the value was when the pipeline was
replaced. Backends that must
collapse them to a single series alternate between the two, which reads as a
large spurious rate
spike; backends that reject out-of-order samples can drop the counter
entirely.
This is invisible under delta temporality, because the abandoned reader
honestly reports "nothing
new" forever. It only surfaces once counters are reported as cumulative.
### Reproduction
Against `opentelemetry-sdk==1.42.1`, mimicking `get_otel_logger()`: build a
provider, record 48 to
a counter, reset the `Once()` guard, install a second provider, record 1000
to the counter.
Before:
```
A-first-init batches= 9 last_value= 48
start_time=1787084407400194000
B-second-init batches= 5 last_value= 1000
start_time=1787084408103536000
A exports occurring AFTER B came online: 5
same metric name: True different start_time: True
live reader threads: ['OtelPeriodicExportingMetricReader',
'OtelPeriodicExportingMetricReader']
```
After:
```
A exports occurring AFTER B came online: 0
live reader threads: ['OtelPeriodicExportingMetricReader']
```
The new test in `test_otel_logger.py` asserts the process does not
accumulate exporter threads
across a re-initialisation. It fails without this change
(`leaked_readers=1`) and passes with it.
### Fix
Shut down the outgoing provider before installing the replacement.
`MeterProvider.shutdown()`
force-flushes first, so the final datapoints are not lost, which keeps the
guarantee #64703 was
written to restore.
---
- [x] Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst)
- [x] Tests added
(`shared/observability/tests/observability/metrics/test_otel_logger.py`)
- [x] Newsfragment added
**Was generative AI tooling used to co-author this PR?**
- [x] Yes (please specify the tool below)
Claude Code (Opus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
