dodjdnh opened a new pull request, #3765:
URL: https://github.com/apache/dubbo-go/pull/3765
## Summary
Change RPC QPS, aggregate window counts, and latency quantiles (P50 / P90 /
P95 / P99) to **dynamic evaluation during Prometheus collection**, fixing the
continued export of stale values for these three window metric categories in
the built-in Prometheus Registry after traffic stops, while preserving the
existing metric contract and cumulative Counter behavior.
## Motivation
Previously, the relevant metric vectors calculated values in place while
recording RPC requests and wrote them to static Gauges. Once traffic stopped
and no new records arrived, each Prometheus scrape still received the
previously cached values even after the underlying time window had fully
expired (changing Grafana dashboard queries cannot eliminate residual data
inside the SDK).
## Changes
1. **Dynamic collection callbacks**: When a window is first created for a
label set, register collection callbacks for QPS, window counts, and
P50/P90/P95/P99. Subsequent records update only the sliding window rather than
writing static snapshots to Gauges.
2. **Optional extension interface**: Add the optional `GaugeFuncRegistry`
extension, implemented by the built-in Prometheus Registry, to avoid a breaking
change from adding mandatory methods to the original `MetricRegistry`.
3. **Reuse the export pipeline**: Evaluate dynamic metrics through the
existing `Gatherer`. The HTTP `/metrics` endpoint, local `Scrape()`, and
Pushgateway pushes share the same logic; ordinary Gauges and Counters keep
their existing pipelines.
4. **Consistent QPS snapshots and empty-window protection**: The numerator
count and denominator (valid bucket duration) use exactly the same timestamp
and window snapshot. Return 0 when the denominator is empty, preserving
existing expiration and denominator rules.
5. **Aligned empty-window semantics**: When a window expires, QPS and counts
export 0. Quantiles retain the `NaN` produced by an empty TDigest, neither
reporting old latency nor falsely reporting 0ms latency.
6. **TDigest concurrency protection**: TDigest's `Centroids()` compresses
and mutates internal state. Protect merge operations with an exclusive lock to
prevent conflicts between concurrent collection and recording (the quantile
algorithm itself is unchanged).
7. **Label isolation**: Copy label values into an independent slice during
registration and use unambiguous series keys to prevent collisions between
label sets or caller mutations of maps after registration from changing
existing series identities.
8. **Automated regression suite**: Add regression tests for Provider /
Consumer window expiration, recording resumption after idle periods, repeated
collection, preservation of cumulative values, application/method isolation,
quantile correctness, export paths, and concurrency safety.
## Verification
| **Area** | **Check** | **Result** |
| --- | --- | --- |
| **Reproduction / Regression** | Run the new idle-expiration case against
the old pre-fix code: check whether Provider/Consumer counts become zero after
traffic stops | **REPRODUCED**: The old code retains a count of 6 (expected 0),
failing as expected before the fix |
| **Unit / Metric-package regression** | `go test ./metrics/... -count=1` |
**PASS** |
| **Unit / Window expiration** | Virtual-time tests for window expiration,
traffic resumption, repeated collection, different quantile values, and label
isolation | **PASS** |
| **Export integration** | Test `Scrape()`, a real `promhttp` Handler, and a
local HTTP server receiving Pushgateway requests | **PASS** (local integration
tests, not deployed-cluster acceptance) |
| **Compatibility regression** | Verify ordinary Gauges, Counters, and
legacy `MetricRegistry` implementations without the new interface | **PASS** |
| **Concurrency** | Focused `-race` tests under WSL Ubuntu, Go 1.25.0, and
native x86-64 GCC (three consecutive runs) | **PASS** (no race warnings) |
| **Static** | `go vet ./metrics/...` | **PASS** |
| **Formatting** | Run `gofmt -l` on all nine changed Go files | **PASS**
(no unformatted files reported) |
| **Diff** | Run `git diff --check` and review staged changes before
committing | **PASS** |
| **Repository-wide CI** | Upstream `make fmt`, `make test`, and fully
configured lint / CI checks | **NOT VERIFIED**: Repository-wide CI was not run
locally |
| **Runtime end-to-end** | Real Dubbo RPC → external Prometheus scraping /
querying → Admin / Grafana | **NOT VERIFIED**: Outside the scope of this proof |
| **Browser verification** | Admin / Grafana interactive UI checks | **NOT
VERIFIED**: No browser verification performed |
### Mapping to Issue Acceptance Criteria
| Issue acceptance criterion | Verification evidence |
| --- | --- |
| 1: QPS / window counts become zero after idle-window expiration | Provider
/ Consumer virtual-time expiration tests; pre-fix comparison reproducing a
residual count of 6 |
| 2: Empty-window and quantile semantics | Regression tests for empty-window
`NaN` and different quantile values |
| 3: Resumption and repeated collection | Tests for recording resumption,
repeated scraping, and preservation of cumulative counts |
| 4: Label isolation and concurrency safety | Application / method isolation
tests and focused `-race` checks |
| 5: Actual exported results | `Scrape()`, `promhttp` Handler, and local
Pushgateway request-receiving tests; not external-cluster E2E |
| 6: Existing metric behavior and contract | Ordinary Gauge / Counter
regression tests; diff review of names, labels, types, units, and window rules |
| 7: Custom Registry compatibility | Legacy `MetricRegistry` implementation
regression tests; capability boundary described below |
| 8: Relevant checks and verification boundaries | Metric-package tests, `go
vet`, formatting, and diff checks pass; repository-wide CI / external E2E not
verified |
_Note: Window-expiration tests use `testing/synctest`, supported as of Go
1.25, to advance virtual time without blocking tests for minutes while waiting
for real window expiration._
**Focused race-test command already executed**:
```sh
go test -p 2 -race ./metrics ./metrics/prometheus ./metrics/util/aggregate \
-run
'TestLegacyWindowMetricRegistry|TestWindowMetrics|TestWindowQuantileValues|TestGaugeFunc|TestTimeWindowCounterQPS|TestDefault(QpsMetricVec|AggregateCounterVec|QuantileMetricVec)'
\
-count=3
```
**Other reproducible check commands**:
```sh
go test ./metrics/... -count=1
go vet ./metrics/...
git diff --check 5d892b0a7fd4fb5e6679112a18eccbf8a6eabab2 HEAD
```
## Review Focus
1. **Collection-time evaluation semantics**: Whether expired windows no
longer depend on new requests; whether empty-window `NaN` is strictly
distinguished from zero latency; whether cumulative Counters remain unaffected
by idle periods.
2. **Compatibility boundary**: `GaugeFuncRegistry` is a non-intrusive
optional extension. Third-party registries without it retain source
compatibility and fall back to record-time refresh (do not assume that all
third-party registries support dynamic evaluation).
3. **Callback ownership and label isolation**: Ensure each label set retains
its own callback on first registration, label values are copied, and callbacks
execute outside the Collector's series lock.
4. **TDigest locking**: Since `Centroids()` internally performs mutating
compression, merging uses the exclusive write lock `Lock()` on `sync.RWMutex`,
rather than the shared read lock `RLock()`. The digest merged for each
computation must be a local, independent object.
5. **Computation cost and existing rules**: The QPS denominator and bucket
rules remain unchanged. Quantile computation moves to scrape time, with each
quantile evaluated separately. This PR does not include performance benchmarks
for extremely large label cardinality or label-eviction policies.
## Compatibility / Impact
- **Metric contract**: Metric names, label sets, types, units, default
window duration, and bucket expiration rules remain unchanged; no new exported
RPC metric families or labels are added.
- **Behavior change**: After data expires, idle series automatically export
QPS / window counts of `0`, and quantiles become `NaN`. The series remain
registered and are not unregistered.
- **Independence**: Ordinary Gauge and cumulative Counter logic remains
unchanged; the `processing` Gauge is unchanged, and metric-event loss is not
addressed.
- **No background goroutines**: No background timer is introduced. Pull and
push share the `Gatherer`, but an external Pushgateway still caches the
previously pushed values until the next push.
- **Deployment dependencies**: No new dashboard queries or scrape
configuration are required. Applications must be rebuilt with an SDK containing
this fix and redeployed to gain the new behavior.
- **Build environment**: The module's declared Go version is unchanged.
Focused race tests pass on Go 1.25.0, but this is not a claim that
repository-wide CI has passed.
## Scope
### Included
- Collection-time refresh of QPS, window counts, and latency quantiles in
the built-in Prometheus implementation.
- QPS empty-window protection, concurrency-safe quantile merging, and export
/ isolation / compatibility regression tests.
### Not Included
- Fixing dropped metric events / `processing` drift, or sentinel-value logic
for aggregate RT extrema in empty windows.
- Refactoring the sliding-window model, changing QPS denominator rules,
label cleanup/eviction, or dynamic metric configuration.
- Changes to Dubbo Admin, the Java SDK, or Grafana templates, or real
end-to-end browser integration.
## Related Issue
Fixes #3764
This PR does not close the overarching OSPP Issue in the Admin repository
across repositories.
## Checklist
- [x] Confirm the target branch is `develop`.
- [ ] Run `make fmt` to format the code.
- [ ] Run `make test` for local tests.
- [x] Add tests demonstrating the fix.
_(Note: Unchecked items mean the corresponding repository-wide commands have
not been executed locally in this iteration, not that failures were found.
File-level `gofmt`, complete metric-package tests, `go vet`, and focused race
checks have all passed.)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]