dodjdnh opened a new pull request, #3765:
URL: https://github.com/apache/dubbo-go/pull/3765

   ## Summary
   
   Change RPC QPS, aggregate window counts, and latency quantiles (P50 / P90 / 
P95 / P99) to **dynamic evaluation during Prometheus collection**, fixing the 
continued export of stale values for these three window metric categories in 
the built-in Prometheus Registry after traffic stops, while preserving the 
existing metric contract and cumulative Counter behavior.
   
   ## Motivation
   
   Previously, the relevant metric vectors calculated values in place while 
recording RPC requests and wrote them to static Gauges. Once traffic stopped 
and no new records arrived, each Prometheus scrape still received the 
previously cached values even after the underlying time window had fully 
expired (changing Grafana dashboard queries cannot eliminate residual data 
inside the SDK).
   
   ## Changes
   
   1. **Dynamic collection callbacks**: When a window is first created for a 
label set, register collection callbacks for QPS, window counts, and 
P50/P90/P95/P99. Subsequent records update only the sliding window rather than 
writing static snapshots to Gauges.
   2. **Optional extension interface**: Add the optional `GaugeFuncRegistry` 
extension, implemented by the built-in Prometheus Registry, to avoid a breaking 
change from adding mandatory methods to the original `MetricRegistry`.
   3. **Reuse the export pipeline**: Evaluate dynamic metrics through the 
existing `Gatherer`. The HTTP `/metrics` endpoint, local `Scrape()`, and 
Pushgateway pushes share the same logic; ordinary Gauges and Counters keep 
their existing pipelines.
   4. **Consistent QPS snapshots and empty-window protection**: The numerator 
count and denominator (valid bucket duration) use exactly the same timestamp 
and window snapshot. Return 0 when the denominator is empty, preserving 
existing expiration and denominator rules.
   5. **Aligned empty-window semantics**: When a window expires, QPS and counts 
export 0. Quantiles retain the `NaN` produced by an empty TDigest, neither 
reporting old latency nor falsely reporting 0ms latency.
   6. **TDigest concurrency protection**: TDigest's `Centroids()` compresses 
and mutates internal state. Protect merge operations with an exclusive lock to 
prevent conflicts between concurrent collection and recording (the quantile 
algorithm itself is unchanged).
   7. **Label isolation**: Copy label values into an independent slice during 
registration and use unambiguous series keys to prevent collisions between 
label sets or caller mutations of maps after registration from changing 
existing series identities.
   8. **Automated regression suite**: Add regression tests for Provider / 
Consumer window expiration, recording resumption after idle periods, repeated 
collection, preservation of cumulative values, application/method isolation, 
quantile correctness, export paths, and concurrency safety.
   
   ## Verification
   
   | **Area** | **Check** | **Result** |
   | --- | --- | --- |
   | **Reproduction / Regression** | Run the new idle-expiration case against 
the old pre-fix code: check whether Provider/Consumer counts become zero after 
traffic stops | **REPRODUCED**: The old code retains a count of 6 (expected 0), 
failing as expected before the fix |
   | **Unit / Metric-package regression** | `go test ./metrics/... -count=1` | 
**PASS** |
   | **Unit / Window expiration** | Virtual-time tests for window expiration, 
traffic resumption, repeated collection, different quantile values, and label 
isolation | **PASS** |
   | **Export integration** | Test `Scrape()`, a real `promhttp` Handler, and a 
local HTTP server receiving Pushgateway requests | **PASS** (local integration 
tests, not deployed-cluster acceptance) |
   | **Compatibility regression** | Verify ordinary Gauges, Counters, and 
legacy `MetricRegistry` implementations without the new interface | **PASS** |
   | **Concurrency** | Focused `-race` tests under WSL Ubuntu, Go 1.25.0, and 
native x86-64 GCC (three consecutive runs) | **PASS** (no race warnings) |
   | **Static** | `go vet ./metrics/...` | **PASS** |
   | **Formatting** | Run `gofmt -l` on all nine changed Go files | **PASS** 
(no unformatted files reported) |
   | **Diff** | Run `git diff --check` and review staged changes before 
committing | **PASS** |
   | **Repository-wide CI** | Upstream `make fmt`, `make test`, and fully 
configured lint / CI checks | **NOT VERIFIED**: Repository-wide CI was not run 
locally |
   | **Runtime end-to-end** | Real Dubbo RPC → external Prometheus scraping / 
querying → Admin / Grafana | **NOT VERIFIED**: Outside the scope of this proof |
   | **Browser verification** | Admin / Grafana interactive UI checks | **NOT 
VERIFIED**: No browser verification performed |
   
   ### Mapping to Issue Acceptance Criteria
   
   | Issue acceptance criterion | Verification evidence |
   | --- | --- |
   | 1: QPS / window counts become zero after idle-window expiration | Provider 
/ Consumer virtual-time expiration tests; pre-fix comparison reproducing a 
residual count of 6 |
   | 2: Empty-window and quantile semantics | Regression tests for empty-window 
`NaN` and different quantile values |
   | 3: Resumption and repeated collection | Tests for recording resumption, 
repeated scraping, and preservation of cumulative counts |
   | 4: Label isolation and concurrency safety | Application / method isolation 
tests and focused `-race` checks |
   | 5: Actual exported results | `Scrape()`, `promhttp` Handler, and local 
Pushgateway request-receiving tests; not external-cluster E2E |
   | 6: Existing metric behavior and contract | Ordinary Gauge / Counter 
regression tests; diff review of names, labels, types, units, and window rules |
   | 7: Custom Registry compatibility | Legacy `MetricRegistry` implementation 
regression tests; capability boundary described below |
   | 8: Relevant checks and verification boundaries | Metric-package tests, `go 
vet`, formatting, and diff checks pass; repository-wide CI / external E2E not 
verified |
   
   _Note: Window-expiration tests use `testing/synctest`, supported as of Go 
1.25, to advance virtual time without blocking tests for minutes while waiting 
for real window expiration._
   
   **Focused race-test command already executed**:
   
   ```sh
   go test -p 2 -race ./metrics ./metrics/prometheus ./metrics/util/aggregate \
     -run 
'TestLegacyWindowMetricRegistry|TestWindowMetrics|TestWindowQuantileValues|TestGaugeFunc|TestTimeWindowCounterQPS|TestDefault(QpsMetricVec|AggregateCounterVec|QuantileMetricVec)'
 \
     -count=3
   ```
   
   **Other reproducible check commands**:
   
   ```sh
   go test ./metrics/... -count=1
   go vet ./metrics/...
   git diff --check 5d892b0a7fd4fb5e6679112a18eccbf8a6eabab2 HEAD
   ```
   
   ## Review Focus
   
   1. **Collection-time evaluation semantics**: Whether expired windows no 
longer depend on new requests; whether empty-window `NaN` is strictly 
distinguished from zero latency; whether cumulative Counters remain unaffected 
by idle periods.
   2. **Compatibility boundary**: `GaugeFuncRegistry` is a non-intrusive 
optional extension. Third-party registries without it retain source 
compatibility and fall back to record-time refresh (do not assume that all 
third-party registries support dynamic evaluation).
   3. **Callback ownership and label isolation**: Ensure each label set retains 
its own callback on first registration, label values are copied, and callbacks 
execute outside the Collector's series lock.
   4. **TDigest locking**: Since `Centroids()` internally performs mutating 
compression, merging uses the exclusive write lock `Lock()` on `sync.RWMutex`, 
rather than the shared read lock `RLock()`. The digest merged for each 
computation must be a local, independent object.
   5. **Computation cost and existing rules**: The QPS denominator and bucket 
rules remain unchanged. Quantile computation moves to scrape time, with each 
quantile evaluated separately. This PR does not include performance benchmarks 
for extremely large label cardinality or label-eviction policies.
   
   ## Compatibility / Impact
   
   - **Metric contract**: Metric names, label sets, types, units, default 
window duration, and bucket expiration rules remain unchanged; no new exported 
RPC metric families or labels are added.
   - **Behavior change**: After data expires, idle series automatically export 
QPS / window counts of `0`, and quantiles become `NaN`. The series remain 
registered and are not unregistered.
   - **Independence**: Ordinary Gauge and cumulative Counter logic remains 
unchanged; the `processing` Gauge is unchanged, and metric-event loss is not 
addressed.
   - **No background goroutines**: No background timer is introduced. Pull and 
push share the `Gatherer`, but an external Pushgateway still caches the 
previously pushed values until the next push.
   - **Deployment dependencies**: No new dashboard queries or scrape 
configuration are required. Applications must be rebuilt with an SDK containing 
this fix and redeployed to gain the new behavior.
   - **Build environment**: The module's declared Go version is unchanged. 
Focused race tests pass on Go 1.25.0, but this is not a claim that 
repository-wide CI has passed.
   
   ## Scope
   
   ### Included
   
   - Collection-time refresh of QPS, window counts, and latency quantiles in 
the built-in Prometheus implementation.
   - QPS empty-window protection, concurrency-safe quantile merging, and export 
/ isolation / compatibility regression tests.
   
   ### Not Included
   
   - Fixing dropped metric events / `processing` drift, or sentinel-value logic 
for aggregate RT extrema in empty windows.
   - Refactoring the sliding-window model, changing QPS denominator rules, 
label cleanup/eviction, or dynamic metric configuration.
   - Changes to Dubbo Admin, the Java SDK, or Grafana templates, or real 
end-to-end browser integration.
   
   ## Related Issue
   
   Fixes #3764
   
   This PR does not close the overarching OSPP Issue in the Admin repository 
across repositories.
   
   ## Checklist
   
   - [x] Confirm the target branch is `develop`.
   - [ ] Run `make fmt` to format the code.
   - [ ] Run `make test` for local tests.
   - [x] Add tests demonstrating the fix.
   
   _(Note: Unchecked items mean the corresponding repository-wide commands have 
not been executed locally in this iteration, not that failures were found. 
File-level `gofmt`, complete metric-package tests, `go vet`, and focused race 
checks have all passed.)_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to