dodjdnh opened a new issue, #3764:
URL: https://github.com/apache/dubbo-go/issues/3764

   ## Motivation / Problem
   
   The built-in Prometheus exporter in dubbo-go continues to export the QPS, 
aggregate window count, and latency quantiles computed when the last request 
was recorded, even after application traffic stops and all observations in the 
sliding window expire. Even when the Prometheus server scrapes normally, the 
exported metrics do not reflect the actual state of the current window.
   
   As a result, idle applications still appear to have request load on 
monitoring dashboards, and latency panels retain historical P50 / P90 / P95 / 
P99 observations indefinitely. This is a data-refresh defect on the SDK 
producer side and cannot be resolved solely by adjusting Dubbo Admin or Grafana 
queries.
   
   ## Current State
   
   In the verified baseline, `DefaultQpsMetricVec`, 
`DefaultAggregateCounterVec`, and `DefaultQuantileMetricVec` calculate window 
data in place only when `Record()` or `Inc()` is triggered and write the 
snapshot to a static Gauge. Once traffic stops, no new records trigger 
recalculation, so each Prometheus scrape can only read the last values written.
   
   Although the underlying sliding window already excludes expired buckets when 
actively read, the exported Gauge does not trigger a fresh window read during 
scraping.
   
   | **Metric** | **Actual behavior after all observations expire** | 
**Expected behavior** |
   | --- | --- | --- |
   | **Provider / Consumer QPS** | Retains the nonzero value from the last 
recording indefinitely | Automatically refreshes to `0` |
   | **Window request count (`*_aggregate`)** | Retains the window count from 
the last recording indefinitely | Automatically refreshes to `0` |
   | **Window latency (P50 / P90 / P95 / P99)** | Retains the latency value 
from the last recording indefinitely | Returns `NaN` (explicitly indicating no 
samples in the window) |
   | **Cumulative request Counter** | Retains the historical cumulative value | 
**Unchanged** (must never reset to zero because traffic stops) |
   
   _Note: This issue concerns only the three sliding-window metric categories 
above. It does not imply that all latency metrics use this export mechanism. 
Aggregate RT minimum, maximum, and average use a separate Collector._
   
   ## Desired Behavior
   
   Window metrics exported by the built-in Prometheus implementation should 
reflect the current valid window rather than the historical snapshot from the 
last request. Even without new RPC calls, the next collection after all samples 
expire should yield QPS `0`, window count `0`, and latency quantiles `NaN`.
   
   After traffic resumes, only samples still within the valid window should be 
counted. Repeated collection should neither create new observations nor 
increase cumulative counts, and different label series should remain isolated. 
HTTP scraping, local `Scrape()`, and the next Pushgateway push should have 
consistent window semantics.
   
   Preserve the existing metric contract, window rules, and source 
compatibility for third-party registries. The fix should support concurrent 
recording and collection and explicitly describe the capability boundary of 
custom registries, without claiming that all implementations automatically gain 
idle-window refresh.
   
   ## Scope
   
   ### Included
   
   - Refreshing Provider / Consumer QPS, window request counts, and latency 
quantiles in the built-in Prometheus implementation.
   - Correctness for empty windows, resumption after idle periods, repeated 
collection, label isolation, and concurrency safety.
   - Verification of exported results through HTTP `/metrics`, local 
`Scrape()`, and Pushgateway pushes.
   - Compatibility regression coverage for cumulative Counters, ordinary 
Gauges, and third-party registries.
   
   ### Not Included
   
   - Fixing dropped metric events or drift in the `processing` Gauge.
   - Fixing sentinel-value logic for aggregate RT extrema (Min / Max) in empty 
windows.
   - Redesigning the QPS denominator algorithm, bucket boundaries, or 
window-duration configuration.
   - Dynamic label eviction, cardinality cleanup, persistent background refresh 
timers, or performance benchmarks.
   - Changes to Dubbo Admin, the Java SDK, Grafana dashboards, Kubernetes, or 
Prometheus server scrape rules.
   - Browser-based end-to-end acceptance from real production RPCs to external 
Prometheus and Admin / Grafana.
   
   ## Acceptance Criteria
   
   1. After all samples expire for registered Provider / Consumer series, the 
next collection returns QPS and window counts of `0` without new records.
   2. P50 / P90 / P95 / P99 are all `NaN` in empty windows; with valid samples, 
each reflects its corresponding quantile.
   3. After recording resumes, window metrics reflect only valid samples; 
repeated collection neither creates extra observations nor increments 
cumulative counts.
   4. Label series for different applications and methods remain isolated, and 
focused race tests for concurrent recording and collection pass.
   5. Actual exported results are verified through HTTP `/metrics`, `Scrape()`, 
and Pushgateway pushes and follow the expiration and resumption semantics above.
   6. Cumulative Counters retain their values while idle and increment normally 
after traffic resumes; ordinary Gauge behavior and existing metric names, 
labels, types, units, and window rules remain unchanged.
   7. Custom implementations of the original `MetricRegistry` still compile and 
run, with an explicit description of which implementations support idle-window 
refresh.
   8. Relevant metric-package unit tests and static checks pass; whether 
repository-wide CI and external end-to-end verification were performed must be 
stated separately, rather than substituting package-level tests for them.
   
   ## Context / Design Notes
   
   - This is a data-quality issue in the dubbo-go SDK metric export layer. It 
relates to Admin query compatibility, but the fix does not depend on pending 
Admin PRs.
   - The default window is 120 seconds with 10 buckets. Sample expiration 
follows existing bucket rules rather than simply the time of the last request. 
This change does not redefine the window or QPS denominator algorithm.
   - Pushgateway stores the most recently pushed snapshot. Correct SDK exports 
after traffic stops do not mean that an external gateway refreshes itself 
without a subsequent push.
   - See the corresponding PR for the collection mechanism, optional interface, 
and concurrency protection; this Issue does not prescribe an implementation.
   
   ### Minimal Reproduction
   
   This example verifies SDK export behavior directly using the built-in metric 
vectors, without a registry center or RPC service:
   
   ```go
   package main
   
   import (
        "fmt"
        "time"
   
        prom "github.com/prometheus/client_golang/prometheus"
   
        "dubbo.apache.org/dubbo-go/v3/common"
        "dubbo.apache.org/dubbo-go/v3/metrics"
        dubboProm "dubbo.apache.org/dubbo-go/v3/metrics/prometheus"
   )
   
   func main() {
        registry := dubboProm.NewPromMetricRegistry(prom.NewRegistry(), 
common.NewURLWithOptions())
        qps := metrics.NewQpsMetricVec(registry, 
metrics.NewMetricKey("dubbo_provider_qps_total", "Window QPS"))
        count := metrics.NewAggregateCounterVec(registry, 
metrics.NewMetricKey("dubbo_provider_requests_total_aggregate", "Window request 
count"))
        p99 := metrics.NewQuantileMetricVec(registry,
                
[]*metrics.MetricKey{metrics.NewMetricKey("dubbo_provider_rt_milliseconds_p99", 
"Window P99")},
                []float64{0.99})
   
        labels := map[string]string{
                "application_name": "idle-window-repro",
                "interface":        "DemoService",
                "method":           "Call",
        }
   
        // Simulate recording six consecutive requests (100ms each), then 
scrape once.
        for range 6 {
                qps.Record(labels)
                count.Inc(labels)
                p99.Record(labels, 100)
        }
        printScrape(registry)
   
        // Stop recording and wait 125 seconds to cover expiration of the 
default window.
        time.Sleep(125 * time.Second)
        printScrape(registry)
   }
   
   func printScrape(registry interface{ Scrape() (string, error) }) {
        text, err := registry.Scrape()
        if err != nil {
                panic(err)
        }
        fmt.Print(text)
   }
   ```
   
   - **Execution comparison**:
   
     - **Pre-fix baseline**: The second scrape still retains the historical 
QPS, window count, and P99 values from the initial recording.
     - **Expected correct result**: On the second scrape, QPS is `0`, window 
count is `0`, and P99 is `NaN`.
   
   ### Environment and Version Boundaries
   
   - **Code baseline**: `apache/dubbo-go`, commit 
`5d892b0a7fd4fb5e6679112a18eccbf8a6eabab2` (other older versions may exhibit 
the same behavior).
   - **Component configuration**: Built-in Prometheus Registry, default 
120-second window / 10 buckets.
   - **Local verification environment**: Go 1.26.5 on Windows; focused race 
analysis used WSL Ubuntu, Go 1.25.0, and native x86-64 GCC.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to