For future googlers, this is the reason: https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#counter-sums
Monarch, which is Google's implementation ignores the first reported value. So if the first value it scrapes is 5, then all future values will be off by 5. As far as I could see this is per label. On Monday, February 26, 2024 at 4:03:11 PM UTC+2 Ittay Dror wrote: > I have a counter with a type label. The type can be one of 4 values. Using > prom-client (15.0.0) in a node (express) app with Typescript. The code only > does 'inc' in the counter. The counter is initialized only with name, help > and labelNames configuration. > > If I fetch /metrics directly from the pod I see correct values. Say type > "A" has 1, type "B" has 2, "C" 2 and "D" 1. The sum is 6 > > But if I issue a promQL, the results are off by one. That is, the values > are 0,1,1,0 respectively. The sum is 2. > > This happens across several counters. > > Using GCE for the (managed) prometheus server. > > There are no restarts of the pod and using 'increase' doesn't help. > > What is the reason and what is the approach to solve this? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/58a3b451-885c-43d7-aa85-522763343ed3n%40googlegroups.com.
