Also, we don't use collectDefaultMetrics. Maybe that is related? E.g. maybe the 'off by one' is swallowed by default metrics that start accumulating when the server starts and in our case it is noticed because the first metrics we have are at the application level?
On Wednesday, February 28, 2024 at 1:37:30 PM UTC+2 Ittay Dror wrote: > 1. That is the only data, there were no metrics before. Also, if looking > the data as a chart I can see the time in which it was indexed. > 2. The data is always off by one, and we see metrics change when there's > activity in the intended cluster. > > So the targets are expected and scraped recently. Note this happened in > two environments (of the same software, test & main) > > I also think this is extremely bizarre. Note in particular that we see > metric values of "0" being scraped, but only after the counter is actually > "1". So when the system just started and is idle, no value, then on the > first increment, the value is "0", but only on the prometheus server side, > the client reports correctly. > > Maybe there's a configuration that rebases values, or assigns a type that > causes things to be treated differently, or maybe the first scrape is > expected to get 0 as value and doesn't, or something like that? > > On Monday, February 26, 2024 at 10:51:22 PM UTC+2 Fabian Stäber wrote: > >> Thanks Ittay. >> >> Hmmm, I can only imagine two reasons: >> >> 1. Prometheus hasn't scraped yet, so you are seeing some older values >> than if you look directly at /metrics. >> 2. Prometheus scrapes another target than you are looking at. Maybe >> something with service discovery is going wrong. >> >> Could you use the Prometheus UI on port 9090 to check that the target is >> as expected, and when the last scrape to your target happened? >> >> Fabian >> >> On Mon, Feb 26, 2024 at 4:08 PM Ittay Dror <ittay...@gmail.com> wrote: >> >>> It is literaly just 'my_total' >>> >>> On Monday, February 26, 2024 at 4:13:21 PM UTC+2 Fabian Stäber wrote: >>> >>>> Hi Ittay, >>>> >>>> Please post the PromQL query you are using. >>>> >>>> Fabian >>>> >>>> On Mon, Feb 26, 2024 at 3:03 PM Ittay Dror <ittay...@gmail.com> wrote: >>>> >>>>> I have a counter with a type label. The type can be one of 4 values. >>>>> Using prom-client (15.0.0) in a node (express) app with Typescript. The >>>>> code only does 'inc' in the counter. The counter is initialized only with >>>>> name, help and labelNames configuration. >>>>> >>>>> If I fetch /metrics directly from the pod I see correct values. Say >>>>> type "A" has 1, type "B" has 2, "C" 2 and "D" 1. The sum is 6 >>>>> >>>>> But if I issue a promQL, the results are off by one. That is, the >>>>> values are 0,1,1,0 respectively. The sum is 2. >>>>> >>>>> This happens across several counters. >>>>> >>>>> Using GCE for the (managed) prometheus server. >>>>> >>>>> There are no restarts of the pod and using 'increase' doesn't help. >>>>> >>>>> What is the reason and what is the approach to solve this? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to prometheus-use...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/0383adcd-5878-4887-9273-552874fe9f11n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/prometheus-users/0383adcd-5878-4887-9273-552874fe9f11n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/8e09d780-d94a-4a8c-b7c1-8cbdd14c8af3n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/8e09d780-d94a-4a8c-b7c1-8cbdd14c8af3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1ba5110c-89bd-4dcb-8d8c-d3a2262ccb53n%40googlegroups.com.
