FYI, we have work in progress to start supporting the OpenMetrics "created timestamp" feature. This will allow Prometheus to know when a counter started at 0 so it can do appropriate rate/increase calculations for newly created series.
On Tue, Jul 4, 2023 at 6:19 PM Brian Candler <[email protected]> wrote: > It's working as designed. > > Firstly, note that increase() is actually the same as rate() multiplied by > the time window size. It can give you non-integer results. For example, > consider that rate(foo[5m]) takes the rate between the first and last data > points within that window; if you are scraping at 1 minute intervals then > you will have first and last points which are 4 minutes apart, so it > calculates a per-second rate over that 4 minute period. If you use > increase() instead of rate() then this rate is multiplied by 300 to > estimate the amount it would have increased over 5 minutes. > > To get an exact difference between the value now and the value 5 minutes > ago, then you can use "foo - foo offset 5m". But if the counter ever > resets, then you will get nonsense values (huge negative spikes). rate() > and increase() skip over those counter drops and calculate a rate using the > remaining data points. > > Also, "foo - foo offset 5m" will give you no results if the time series > did not exist 5 minutes ago (or it had existed, but there's a gap of more > than 5 minutes to the previous data point: this is the default "lookback > delta" window) > > You could do something like this: > > (foo - foo offset 5m) or foo > > which will give you the current value of foo if there's no previous value > to take a difference against. This takes care of the case where the counter > doesn't have an initial zero value. But again: it will give you nonsense > results if there's ever a gap in the timeseries, or if the counter resets. > > You can guard against negative spikes with a filter: > > ((foo - foo offset 5m) > 0) or foo > > This might be good enough for what you need. It will still give you > nonsense values if there are *gaps* in the timeseries, e.g. due to scrape > failures. But more seriously, it will fail if the counter didn't start at > zero. > > In general, you can't assume that counters start at zero. The counter may > have been running and incrementing for a long time, before Prometheus > starts scraping it; the first value you see could represent days or years > of accumulation. That's why rate() and increase() only give a value if > there are two or more adjacent data points. It's the only way to be sure > that the value has actually increased over that time period. And if you > don't generate an initial timeseries value of zero, then it can only use > the next two values to calculate a rate between those two values. > > On Tuesday, 4 July 2023 at 16:39:21 UTC+1 Johan Jourdan wrote: > >> Hello, >> >> I have different prometheus counter per Objects. >> I want to create a Stacked Bar Chart over the time, an “Acitvity per Day >> Panel”. >> A counter is incremented each time an object is used on the back end side. >> >> My data looks like : >> object_usage_count_total{object="tc1"} 2.0 object_usage_count_total{ >> object="tc5"} 3.0 >> >> I'm using the following query : >> >> increase(object_usage_count_total[5m]) >> (I’m using 5m for test purpose even if my goal is 1d) >> >> I have the following result for the moment : >> [image: image] >> >> Which is pretty nice, it looks like what I want, except that the first >> usage of an object is never shown. >> For this example I used 3 time TC5 and 2 time TC1. And as we can see >> there’s a total of 2 activity for TC5 and 1 activity for TC1. >> >> [image: image] >> I thought of initializing the counter to 0 before, but the problem is >> that a new object can be created at any moment, and when it’s created, it >> is automatically used once. The state “used 0 time” doesn’t really exist. >> >> I have 2 questions: >> >> Is it considered as a bug or is the behavior of “increase” like this on >> purpose ? >> >> Do you have a solution that might work in my case ? >> >> Thanks >> >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f13f9992-f3d8-47d6-9562-a8de595fc0b3n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/f13f9992-f3d8-47d6-9562-a8de595fc0b3n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpDr60ncZ-yQGoTG3Y1EWMZ8yiU4iX%3D%3DyoxoOEaQhgjvQ%40mail.gmail.com.
