It's working as designed.
Firstly, note that increase() is actually the same as rate() multiplied by
the time window size. It can give you non-integer results. For example,
consider that rate(foo[5m]) takes the rate between the first and last data
points within that window; if you are scraping at 1 minute intervals then
you will have first and last points which are 4 minutes apart, so it
calculates a per-second rate over that 4 minute period. If you use
increase() instead of rate() then this rate is multiplied by 300 to
estimate the amount it would have increased over 5 minutes.
To get an exact difference between the value now and the value 5 minutes
ago, then you can use "foo - foo offset 5m". But if the counter ever
resets, then you will get nonsense values (huge negative spikes). rate()
and increase() skip over those counter drops and calculate a rate using the
remaining data points.
Also, "foo - foo offset 5m" will give you no results if the time series did
not exist 5 minutes ago (or it had existed, but there's a gap of more than
5 minutes to the previous data point: this is the default "lookback delta"
window)
You could do something like this:
(foo - foo offset 5m) or foo
which will give you the current value of foo if there's no previous value
to take a difference against. This takes care of the case where the counter
doesn't have an initial zero value. But again: it will give you nonsense
results if there's ever a gap in the timeseries, or if the counter resets.
You can guard against negative spikes with a filter:
((foo - foo offset 5m) > 0) or foo
This might be good enough for what you need. It will still give you
nonsense values if there are *gaps* in the timeseries, e.g. due to scrape
failures. But more seriously, it will fail if the counter didn't start at
zero.
In general, you can't assume that counters start at zero. The counter may
have been running and incrementing for a long time, before Prometheus
starts scraping it; the first value you see could represent days or years
of accumulation. That's why rate() and increase() only give a value if
there are two or more adjacent data points. It's the only way to be sure
that the value has actually increased over that time period. And if you
don't generate an initial timeseries value of zero, then it can only use
the next two values to calculate a rate between those two values.
On Tuesday, 4 July 2023 at 16:39:21 UTC+1 Johan Jourdan wrote:
> Hello,
>
> I have different prometheus counter per Objects.
> I want to create a Stacked Bar Chart over the time, an “Acitvity per Day
> Panel”.
> A counter is incremented each time an object is used on the back end side.
>
> My data looks like :
> object_usage_count_total{object="tc1"} 2.0 object_usage_count_total{object
> ="tc5"} 3.0
>
> I'm using the following query :
>
> increase(object_usage_count_total[5m])
> (I’m using 5m for test purpose even if my goal is 1d)
>
> I have the following result for the moment :
> [image: image]
>
> Which is pretty nice, it looks like what I want, except that the first
> usage of an object is never shown.
> For this example I used 3 time TC5 and 2 time TC1. And as we can see
> there’s a total of 2 activity for TC5 and 1 activity for TC1.
>
> [image: image]
> I thought of initializing the counter to 0 before, but the problem is that
> a new object can be created at any moment, and when it’s created, it is
> automatically used once. The state “used 0 time” doesn’t really exist.
>
> I have 2 questions:
>
> Is it considered as a bug or is the behavior of “increase” like this on
> purpose ?
>
> Do you have a solution that might work in my case ?
>
> Thanks
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/f13f9992-f3d8-47d6-9562-a8de595fc0b3n%40googlegroups.com.
