One possibility is to use a recording rule for the expensive repeated query.
If I
rewrite avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval])
to X, I get:
((50 - 0) / (12 - 0) * ((X <= 12) - 0) + 0) or
((100 - 51) / (35.4 - 12.1) * ((X > 12 and X <= 35.4) - 12.1) + 51) or
((150 - 101) / (55.4 - 35.5) * ((X > 35.4 and X <= 55.4) - 35.5) + 101) or
((200 - 151) / (150.4 - 55.5) * ((X > 55.4 and X <= 150.4) - 55.5) + 151) or
((300 - 201) / (250.4 - 150.5) * ((X > 150.4 and X <= 250.4) - 150.5) +
201) or
((400 - 301) / (350.4 - 250.5) * ((X > 250.4 and X <= 350.4) - 250.5) +
301) or
((500 - 401) / (500.4 - 350.5) * ((X > 350.4 and X <= 500.4) - 350.5) +
401) or
clamp_max(X, 600)
I guess you're trying to apply different scaling for different ranges of X:
- if X is between 0 and 12 (or negative) then rescale to 0 to 50
- if X is between 12 and 35.4 then rescale to 50(?) to 100
- if X is between 35.4 and 55.4 then rescale to 100(?) to 150
etc (except there seem to be some small discontinuities at the boundaries,
e.g. 12 versus 12.1, 50 versus 51)
"A or B" will suppress elements in the B vector where the A vector has a
value (i.e. with a matching label set). That means it's unnecessary to test
the lower bounds, and I think your expression could simplify to something
like this:
(X <= 12) * k1 + o1 or
(X <= 35.4) * k2 + o2 or
(X <= 55.4) * k3 + o3 or
(X <= 150.4) * k4 + o4 or
(X <= 250.4) * k5 + o5 or
(X <= 350.4) * k6 + o6 or
(X <= 500.4) * k7 + o7 or
clamp_max(X, 600)
That would roughly halve the number of the subexpressions X.
On Monday, 18 September 2023 at 07:01:50 UTC+1 Ben Kochie wrote:
> One thing you can do to speed things up is to eliminate the `=~` in your
> query. Using regexp matching means it has to do a string search over every
> instance in your Prometheus for each metric. Using exact matching (`=`)
> will speed things up a lot. Although you won't be able to do multiple
> matching if you want that in your dashboard variables.
>
> On Mon, Sep 18, 2023 at 7:58 AM David Leibovic <david.l...@gmail.com>
> wrote:
>
>> Hi there, I'm trying to optimize a slow query of this form:
>>
>> (1 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 10) or
>> (2 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 20) or
>> (3 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 30) or
>> (10 * avg_over_time(foo{instance=~"$i"}[$interval]))
>>
>> I suspect it's slow because of the many duplicate calls to
>> avg_over_time(foo{instance=~"$i"}[$interval])
>>
>> Is there some way to only call the avg_over_time function once and
>> re-use the results subsequently? I'm using Prometheus in conjunction with
>> Grafana, in case it's relevant.
>>
>> The full query I'm trying to optimize is much more complicated, but I
>> figured the above would be enough to understand the problem. But in case
>> it's helpful, here is the full query I am trying to optimize (it's an Air
>> Quality Index computation):
>>
>> ((50 - 0) / (12 - 0) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 12) - 0) + 0) or
>> ((100 - 51) / (35.4 - 12.1) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 12
>> and avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 35.4) - 12.1) + 51) or
>> ((150 - 101) / (55.4 - 35.5) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>> 35.4 and
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 55.4) - 35.5) + 101) or
>> ((200 - 151) / (150.4 - 55.5) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>> 55.4 and
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 150.4) - 55.5) + 151) or
>> ((300 - 201) / (250.4 - 150.5) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>> 150.4 and
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 250.4) - 150.5) + 201) or
>> ((400 - 301) / (350.4 - 250.5) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>> 250.4 and
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 350.4) - 250.5) + 301) or
>> ((500 - 401) / (500.4 - 350.5) *
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
>> 350.4 and
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
>> 500.4) - 350.5) + 401) or
>> clamp_max(avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]),
>>
>> 600)
>>
>> Thanks for any help you can provide!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/664689b3-9f45-4b05-9438-4225e2dce773n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/664689b3-9f45-4b05-9438-4225e2dce773n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/29177e06-f10a-4da3-beab-d437fa5ca89dn%40googlegroups.com.
