Thanks Ben! I will learn more about recording rules and give this a shot.
On Friday, May 1, 2020 at 1:53:57 AM UTC-7, Ben Kochie wrote:
>
> The other way to solve this is to use recording rules to pre-summarize the
> data.
>
> For example:
>
> groups:
> - name: User Egress
> interval: 5m
> - record: category_user:user_egress_total:increase5m
> expr: sum by (category, user)
> (increase(user_egress_total{job="test"}[5m]))
>
> With this, you can now summarize with fewer samples over longer periods of
> time.
>
> sum_over_time(category_user:user_egress_total:increase5m{user=~"$user",
> category=~"$category"}[$__range])
>
> On Fri, May 1, 2020 at 10:16 AM O <[email protected] <javascript:>>
> wrote:
>
>> Thanks for your responses. I am using grok exporter for parsing logs to
>> convert them into prometheus metrics.
>> grok exporter service won't restart very often but it could still happen.
>>
>> Here's the query that I am using to calculate the egress data by
>> different users in the selected time range in Grafana:
>> sum by (category, user)(increase(user_egress_total{job="test",
>> user=~"$user", category=~"$category"}[$__range]))
>>
>> Cardinality is too high for the metric, so I end up getting ~1200
>> timeseries. After applying the step and changing [$__range] to
>> [$__range:1m], I was able to make it work for ~15 days.
>> I understand that high cardinality metrics are not recommended for
>> Prometheus. But, I am wondering if there is a better way of implementing it
>> in Prometheus either using a different exporter or by rewriting the query.
>>
>> Appreciate your inputs. Thanks!
>>
>>
>> On Thursday, April 30, 2020 at 11:51:43 PM UTC-7, Brian Brazil wrote:
>>>
>>> On Fri, 1 May 2020 at 07:46, Christian Hoffmann <
>>> [email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> On 5/1/20 3:57 AM, O wrote:
>>>> > I am using increase() function to calculate the increase in counter
>>>> > value over a time period. These values are being displayed in a table
>>>> in
>>>> > Grafana. But, for a duration of 15 days or so, it errors out because
>>>> the
>>>> > number of samples that are being pulled is too high and the limit
>>>> > for |--query.max-samples| flag is crossed.
>>>> >
>>>> > So, my question is if there is a better way to calculate the increase
>>>> in
>>>> > counter and display it in the Grafana table without pulling so many
>>>> > labels from Prometheus.
>>>>
>>>> increase() tries to detect counter resets. In order for this to work,
>>>> each data point has to be considered (at least I assume that this is the
>>>> case). I don't see a a way around this.
>>>>
>>>
>>> You're correct.
>>>
>>>
>>>>
>>>> If you know for sure that your counter does not reset (at least in the
>>>> timeframe you are interested in), you might achieve what you want by a
>>>> simple substraction which should be less resource-intensive:
>>>>
>>>> your_metric - your_metric offset 14d
>>>>
>>>> Of course, you can also increase the max-samples value. It is primarily
>>>> there as a safeguard against high resource usage (i.e. you might need
>>>> more RAM and longer processing times).
>>>>
>>>
>>> Even with 15d of data at a 1s interval, that's only 1.3M samples that
>>> need to be in memory at a time to calculate the rate() - so it's not the
>>> rate() function that's the issue here.
>>>
>>> --
>>> Brian Brazil
>>> www.robustperception.io
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/4d076cee-9ac9-4b0d-8247-83c2bceb0ff3%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/4d076cee-9ac9-4b0d-8247-83c2bceb0ff3%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/98884034-59f2-4efa-b469-291b306a0c88%40googlegroups.com.
