Adding a constant label to every timeseries should have almost zero impact
on memory usage.
Can you clarify what you're saying, and how you've come to your diagnosis?
What version of prometheus are you running? When you say "backends" in the
plural, how have you set this up?
At one point you seem to be saying it's something to do with ingestion, but
then you seem to be saying it's something to do with queries (*"Without
this filter, the queries run reasonably well"*). Can you give specific
examples of filters which show the difference in behaviour?
Again: the queries
my_series{global_label="constant-value", l1="..", l2=".."}
my_series{l1="..", l2=".."}
should perform almost identically, as they will select the same subset of
timeseries.
On Wednesday, 5 April 2023 at 17:42:33 UTC+1 Johny wrote:
> There is a performance related issue we're facing in Prometheus coming
> from a label with a constant value across all (thousands of) time series.
> The label filter in query causes a large quantity of metadata to load in
> memory overwhelming Prometheus backends. Without this filter, the queries
> run reasonably well. We are planning to exclude this label in ingestion in
> future, but for now we need a workaround.
>
> my_series{*global_label="constant-value"*, l1="..", l2=".."}
>
> Is there a mechanism to automatically exclude global_label in query
> configuration: remote_read subsection, or elsewhere?
>
> thanks,
> Johny
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ca99f732-2785-4149-950a-cc5d479b3863n%40googlegroups.com.
