Prometheus documentation <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config> does include this phrase:
> Care must be taken with labeldrop and labelkeep to ensure that metrics are still uniquely labeled once the labels are removed. Also note that PromLabs is an independent company, albeit run by a Prometheus founder. Bryan On Tuesday, 21 January 2025 at 18:09:00 UTC Mike W wrote: > That makes a lot of sense now that you describe it that way. I was > thinking about implementing a solution for this that involves recording > rules like your links suggest, but was concerned we'd run into resource > consumption issues when making these constant calculations. We were able to > get the underlying metric fixed before I was able to try. > > It might be useful for others if a note was added in documentation that > stated metric relabeling is not helpful to solve high cardinality issues > due to the deduplication/collision. Could be a good fit here > https://training.promlabs.com/training/relabeling/introduction-to-relabeling/relabeling-overview/ > > . > > Thank you very much for your answer. > On Tuesday, January 21, 2025 at 10:12:50 AM UTC-5 Bjoern Rabenstein wrote: > >> On 17.01.25 08:49, Mike W wrote: >> > >> > Background: We have a metric named "order_total" that has a very high >> > cardinality label "store_number" slowing down our queries and causing >> > massive resource usage that we need to drop during/before ingestion. >> >> In general, you cannot reduce the cardinality by just removing the >> label or by changing the label value to the same. The many series will >> not disappear by that, they will just collide, so that their values >> will all be written into the same series in the TSDB, creating a >> nonsensical series with possibly many duplicate timestamp errors, too. >> >> What you are looking for is a feature to actually aggregate a set of >> high cardinality series into fewer series, but vanilla Prometheus does >> not have such a feature (yet). It is offered by some of the cloud >> vendors ingesting Prometheus metrics. >> >> For ideas and work in Prometheus itself, see the following: >> - https://github.com/prometheus/prometheus/issues/394 >> - https://github.com/prometheus/prometheus/pull/10529 >> - >> https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit?tab=t.0#bookmark=id.hs3vj63cj7uk >> >> >> >> -- >> Björn Rabenstein >> [PGP-ID] 0x851C3DA17D748D03 >> [email] bjo...@rabenste.in >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/9aa40de8-de5e-4e2a-8761-aaa20f397588n%40googlegroups.com.
