My hypothesis is that this is a bug in Prometheus and it's only changing the metric label for a single "store_number", and dropping the rest of the metrics. On Friday, January 17, 2025 at 11:49:49 AM UTC-5 Mike W wrote:
> We may be running into a bug with metrics relabeling, but I wanted to post > here in case anyone has an idea of what my team and I are doing wrong here > with this relabeling. > > Background: We have a metric named "order_total" that has a very high > cardinality label "store_number" slowing down our queries and causing > massive resource usage that we need to drop during/before ingestion. > > We have tried multiple methods and variations to do this, (*1*) by the > labeldrop action- > Everything I'm seeing online and in the Prometheus documentation > <https://training.promlabs.com/training/relabeling/writing-relabeling-rules/keeping-and-dropping-labels/> > says > just to add the following to the scrape config target/job and it should > work. > And (*2*) by relabeling this to a single value for all ("0" in our case), > just to remove the high cardinality aspect. > > *1*: (green line in screenshot) > metric_relabel_configs: > - action: labeldrop > regex: store_number > > *2: *(blue line in screenshot {store_number="0"}) > metric_relabel_configs: > - source_labels: [store_number] > separator: ; > regex: (.*) > target_label: store_number > replacement: "0" > > But as you can see, the numbers aren't nearly close to what we are > expecting. This is during a simulated load test that is sending a constant > rate of orders, hence the reason why the counts are at a constant increase. > I am expecting to see both blue and green lines at exponentially higher > values than the unique store_number labels. If this isn't a bug, does > anyone have a clue what we are doing wrong? Any replies are greatly > appreciated. > > [image: Screenshot 2025-01-16 at 8.43.55 PM.png] > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/e1909353-137c-4d57-86a3-1fe258ae61e7n%40googlegroups.com.
