I have currently a pod running in kubernetes exporting some metrics to prometheus. If we have alerts on them, they will be retriggered (resolved & trigger shortly after eachother) every time I update the software (as the pod would have a different name/ip). I came to the conclusing that this is because of the unique pod and instance labels on these metrics. I have added a config to drop these labels, but this seems to cause the error " persist head block: write compaction: add series: out-of-order series added with label set", while it solves the issue. What is the correct way to solve this issue.
-- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/95b5e421-8238-4e26-a361-51bd4a973a4c%40googlegroups.com.
