The label set in the error is empty. There are still a bunch of labels on the metric though. The metrics don't have timestamps, just generated everytime prometheus scrapes them.
How do I improve the alert rule to ignore these? Looking at the docs, is the only way to wrap these in 2 label_replace, or otherwise sum and specify every relevant label? On Thursday, 27 February 2020 14:43:05 UTC+1, Julien Pivotto wrote: > > On 27 Feb 05:33, Paul van der Linden wrote: > > I have currently a pod running in kubernetes exporting some metrics to > > prometheus. If we have alerts on them, they will be retriggered > (resolved & > > trigger shortly after eachother) every time I update the software (as > the > > pod would have a different name/ip). I came to the conclusing that this > is > > because of the unique pod and instance labels on these metrics. I have > > added a config to drop these labels, but this seems to cause the error " > > persist head block: write compaction: add series: out-of-order series > added > > with label set", while it solves the issue. What is the correct way to > > solve this issue. > > Is the label set in the error empty? Are you exposing metrics with > timestamps? > > The correct way to deal with this would probably be to ingest with the > instance and pod name and improve your alert rule to ignore that at this > point. > > Thanks > > -- > (o- Julien Pivotto > //\ Open-Source Consultant > V_/_ Inuits - https://www.inuits.eu > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/54ff4ef8-7154-4517-bcda-8e52411fddea%40googlegroups.com.
