On 27 Feb 05:33, Paul van der Linden wrote: > I have currently a pod running in kubernetes exporting some metrics to > prometheus. If we have alerts on them, they will be retriggered (resolved & > trigger shortly after eachother) every time I update the software (as the > pod would have a different name/ip). I came to the conclusing that this is > because of the unique pod and instance labels on these metrics. I have > added a config to drop these labels, but this seems to cause the error " > persist head block: write compaction: add series: out-of-order series added > with label set", while it solves the issue. What is the correct way to > solve this issue.
Is the label set in the error empty? Are you exposing metrics with timestamps? The correct way to deal with this would probably be to ingest with the instance and pod name and improve your alert rule to ignore that at this point. Thanks -- (o- Julien Pivotto //\ Open-Source Consultant V_/_ Inuits - https://www.inuits.eu -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200227134300.GA12264%40oxygen.
signature.asc
Description: PGP signature
