There are multiple labels on those metrics indeed. How can I figure out
what is causing this. Looking at the git changes for the deployment
scripts, the only thing changed was this metric job. I still get a
continuous stream of these in my logs, and they started around the time I
added the label drop:
level=error ts=2020-02-27T14:38:24.235Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=error ts=2020-02-27T14:39:26.332Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=error ts=2020-02-27T14:40:26.960Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=warn ts=2020-02-27T14:40:36.044Z caller=klog.go:86
component=k8s_client_runtime func=Warningf
msg="/app/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints
ended with: too old resource version: 95336324 (95337598)"
level=error ts=2020-02-27T14:41:28.247Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=warn ts=2020-02-27T14:41:34.021Z caller=klog.go:86
component=k8s_client_runtime func=Warningf
msg="/app/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints
ended with: too old resource version: 95336132 (95337864)"
level=warn ts=2020-02-27T14:42:18.027Z caller=klog.go:86
component=k8s_client_runtime func=Warningf
msg="/app/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints
ended with: too old resource version: 95336301 (95338065)"
level=error ts=2020-02-27T14:42:28.877Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=error ts=2020-02-27T14:43:30.582Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=error ts=2020-02-27T14:44:32.665Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=error ts=2020-02-27T14:45:33.427Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=warn ts=2020-02-27T14:46:32.003Z caller=klog.go:86
component=k8s_client_runtime func=Warningf
msg="/app/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints
ended with: too old resource version: 95338072 (95339190)"
level=error ts=2020-02-27T14:46:34.061Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
level=error ts=2020-02-27T14:47:35.562Z caller=db.go:617 component=tsdb
msg="compaction failed" err="persist head block: write compaction: add
series: out-of-order series added with label set \"{}\""
On Thursday, 27 February 2020 15:25:31 UTC+1, Julien Pivotto wrote:
>
> On 27 Feb 06:14, Paul van der Linden wrote:
> > Thanks, completely missed that option. The error doesn't disappear
> though,
> > does it mean this error keeps happening until the ingested metrics with
> the
> > labelsdrop get deleted because of retention?
>
> I don't know what is the cause of the original issue. Does not sound
> like the relabel config you have shown because there would still be a
> 'job' label.
>
> >
> > On Thursday, 27 February 2020 14:51:31 UTC+1, Julien Pivotto wrote:
> > >
> > > On 27 Feb 05:48, Paul van der Linden wrote:
> > > > The label set in the error is empty. There are still a bunch of
> labels
> > > on
> > > > the metric though. The metrics don't have timestamps, just generated
> > > > everytime prometheus scrapes them.
> > > >
> > > > How do I improve the alert rule to ignore these? Looking at the
> docs, is
> > > > the only way to wrap these in 2 label_replace, or otherwise sum and
> > > specify
> > > > every relevant label?
> > >
> > > You can use `without(pod, instance)` as well.
> > >
> > > >
> > > > On Thursday, 27 February 2020 14:43:05 UTC+1, Julien Pivotto wrote:
> > > > >
> > > > > On 27 Feb 05:33, Paul van der Linden wrote:
> > > > > > I have currently a pod running in kubernetes exporting some
> metrics
> > > to
> > > > > > prometheus. If we have alerts on them, they will be retriggered
> > > > > (resolved &
> > > > > > trigger shortly after eachother) every time I update the
> software
> > > (as
> > > > > the
> > > > > > pod would have a different name/ip). I came to the conclusing
> that
> > > this
> > > > > is
> > > > > > because of the unique pod and instance labels on these metrics.
> I
> > > have
> > > > > > added a config to drop these labels, but this seems to cause the
> > > error "
> > > > > > persist head block: write compaction: add series: out-of-order
> > > series
> > > > > added
> > > > > > with label set", while it solves the issue. What is the correct
> way
> > > to
> > > > > > solve this issue.
> > > > >
> > > > > Is the label set in the error empty? Are you exposing metrics with
> > > > > timestamps?
> > > > >
> > > > > The correct way to deal with this would probably be to ingest with
> the
> > > > > instance and pod name and improve your alert rule to ignore that
> at
> > > this
> > > > > point.
> > > > >
> > > > > Thanks
> > > > >
> > > > > --
> > > > > (o- Julien Pivotto
> > > > > //\ Open-Source Consultant
> > > > > V_/_ Inuits - https://www.inuits.eu
> > > > >
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > Groups "Prometheus Users" group.
> > > > To unsubscribe from this group and stop receiving emails from it,
> send
> > > an email to [email protected] <javascript:>.
> > > > To view this discussion on the web visit
> > >
> https://groups.google.com/d/msgid/prometheus-users/54ff4ef8-7154-4517-bcda-8e52411fddea%40googlegroups.com.
>
>
> > >
> > >
> > >
> > > --
> > > (o- Julien Pivotto
> > > //\ Open-Source Consultant
> > > V_/_ Inuits - https://www.inuits.eu
> > >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected] <javascript:>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/28465380-afa9-48ac-94f9-e01ad901ae7a%40googlegroups.com.
>
>
>
>
> --
> (o- Julien Pivotto
> //\ Open-Source Consultant
> V_/_ Inuits - https://www.inuits.eu
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ec686aa7-7f54-4485-9121-507527137b20%40googlegroups.com.
