It's very rare that someone have Prometheus -> remote read -> Thanos, it's rather Thanos sidecar connected to remote read that Prometheus expose.
Rodrigo, can you confirm what situation you have? Kind Regards, Bartek Płotka (@bwplotka) On Tue, 25 Aug 2020 at 12:43, Julien Pivotto <[email protected]> wrote: > > Can we know which prometheus version you are running? > > > On 25 Aug 13:42, Julien Pivotto wrote: > > On 25 Aug 12:12, Bartłomiej Płotka wrote: > > > Hey, > > > > > > Interesting. Can you double-check your rules, what are they asking for? > > > What queries are they making? Sounds like some typo in rule > configuration > > > you have with double asked matchers in the PromQL? Not sure how this is > > > related to Thanos if you see this on Prometheus PromQL log. > > > > > > If thanos in configured as remote read, you can have the errors below > > because we see the metrics with and without the external labels. > > > > > > > > Kind Regards, > > > Bartek Płotka (@bwplotka) > > > > > > > > > On Mon, 24 Aug 2020 at 21:04, Rodrigo Martinez <[email protected]> > wrote: > > > > > > > > > > > I have noticed that when enabling thanos I now see in prometheus > logs of > > > > > > > > Error executing query: found duplicate series for the match group > > > > {namespace="monitoring", pod="kube-state-metrics-567789848b-9d77w"} > on the > > > > right hand-side of the operation: > > > > [{__name__="node_namespace_pod:kube_pod_info:", > namespace="monitoring", > > > > node="test01", pod="kube-state-metrics-567789848b-9d77w"}, > > > > {__name__="node_namespace_pod:kube_pod_info:", > namespace="monitoring", > > > > node="test01", > pod="kube-state-metrics-567789848b-9d77w"}];many-to-many > > > > matching not allowed: matching labels must be unique on one side > > > > > > > > Primarily this is coming from kube-state-metrics job and other > > > > prometheusrule with metrics that have honor_labels set to true > > > > > > > > > > > > > > > > Just wondering overall how enabling thanos causes these errors now > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > > "Prometheus Users" group. > > > > To unsubscribe from this group and stop receiving emails from it, > send an > > > > email to [email protected]. > > > > To view this discussion on the web visit > > > > > https://groups.google.com/d/msgid/prometheus-users/2da7a037-f8f1-4e84-b238-c7e45ae54498n%40googlegroups.com > > > > < > https://groups.google.com/d/msgid/prometheus-users/2da7a037-f8f1-4e84-b238-c7e45ae54498n%40googlegroups.com?utm_medium=email&utm_source=footer > > > > > > . > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAMssQwYw1EaGTRaOdet0wOwHTY7aoyLF6XCwFRVcY6cbiP%3D8KA%40mail.gmail.com > . > > > > -- > > Julien Pivotto > > @roidelapluie > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMssQwZFhtHOg8BZkrKGo0R_aqPytv14WjxYFQ%3DBxM9BSK87hA%40mail.gmail.com.
