There are external_labels, yes. "instance" is also unique, there is no overlap
> On 4 Feb 2022, at 17:28, Brian Candler <[email protected]> wrote: > > Have you given each of your "local" prometheus servers unique labels, using > the global external_labels setting (recommended), or some other way? This is > to ensure all timeseries have a unique label set. > >> On Friday, 4 February 2022 at 13:19:22 UTC Bogdan L wrote: >> Hi, >> >> I have a situation where I have a few "local" Prometheus servers sending >> data to a "global" server using the remote write API. I get errors that look >> like this on the remote write receiver: >> >> ts=2022-02-03T12:41:11.244Z caller=write_handler.go:57 level=error >> component=web msg="Out of order sample from remote write" err="duplicate >> sample for timestamp" >> >> The senders get the same error from the receiver, with a 400 HTML code. >> >> After much trial and error I figured out that it happens because I have the >> same recording rules on all servers, on both senders and receiver. >> recording-rules.yaml looks like this: >> ``` >> groups: >> - name: node-exporter >> rules: >> # CPU cores per node >> - record: instance:node_cpus:count >> expr: count(node_cpu_seconds_total{mode="idle"}) without (cpu,mode) >> >> # CPU in use by CPU >> - record: instance_cpu:node_cpu_seconds_not_idle:rate5m >> expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) without >> (mode) >> ``` >> >> However, if I delete the second rule, the errors are gone. So if I change >> recording-rules.yaml on all servers to: >> ``` >> groups: >> - name: node-exporter >> rules: >> # CPU cores per node >> - record: instance:node_cpus:count >> expr: count(node_cpu_seconds_total{mode="idle"}) without (cpu,mode) >> ``` >> >> Why? >> >> 1. Why are there duplicates in the first case, does the remote write >> receiver also run the rules when it receives data? >> 2. Why aren't there errors any more when the only rule is the CPU count? >> Shouldn't there be duplicates in that case too? > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Prometheus Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/prometheus-users/vRTNtIlbdV8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/600dd093-c5cc-4003-9fa0-5e531f6667ban%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CD6163EE-3C5E-4CDF-A597-A22459502414%40neant.ro.
