Have you given each of your "local" prometheus servers unique labels, using
the global external_labels
<https://prometheus.io/docs/prometheus/latest/configuration> setting
(recommended), or some other way? This is to ensure all timeseries have a
unique label set.
On Friday, 4 February 2022 at 13:19:22 UTC Bogdan L wrote:
> Hi,
>
> I have a situation where I have a few "local" Prometheus servers sending
> data to a "global" server using the remote write API. I get errors that
> look like this on the remote write receiver:
>
> ts=2022-02-03T12:41:11.244Z caller=write_handler.go:57 level=error
> component=web msg="Out of order sample from remote write" err="duplicate
> sample for timestamp"
>
> The senders get the same error from the receiver, with a 400 HTML code.
>
> After much trial and error I figured out that it happens because I have
> the same recording rules on all servers, on both senders and receiver.
> recording-rules.yaml looks like this:
> ```
> groups:
> - name: node-exporter
> rules:
> # CPU cores per node
> - record: instance:node_cpus:count
> expr: count(node_cpu_seconds_total{mode="idle"}) without (cpu,mode)
>
> # CPU in use by CPU
> - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
> expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) without
> (mode)
> ```
>
> However, if I delete the second rule, the errors are gone. So if I change
> recording-rules.yaml on all servers to:
> ```
> groups:
> - name: node-exporter
> rules:
> # CPU cores per node
> - record: instance:node_cpus:count
> expr: count(node_cpu_seconds_total{mode="idle"}) without (cpu,mode)
> ```
>
> Why?
>
> 1. Why are there duplicates in the first case, does the remote write
> receiver also run the rules when it receives data?
> 2. Why aren't there errors any more when the only rule is the CPU count?
> Shouldn't there be duplicates in that case too?
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/600dd093-c5cc-4003-9fa0-5e531f6667ban%40googlegroups.com.
