I found out what the issue was. I had the same set of rules defined in
projects I was scraping. I only need the rules in my main prometheus.
I remove the rules from the projects I was using federate against and the
errors have stopped.
On Monday, February 15, 2021 at 3:45:39 PM UTC-5 Brian Beynon wrote:
> Hello,
> I recently updated our prometheus setup from using the helm chart to using
> the prometheus-operator.
>
> *Summary of current setup:*
> Platform: Google Cloud
> 1. Google Project used for monitoring:
> Prometheus Operator (prometheus,alertmanager,grafana, node-exporter,
> kube-state-metrics...ect.)
>
> 2. Then multiple other Google Projects that now run the
> Prometheus-Operator (node-exporter, kube-state-metrics...ect) but without
> Alertmanager/Grafana.
>
> So the main google project (#1 above) has federate scrapes/jobs that
> connect to each of the other google projects prometheus (#2 above).
>
> Since updating to the prometheus-operators I'm now running into these
> errors coming from the main prometheus logs:
> *msg="Error on ingesting samples with different value but same timestamp"*
> and *msg="Error on ingesting out-of-order samples".*
>
> Below is an example of one of the federate jobs where the errors are
> coming from.
> When I have the job "vms" and job "node-exporter" both enabled the errors
> occur. If I disable either of those jobs I no longer see the errors.
>
> - job_name: 'test-abc-123'
> scrape_interval: 60s
> scrape_timeout: 30s
> honor_labels: true
> metrics_path: '/federate'
> scheme: 'https'
> basic_auth:
> username: '###################'
> password: '###################'
> params:
> 'match[]':
> * - '{job="vms"} '*
> * - '{job="node-exporter"} '*
> - '{job="postgres"} '
> - '{job="barman"} '
> - '{job="apiserver"} '
> - '{job="kube-state-metrics"} '
> static_configs:
> - targets:
> - 'test-abc-123.com'
> labels:
> project: 'test-abc-123'
>
> Here is the node-exporter serviceMonitor from project test-abc-123:
>
> apiVersion: monitoring.coreos.com/v1
> kind: ServiceMonitor
> metadata:
> labels:
> app.kubernetes.io/component: exporter
> app.kubernetes.io/name: node-exporter
> app.kubernetes.io/part-of: kube-prometheus
> app.kubernetes.io/version: 1.1.0
> name: node-exporter
> namespace: monitoring
> spec:
> endpoints:
> - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
> interval: 15s
> port: https
> relabelings:
> - action: replace
> regex: (.*)
> replacement: $1
> sourceLabels:
> - __meta_kubernetes_pod_node_name
> targetLabel: instance
> scheme: https
> tlsConfig:
> insecureSkipVerify: true
> jobLabel: app.kubernetes.io/name
> selector:
> matchLabels:
> app.kubernetes.io/component: exporter
> app.kubernetes.io/name: node-exporter
> app.kubernetes.io/part-of: kube-prometheus
>
> Here is the "vms" job from project test-abc-123:
>
> - job_name: 'vms'
> static_configs:
> - targets: ['db-prod-1:9100','db-prod-2:9100','util-1:9100']
> labels:
> project: 'client-vms'
>
> I have tried updating labels but maybe not in the right way. Any
> suggestions or pointers would be appreciated.
>
> Thank you
>
>
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/26b77e64-2ae5-4364-87a0-8b05bbbe0c19n%40googlegroups.com.
