Hi My log file is full with messages like this
level=warn ts=2021-05-14T05:23:49.382Z caller=scrape.go:1372 component="scrape manager" scrape_pool=federate target="https://prometheus.qna-prod-in.k8s.example.org:9090/federate?match%5B%5D=%7Bjob%3D%22ambassador-admin%22%7D&match%5B%5D=%7Bjob%3D%22kubelet%22%7D&match%5B%5D=%7Bjob%3D%22kube-state-metrics%22%7D&match%5B%5D=%7Bjob%3D%22node-exporter%22%7D"; msg="Error on ingesting out-of-order samples" num_dropped=1714
full log fle:https://privatebin.at/?010661c7e7e29590#AdjCnhUs9zrHQmYbfWcXPrCmAV9ofpkvDTgeY5JtfwSV (warning: takes some time to load because 6.2 MB)
federation config:
- job_name: federate
scrape_interval: 30s
scrape_timeout: 30s
honor_labels: true
metrics_path: /federate
scheme: https
tls_config:
insecure_skip_verify: true
params:
match[]:
- '{job="ambassador-admin"}'
- '{job="kubelet"}'
- '{job="kube-state-metrics"}'
- '{job="node-exporter"}'
file_sd_configs:
- files:
- DC-43-east.yml
relabel_configs:
- source_labels:
- __address__
regex: (.*)
replacement: ${1}:9090
target_label: __address__
When I curl manually the federate endpoint I get 54 MB of metrics.
What you advise me to fix "Error on ingesting out-of-order"?
It's only a warning in log.
best regards
Evelyn
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9bb19fbf-cebd-8156-dfe9-38da27f4de01%40disroot.org.
OpenPGP_0x61776FA8E38403FB.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
