Is this proxying to more than one backend? And in dev, there is only one? /MR
On Tue, Nov 17, 2020 at 2:07 PM bruno bourdolle <[email protected]> wrote: > hi, > I localise the part of the conf that generate the error, but I'm not sure > to understand how to correct it. What is strange is that the same conf on > dev env works without these warns > > - job_name: 'my-proxy' > metrics_path: '/metrics' > scheme: https > tls_config: > ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > insecure_skip_verify: true > bearer_token_file: /var/run/secrets/ > kubernetes.io/serviceaccount/token > static_configs: > - targets: > - '${SCRAPE_CONFIGS_TARGET}' > > on target , I see > > > https://userprom.xxxt-intraxxx:443/metrics > <https://userprom.kermit-eu-b.itn.intraorange:443/metrics> > > up > group="production" instance="userprom.xxx.intraxxx:443" job="my-proxy" > > on dev no warns > on prod a lot of warns > > if I remove this block, I works without warns on prod , others conf are ok > but If I had this one, I have warns on prod > > Any idea to correct and explain why, I'm a newbie > > Le lundi 16 novembre 2020 à 16:48:25 UTC+1, [email protected] a > écrit : > >> This can happen in a few ways: >> >> 1. whatever exports the metrics, does so with a timestamp, but actually >> changes the value on you between scrapes without updating the timestamp. >> This is relatively unlikely unless this is something very specialized. >> 1.1 or it actually exposes the same metric twice within one /metrics >> response – but the official client libraries actively prevent that >> 2. (more likely) after all relabeling, you end up with metrics from >> multiple targets without any distinguishing labels between them. >> >> Are you scraping through some kind of proxy? In general, Prometheus >> expects to discover and access each target individually, so that it can >> separately collect the data from all of them. >> >> Look at the raw metrics endpoints that Prometheus would scrape – if they >> do not have duplicates or timestamps, it must be 2. Remove label drop or >> replacement rules that may accidentally coalesce multiple targets into >> having the same labels, especially if you are messing with the "instance" >> label. Looking at Prometheus' target page may also help in identifying >> targets that have the exact same label set. >> >> /MR >> >> >> >> On Mon, Nov 16, 2020 at 2:35 PM bruno bourdolle <[email protected]> >> wrote: >> >>> hello, >>> >>> I don(t understand how to correct the WARN about error on ingesting >>> samples... >>> I'm into kubernetes >>> I upgrade at last version of prometheus >>> I cleared the datas >>> Each time I start I have this line of warning each second >>> >>> Any idea ? >>> >>> bets >>> bruno >>> >>> level=info ts=2020-11-16T14:29:36.045Z caller=main.go:353 msg="Starting >>> Prometheus" version="(version=2.22.1, branch=HEAD, >>> revision=00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)" >>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:358 >>> build_context="(go=go1.15.3, user=root@516b109b1732, >>> date=20201105-14:02:25)" >>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:359 >>> host_details="(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 >>> UTC 2018 x86_64 prometheus-0 (none))" >>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:360 >>> fd_limits="(soft=1048576, hard=1048576)" >>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:361 >>> vm_limits="(soft=unlimited, hard=unlimited)" >>> level=info ts=2020-11-16T14:29:36.050Z caller=web.go:516 component=web >>> msg="Start listening for connections" address=0.0.0.0:9090 >>> level=info ts=2020-11-16T14:29:36.050Z caller=main.go:712 msg="Starting >>> TSDB ..." >>> level=info ts=2020-11-16T14:29:36.058Z caller=head.go:642 component=tsdb >>> msg="Replaying on-disk memory mappable chunks if any" >>> level=info ts=2020-11-16T14:29:36.093Z caller=head.go:656 component=tsdb >>> msg="On-disk memory mappable chunks replay completed" duration=35.773378ms >>> level=info ts=2020-11-16T14:29:36.093Z caller=head.go:662 component=tsdb >>> msg="Replaying WAL, this may take a while" >>> level=info ts=2020-11-16T14:29:37.850Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=0 maxSegment=17 >>> level=info ts=2020-11-16T14:29:38.244Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=1 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=2 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=3 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=4 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=5 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=6 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=7 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=8 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=9 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=10 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=11 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=12 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=13 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=14 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=15 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.842Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=16 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.842Z caller=head.go:714 component=tsdb >>> msg="WAL segment loaded" segment=17 maxSegment=17 >>> level=info ts=2020-11-16T14:29:39.842Z caller=head.go:719 component=tsdb >>> msg="WAL replay completed" checkpoint_replay_duration=97.404µs >>> wal_replay_duration=3.748284098s total_replay_duration=3.78422202s >>> level=info ts=2020-11-16T14:29:39.944Z caller=main.go:732 >>> fs_type=XFS_SUPER_MAGIC >>> level=info ts=2020-11-16T14:29:39.944Z caller=main.go:735 msg="TSDB >>> started" >>> level=info ts=2020-11-16T14:29:39.944Z caller=main.go:861 msg="Loading >>> configuration file" filename=/etc/prometheus/prometheus.yml >>> level=info ts=2020-11-16T14:29:39.945Z caller=kubernetes.go:263 >>> component="discovery manager scrape" discovery=kubernetes msg="Using pod >>> service account via in-cluster config" >>> level=info ts=2020-11-16T14:29:39.947Z caller=kubernetes.go:263 >>> component="discovery manager scrape" discovery=kubernetes msg="Using pod >>> service account via in-cluster config" >>> level=info ts=2020-11-16T14:29:40.030Z caller=main.go:892 msg="Completed >>> loading of configuration file" filename=/etc/prometheus/prometheus.yml >>> totalDuration=85.979364ms remote_storage=5.031µs web_handler=653ns >>> query_engine=1.227µs scrape=250.298µs scrape_sd=2.56867ms notify=33.201µs >>> notify_sd=11.894µs rules=82.217333ms >>> level=info ts=2020-11-16T14:29:40.030Z caller=main.go:684 msg="Server is >>> ready to receive web requests." >>> level=warn ts=2020-11-16T14:30:00.551Z caller=scrape.go:1372 >>> component="scrape manager" scrape_pool=kermit-proxy >>> target=https://userprom.xxx:443/metrics >>> <https://userprom.kermit-eu-b.itn.intraorange:443/metrics> msg="Error >>> on ingesting samples with different value but same timestamp" >>> num_dropped=30 >>> level=warn ts=2020-11-16T14:30:11.443Z caller=scrape.go:1372 >>> component="scrape manager" scrape_pool=kermit-proxy >>> target=https://userprom.xxx:443/metrics >>> <https://userprom.kermit-eu-b.itn.intraorange:443/metrics> msg="Error >>> on ingesting samples with different value but same timestamp" >>> num_dropped=30 >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/16a9cc0b-b9c5-406e-8c6f-92772bf7845dn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/16a9cc0b-b9c5-406e-8c6f-92772bf7845dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c3122f62-64b8-415a-a23f-1e32c67293b3n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c3122f62-64b8-415a-a23f-1e32c67293b3n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbaTYqvL0-i45_%2BTmZANfvGqUHQnzrvjP0SHn6KuTXEAw%40mail.gmail.com.
