Actually, I may have misunderstood – do you want to scrape the proxy itself, or something behind it?
In the former case, and assuming there is no further load balancing involved, you can use a service discovery mechanism that is appropriate in your environment. In a static config you directly provide the target(s). With service discovery, you provide Prometheus with a way to resolve each individual instance and automatically generate the targets. Unfortunately I cannot go into more detail without knowing the specifics of your environment. /MR On Wed, Nov 18, 2020, 10:52 bruno bourdolle <[email protected]> wrote: > maybe It 's an explication but I'm not the owner of xxx-proxy......How to > add a distinct labl/annotation to differentiate them ? > > Le mercredi 18 novembre 2020 à 10:00:12 UTC+1, [email protected] a > écrit : > >> Is this proxying to more than one backend? And in dev, there is only one? >> >> /MR >> >> On Tue, Nov 17, 2020 at 2:07 PM bruno bourdolle <[email protected]> >> wrote: >> >>> hi, >>> I localise the part of the conf that generate the error, but I'm not >>> sure to understand how to correct it. What is strange is that the same conf >>> on dev env works without these warns >>> >>> - job_name: 'my-proxy' >>> metrics_path: '/metrics' >>> scheme: https >>> tls_config: >>> ca_file: /var/run/secrets/ >>> kubernetes.io/serviceaccount/ca.crt >>> insecure_skip_verify: true >>> bearer_token_file: /var/run/secrets/ >>> kubernetes.io/serviceaccount/token >>> static_configs: >>> - targets: >>> - '${SCRAPE_CONFIGS_TARGET}' >>> >>> on target , I see >>> >>> >>> https://userprom.xxxt-intraxxx:443/metrics >>> <https://userprom.kermit-eu-b.itn.intraorange:443/metrics> >>> >>> up >>> group="production" instance="userprom.xxx.intraxxx:443" job="my-proxy" >>> >>> on dev no warns >>> on prod a lot of warns >>> >>> if I remove this block, I works without warns on prod , others conf are >>> ok but If I had this one, I have warns on prod >>> >>> Any idea to correct and explain why, I'm a newbie >>> >>> Le lundi 16 novembre 2020 à 16:48:25 UTC+1, [email protected] a >>> écrit : >>> >>>> This can happen in a few ways: >>>> >>>> 1. whatever exports the metrics, does so with a timestamp, but actually >>>> changes the value on you between scrapes without updating the timestamp. >>>> This is relatively unlikely unless this is something very specialized. >>>> 1.1 or it actually exposes the same metric twice within one /metrics >>>> response – but the official client libraries actively prevent that >>>> 2. (more likely) after all relabeling, you end up with metrics from >>>> multiple targets without any distinguishing labels between them. >>>> >>>> Are you scraping through some kind of proxy? In general, Prometheus >>>> expects to discover and access each target individually, so that it can >>>> separately collect the data from all of them. >>>> >>>> Look at the raw metrics endpoints that Prometheus would scrape – if >>>> they do not have duplicates or timestamps, it must be 2. Remove label drop >>>> or replacement rules that may accidentally coalesce multiple targets into >>>> having the same labels, especially if you are messing with the "instance" >>>> label. Looking at Prometheus' target page may also help in identifying >>>> targets that have the exact same label set. >>>> >>>> /MR >>>> >>>> >>>> >>>> On Mon, Nov 16, 2020 at 2:35 PM bruno bourdolle <[email protected]> >>>> wrote: >>>> >>>>> hello, >>>>> >>>>> I don(t understand how to correct the WARN about error on ingesting >>>>> samples... >>>>> I'm into kubernetes >>>>> I upgrade at last version of prometheus >>>>> I cleared the datas >>>>> Each time I start I have this line of warning each second >>>>> >>>>> Any idea ? >>>>> >>>>> bets >>>>> bruno >>>>> >>>>> level=info ts=2020-11-16T14:29:36.045Z caller=main.go:353 >>>>> msg="Starting Prometheus" version="(version=2.22.1, branch=HEAD, >>>>> revision=00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)" >>>>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:358 >>>>> build_context="(go=go1.15.3, user=root@516b109b1732, >>>>> date=20201105-14:02:25)" >>>>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:359 >>>>> host_details="(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 >>>>> UTC 2018 x86_64 prometheus-0 (none))" >>>>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:360 >>>>> fd_limits="(soft=1048576, hard=1048576)" >>>>> level=info ts=2020-11-16T14:29:36.046Z caller=main.go:361 >>>>> vm_limits="(soft=unlimited, hard=unlimited)" >>>>> level=info ts=2020-11-16T14:29:36.050Z caller=web.go:516 component=web >>>>> msg="Start listening for connections" address=0.0.0.0:9090 >>>>> level=info ts=2020-11-16T14:29:36.050Z caller=main.go:712 >>>>> msg="Starting TSDB ..." >>>>> level=info ts=2020-11-16T14:29:36.058Z caller=head.go:642 >>>>> component=tsdb msg="Replaying on-disk memory mappable chunks if any" >>>>> level=info ts=2020-11-16T14:29:36.093Z caller=head.go:656 >>>>> component=tsdb msg="On-disk memory mappable chunks replay completed" >>>>> duration=35.773378ms >>>>> level=info ts=2020-11-16T14:29:36.093Z caller=head.go:662 >>>>> component=tsdb msg="Replaying WAL, this may take a while" >>>>> level=info ts=2020-11-16T14:29:37.850Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=0 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:38.244Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=1 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=2 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=3 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=4 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=5 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=6 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=7 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=8 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=9 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=10 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=11 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=12 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=13 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=14 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=15 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.842Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=16 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.842Z caller=head.go:714 >>>>> component=tsdb msg="WAL segment loaded" segment=17 maxSegment=17 >>>>> level=info ts=2020-11-16T14:29:39.842Z caller=head.go:719 >>>>> component=tsdb msg="WAL replay completed" >>>>> checkpoint_replay_duration=97.404µs wal_replay_duration=3.748284098s >>>>> total_replay_duration=3.78422202s >>>>> level=info ts=2020-11-16T14:29:39.944Z caller=main.go:732 >>>>> fs_type=XFS_SUPER_MAGIC >>>>> level=info ts=2020-11-16T14:29:39.944Z caller=main.go:735 msg="TSDB >>>>> started" >>>>> level=info ts=2020-11-16T14:29:39.944Z caller=main.go:861 msg="Loading >>>>> configuration file" filename=/etc/prometheus/prometheus.yml >>>>> level=info ts=2020-11-16T14:29:39.945Z caller=kubernetes.go:263 >>>>> component="discovery manager scrape" discovery=kubernetes msg="Using pod >>>>> service account via in-cluster config" >>>>> level=info ts=2020-11-16T14:29:39.947Z caller=kubernetes.go:263 >>>>> component="discovery manager scrape" discovery=kubernetes msg="Using pod >>>>> service account via in-cluster config" >>>>> level=info ts=2020-11-16T14:29:40.030Z caller=main.go:892 >>>>> msg="Completed loading of configuration file" >>>>> filename=/etc/prometheus/prometheus.yml totalDuration=85.979364ms >>>>> remote_storage=5.031µs web_handler=653ns query_engine=1.227µs >>>>> scrape=250.298µs scrape_sd=2.56867ms notify=33.201µs notify_sd=11.894µs >>>>> rules=82.217333ms >>>>> level=info ts=2020-11-16T14:29:40.030Z caller=main.go:684 msg="Server >>>>> is ready to receive web requests." >>>>> level=warn ts=2020-11-16T14:30:00.551Z caller=scrape.go:1372 >>>>> component="scrape manager" scrape_pool=kermit-proxy >>>>> target=https://userprom.xxx:443/metrics >>>>> <https://userprom.kermit-eu-b.itn.intraorange:443/metrics> msg="Error >>>>> on ingesting samples with different value but same timestamp" >>>>> num_dropped=30 >>>>> level=warn ts=2020-11-16T14:30:11.443Z caller=scrape.go:1372 >>>>> component="scrape manager" scrape_pool=kermit-proxy >>>>> target=https://userprom.xxx:443/metrics >>>>> <https://userprom.kermit-eu-b.itn.intraorange:443/metrics> msg="Error >>>>> on ingesting samples with different value but same timestamp" >>>>> num_dropped=30 >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/16a9cc0b-b9c5-406e-8c6f-92772bf7845dn%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/16a9cc0b-b9c5-406e-8c6f-92772bf7845dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/c3122f62-64b8-415a-a23f-1e32c67293b3n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/c3122f62-64b8-415a-a23f-1e32c67293b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/215904ef-5523-4eac-bcb8-51a2641913ecn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/215904ef-5523-4eac-bcb8-51a2641913ecn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gaXvKoNnRTabx4sK9gnTG0UdtEKz4cMZJA5R1vtEHNbKQ%40mail.gmail.com.
