thank you for your response Bryan. So this means it’s scraped and ingested but dedupe is done by tstb. thanks for pointing out the issue , it’s useful.
________________________________ From: prometheus-developers@googlegroups.com <prometheus-developers@googlegroups.com> on behalf of Bryan Boreham <bjbore...@gmail.com> Sent: Thursday, August 17, 2023 1:46 AM To: Prometheus Developers <prometheus-developers@googlegroups.com> Subject: [prometheus-developers] Re: Exemplars & histogram buckets > does it de-dup the exemplars Yes, an exemplar that is a duplicate of one in memory is discarded. > so it doesn't ingest the older exemplars for some buckets, again, for every > scrape Prometheus reads the values on every scrape. I don’t know how it could de-dup without reading them. Did you have some other meaning of “ingest” in mind? I think there is an issue if the older exemplar is not in memory : https://github.com/prometheus/prometheus/issues/9377 Bryan On Wednesday, 16 August 2023 at 21:52:03 UTC+1 Vishwanath wrote: Hi - I instrumented an histogram with golang prom sdk and it includes exemplar. When i curl the metrics endpoint, i see below for my histogram with exemplar. As you can see, each bucket can have exemplar, and exemplar has time and they (exemplar) can be old. When Prometheus with exemplar enabled is scraping this end-point, does it de-dup the exemplars so it doesn't ingest the older exemplars for some buckets, again, for every scrape ? Thanks for reading and your help. # HELP exemplar_my_histogram vishwa's test metric3 histogram # TYPE exemplar_my_histogram histogram exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-3.0"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.9"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.8"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.6999999999999997"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.5999999999999996"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.4999999999999996"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.3999999999999995"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.2999999999999994"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.1999999999999993"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-2.099999999999999"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.9999999999999991"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.899999999999999"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.799999999999999"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.6999999999999988"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.5999999999999988"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.4999999999999987"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.3999999999999986"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.2999999999999985"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.1999999999999984"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-1.0999999999999983"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.9999999999999983"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.8999999999999984"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.7999999999999984"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.6999999999999984"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.5999999999999984"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.49999999999999845"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.39999999999999847"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.2999999999999985"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.19999999999999848"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="-0.09999999999999848"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.5265566588595902e-15"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.10000000000000153"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.20000000000000154"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.30000000000000154"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.4000000000000016"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.5000000000000016"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.6000000000000015"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.7000000000000015"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.8000000000000015"} 0 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="0.9000000000000015"} 1 # {mytraceid="fa128244a04e3167"} 0.81342384094935 1.6916505472120025e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.0000000000000016"} 1 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.1000000000000016"} 1 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.2000000000000017"} 2 # {mytraceid="fa128244a04e3167"} 1.1297683801942673 1.6915601023219924e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.3000000000000018"} 4 # {mytraceid="fa128244a04e3167"} 1.2198116269608419 1.6920360561040416e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.400000000000002"} 5 # {mytraceid="fa128244a04e3167"} 1.358550014547222 1.6919264337043686e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.500000000000002"} 5 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.600000000000002"} 6 # {mytraceid="fa128244a04e3167"} 1.5173031242650117 1.691906796059689e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.7000000000000022"} 6 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.8000000000000023"} 7 # {mytraceid="fa128244a04e3167"} 1.7294844647831784 1.6919916590401986e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="1.9000000000000024"} 7 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.000000000000002"} 9 # {mytraceid="fa128244a04e3167"} 1.952000333558452 1.692091738121497e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.1000000000000023"} 10 # {mytraceid="fa128244a04e3167"} 2.096613085256622 1.6918952017111053e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.2000000000000024"} 13 # {mytraceid="fa128244a04e3167"} 2.1298900181088167 1.6919736919950492e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.3000000000000025"} 14 # {mytraceid="fa128244a04e3167"} 2.2400071381918214 1.691540474782527e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.4000000000000026"} 15 # {mytraceid="fa128244a04e3167"} 2.390128024808927 1.6917299980148625e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.5000000000000027"} 16 # {mytraceid="fa128244a04e3167"} 2.4096549758317476 1.692204001547885e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.6000000000000028"} 18 # {mytraceid="fa128244a04e3167"} 2.5060113851880983 1.6920551035233934e+09 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.700000000000003"} 18 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.800000000000003"} 18 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="2.900000000000003"} 18 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="3.000000000000003"} 18 exemplar_my_histogram_bucket{dim1="e",dim2="f",le="+Inf"} 69970 # {mytraceid="fa128244a04e3167"} 644.3222078518065 1.6922187573901772e+09 exemplar_my_histogram_sum{dim1="e",dim2="f"} 3.5056405397655797e+08 exemplar_my_histogram_count{dim1="e",dim2="f"} 69970 -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com<mailto:prometheus-developers+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f63e600d-e743-4937-bccb-29ee50aa3dbfn%40googlegroups.com<https://groups.google.com/d/msgid/prometheus-developers/f63e600d-e743-4937-bccb-29ee50aa3dbfn%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/MW4PR04MB7170BA92001784049E12A058A51BA%40MW4PR04MB7170.namprd04.prod.outlook.com.
