Ok, found it... As you said, someone else is hitting the /metrics endpoint, the health probes!
(ノ_<。) https://github.com/prometheus/prometheus/issues/2831 Gonna use that feature now... Thanks for the help! On Friday, March 6, 2020 at 9:44:01 AM UTC+1, Brian Brazil wrote: > > On Fri, 6 Mar 2020 at 08:30, Albert Serrallé Ríos <[email protected] > <javascript:>> wrote: > >> To make things easier, I've just removed everything except for one metric: >> >> region: us-east-1metrics: >> - aws_namespace: AWS/CloudFront >> aws_metric_name: Requests >> aws_statistics: [Sum] >> aws_dimensions: [DistributionId, Region] >> aws_dimensions_select: >> Region: [Global] >> >> Then prometheus: >> >> global: >> scrape_interval: 30m >> scrape_timeout: 10s >> >> external_labels: >> monitor: 'prometheus-local' >> account: 'es-newconstruction-pre' >> >> scrape_configs: >> - job_name: 'prometheus' >> static_configs: >> - targets: ['localhost:8080'] >> >> - job_name: 'cloudwatch-exporter' >> honor_labels: false >> metrics_path: '/metrics' >> static_configs: >> - targets: >> - 'localhost:8001' >> >> What I can see now is: >> >> - 20 ListMetrics per 5 minutes >> - 60 GetMetricStatistics per 5 minutes >> >> I have *3 Cloudfront distributions* in that environment. >> > > I would expect 1 ListMetrics then, and 3 GetMetricStatistics. If you > manually hit /metrics how much do the request counters go up by? > > Brian > > >> >> Thanks for helping out on this, Brian. >> >> On Friday, March 6, 2020 at 1:52:02 AM UTC+1, Brian Brazil wrote: >>> >>> On Thu, 5 Mar 2020 at 19:37, Albert Serrallé Ríos <[email protected]> >>> wrote: >>> >>>> I have 11 metrics configured, and my metrics show 100/105 ListMetrics >>>> calls and 380/399 GetMetricStatistics calls *every 5 minutes*. >>>> >>>> Certainly, the scraping interval affects the number of calls to >>>> GetMetricStatistics (I've tested 5 minute vs 30 minute scrape, and the >>>> number of calls is is reduced by 1/3 aprox) but in a way that is not >>>> obvious to me. >>>> >>>> Also, the number of ListMetrics calls seems unecessarily high... >>>> >>> >>> Can you share your config? >>> >>> I'd also suspect that there's more than one Prometheus hitting the >>> exporter. >>> >>> Brian >>> >>> >>>> >>>> On Thursday, March 5, 2020 at 8:21:44 PM UTC+1, Brian Brazil wrote: >>>>> >>>>> On Thu, 5 Mar 2020 at 19:10, Albert Serrallé Ríos <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> >>>>>> I've raised the scrape_interval to 30m and I've noticed that there's >>>>>> a minimum amount of calls that, no matter the value for >>>>>> scrape_interval, will be executed. >>>>>> >>>>>> Is there any theoretical minimum of GetMetricStatistics and >>>>>> ListMetrics calls? Could that be configured or is it "hardcoded"? >>>>>> >>>>> >>>>> You can expect at least one ListMetrics per configured metric, and >>>>> then one GetMetricStatistics per relevant metric it finds. These should >>>>> only ever be happening due to a scrape. >>>>> >>>>> Brian >>>>> >>>>> >>>>>> >>>>>> Context: I'm trying to reduce the Cloudwatch costs as much as >>>>>> possible in dev/staging environments. >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/bb4258f4-56c4-45cd-8d4c-3cec48de0bd0%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/prometheus-users/bb4258f4-56c4-45cd-8d4c-3cec48de0bd0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> >>>>> >>>>> -- >>>>> Brian Brazil >>>>> www.robustperception.io >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/691ad00d-ea13-4018-919d-053fbc2b4d90%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/691ad00d-ea13-4018-919d-053fbc2b4d90%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Brian Brazil >>> www.robustperception.io >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/bd5ef7f3-d5eb-4128-8017-0b27f3a41098%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/bd5ef7f3-d5eb-4128-8017-0b27f3a41098%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/09f70b94-9064-43c1-a6a9-ca48915f851c%40googlegroups.com.
