My two cents: I have a lot of metrics to monitor which aren't specific to the individual pods. Having the service-role supported (non-"blackbox") in the default configs would've saved me a lot of confusion, and I bet I'm not alone. Failing that, calling this use case out more clearly in the docs would've bee helpful. I'm new to prometheus and unfamiliar with its configs, so the solution wasn't as trivial for me as it would be for some.
On Tue, Apr 7, 2020 at 2:12 PM Matthias Rampke <[email protected]> wrote: > Question to the wider audience: would it be sensible to add this to the > example/default Kubernetes configuration? > > /MR > > On Tue, Apr 7, 2020, 20:51 Patrick Schless <[email protected]> > wrote: > >> Hi Matthias, >> >> Thanks for your thoughts. I've experimented some more, and have it >> working now. Best as I can tell, it's simply missing from the default >> config in the prom/prometheus docker image: >> https://gist.github.com/plainlystated/95c79163583d5d107f2d6e85f9d335d5 >> >> In that config, there are only two "role: service" sections. One is the >> pushgateway, and the other forces use of the blackbox app, which seems like >> it shouldn't be necessary for what I'm trying to do. >> >> I added a scrape config, which seems to have done the trick: >> >>> - honor_labels: true >>> job_name: prometheus-service-non-blackbox >>> kubernetes_sd_configs: >>> - role: service >>> relabel_configs: >>> - action: keep >>> regex: service >>> source_labels: >>> - __meta_kubernetes_service_annotation_prometheus_io_scrape >>> - action: replace >>> regex: (.+) >>> source_labels: >>> - __meta_kubernetes_service_annotation_prometheus_io_path >>> target_label: __metrics_path__ >>> >> >> Then I just set the service annotation: >> >>> prometheus.io/scrape: service >>> >> >> Now my prometheus correctly has 1 target for the service, and no targets >> for the endpoints. >> >> Thanks for your help! >> >> - Patrick >> >> On Tue, Apr 7, 2020 at 11:07 AM Matthias Rampke <[email protected]> >> wrote: >> >>> Kubernetes creates the Endpoints object based on the Service object. >>> Does it carry over the annotations while doing that? If so, you are getting >>> returns in both SD configs because there are two objects in Kubernetes to >>> be discovered. >>> >>> I would suggest using a different value for the scrape annotation, and >>> matching on that in the service SD configuration, to distinguish whether >>> you want this thing scraped once or for each instance. >>> >>> /MR >>> >>> On Tue, Apr 7, 2020 at 3:37 AM Patrick Schless < >>> [email protected]> wrote: >>> >>>> I'm using prometheus on k8s, and trying to add an annotation to my >>>> service for the service discovery. The metrics I'm exposing are >>>> business-level (eg "# of customers") so I don't need all pods queried. >>>> Querying the service once (with a random pod) is sufficient. As I >>>> understand it, there is a service discovery role applicable: "service". >>>> >>>> However, when I add this annotation to my service, it is "discovered" >>>> by the endpoints role, which then tries to hit all downstream pods. >>>> >>>> prometheus.io/scrape: true >>>>> >>>> >>>> >>>> The docs clearly make a distinction between service SD and endpoints >>>> SD, but it's not clear to me now to choose one or the other. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/bbec2f68-afea-4366-8156-b729bc04b373%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-users/bbec2f68-afea-4366-8156-b729bc04b373%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAJOxyf3M68sDbhjdkuRzH3hhxDOycpEDaH%2BzNnMdi98UzzYjsQ%40mail.gmail.com.
