Hi Matthias, Thanks for your thoughts. I've experimented some more, and have it working now. Best as I can tell, it's simply missing from the default config in the prom/prometheus docker image: https://gist.github.com/plainlystated/95c79163583d5d107f2d6e85f9d335d5
In that config, there are only two "role: service" sections. One is the pushgateway, and the other forces use of the blackbox app, which seems like it shouldn't be necessary for what I'm trying to do. I added a scrape config, which seems to have done the trick: > - honor_labels: true > job_name: prometheus-service-non-blackbox > kubernetes_sd_configs: > - role: service > relabel_configs: > - action: keep > regex: service > source_labels: > - __meta_kubernetes_service_annotation_prometheus_io_scrape > - action: replace > regex: (.+) > source_labels: > - __meta_kubernetes_service_annotation_prometheus_io_path > target_label: __metrics_path__ > Then I just set the service annotation: > prometheus.io/scrape: service > Now my prometheus correctly has 1 target for the service, and no targets for the endpoints. Thanks for your help! - Patrick On Tue, Apr 7, 2020 at 11:07 AM Matthias Rampke <[email protected]> wrote: > Kubernetes creates the Endpoints object based on the Service object. Does > it carry over the annotations while doing that? If so, you are getting > returns in both SD configs because there are two objects in Kubernetes to > be discovered. > > I would suggest using a different value for the scrape annotation, and > matching on that in the service SD configuration, to distinguish whether > you want this thing scraped once or for each instance. > > /MR > > On Tue, Apr 7, 2020 at 3:37 AM Patrick Schless <[email protected]> > wrote: > >> I'm using prometheus on k8s, and trying to add an annotation to my >> service for the service discovery. The metrics I'm exposing are >> business-level (eg "# of customers") so I don't need all pods queried. >> Querying the service once (with a random pod) is sufficient. As I >> understand it, there is a service discovery role applicable: "service". >> >> However, when I add this annotation to my service, it is "discovered" by >> the endpoints role, which then tries to hit all downstream pods. >> >> prometheus.io/scrape: true >>> >> >> >> The docs clearly make a distinction between service SD and endpoints SD, >> but it's not clear to me now to choose one or the other. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/bbec2f68-afea-4366-8156-b729bc04b373%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/bbec2f68-afea-4366-8156-b729bc04b373%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAJOxyf3%3D%3DSN1X8fnpC-MpAGWbUqx4z2cBkv8Qf-nhXv9gSU15w%40mail.gmail.com.
