Hi Brian, Thanks for responding. I am using grafana-agent because we are using mimir on top of prometheus. so at first, I thought the same so I went to grafanaa-agent pod using exec command and tried to connect other running services using telnet command. For ex. telnet develop.static-service.dev.svc.cluster.local:8080, it throws 400 error and the same error we are getting from sd-config jobs. Either we are missing something or there may be one more reason that we are using istio for communication that is listening on 15020 port but we tried to connect using that port also but it throws error.
On Sunday, 19 March 2023 at 19:45:13 UTC+5:30 Brian Candler wrote: > What you need to do, as I suggested before, is go to the PromQL web > interface and look at Status > Targets to see exactly the scrape URL and > its query parameters, e.g. > > [image: img1.png] > > Then you'll see what's different between the static_configs job and the > kubernetes_sd_configs job, in terms of the generated scrape URL and > params. (And the total number of endpoints being probed, of course). > > Looking at your config, my guess is that the __address__ that you're > getting from kubernetes SD is either a plain address or an address:port > pair, but the target for your http_2xx module requires a complete URL like > http://address:port or https://address:port (maybe with an additional path > too). You'll be able to see if the http:// is missing under Status > > Targets. > > If that's true, then you need to modify your rewriting rules to build the > target appropriately, for example: > > - source_labels: [__address__] > target_label: __param_target > replacement: "http://${1}"; > > That's just a starting point. You need to work on it as appropriate until > your kubernetes SD config gives "target" params which match your static SD > config. > > On Sunday, 19 March 2023 at 13:35:43 UTC dikshant rai wrote: > >> Hi Brian, >> Thanks for the response. >> I have configured static and sd both as different job and when I am >> hitting SD job, I am getting targets as all the services inside the cluster >> but they are throwing 400 or 404 error but when I check the logs, the >> status is 200. >> I am attaching job for your reference. Let me know if I am missing >> something. I also tried to check connection by going into blackbox-exporter >> pod but it is still throwing same error. I have put all under same >> namespace to avoid any communication failure. >> - job_name: "blackbox-kubernetes-services" >> metrics_path: /probe >> params: >> module: [http_2xx] >> kubernetes_sd_configs: >> - role: service >> relabel_configs: >> # Example relabel to probe only some services that have " >> example.io/should_be_probed = true" annotation >> # - source_labels: >> [__meta_kubernetes_service_annotation_example_io_should_be_probed] >> # action: keep >> # regex: true >> - source_labels: [__address__] >> target_label: __param_target >> - target_label: __address__ >> replacement: prometheus-blackbox-exporter:9115 >> - source_labels: [__param_target] >> target_label: instance >> - action: labelmap >> regex: __meta_kubernetes_service_label_(.+) >> - source_labels: [__meta_kubernetes_namespace] >> target_label: kubernetes_namespace >> - source_labels: [__meta_kubernetes_service_name] >> target_label: kubernetes_service_name >> >> On Friday, 17 March 2023 at 21:04:29 UTC+5:30 Brian Candler wrote: >> >>> > When I am using static config then it shows fine but when I am using >>> service discovery for status of service or pod some are coming up some are >>> down >>> >>> Then clearly you're not scraping/probing the same endpoints as you did >>> with the static config. >>> >>> The easiest way to check is to go to the Prometheus web interface and >>> look at the "Status > Targets" page and see what *actual* targets you're >>> probing: first with the static config, and then secondly with the SD config >>> (actually you can do both at the same time, but configure them as separate >>> jobs so you can distinguish them by the "job" label). The first column >>> shows the URL being scraped and the URL params being passed ("module" and >>> "target" are the ones that need to be set for blackbox_exporter, of course). >>> >>> If you can show your static config, and the service discovery config, >>> the problem might be clearer. My guess is the SD discovery isn't returning >>> data in the form that you expect, or that your rewrite rules aren't working >>> in the way that you expect. >>> >>> On Friday, 17 March 2023 at 15:23:28 UTC dikshant rai wrote: >>> >>>> Hi, >>>> I tried to configure blackbox exporter for checking api status with >>>> prometheus on kubernetes. When I am using static config then it shows fine >>>> but when I am using service discovery for status of service or pod some >>>> are >>>> coming up some are down. I used telnet to check connectivity as well that >>>> is fine but still don't know where it is failing. >>>> Please help. >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6487f5b6-9243-4e50-a09c-725b923e91d6n%40googlegroups.com.
