Inside a scrape job, either you are using static_configs: ...
or file_sd_configs: ... or kubernetes_sd_configs: ... (or others). These are all forms of service discovery. The result of all of these is the same: a list of targets to scrape as __address__, and other labels to associate with those targets. I will say this for the final time: if it works with static_configs but doesn't work with kubernetes_sd_configs, it's because you're generating a different __address__ and/or labels. And I've already told you twice how to view the list of targets and labels which the scrape job is using. On Tuesday, 21 March 2023 at 04:53:50 UTC dikshant rai wrote: > Hi Brian, > I hope there is some confusion for static config, we are not using any SD, > instead we are adding targets manually that is exposed to internet and that > is working fine but when we want to monitor pods or services using > sd_config, we are able to list out those with their name along with port > number assigned. The only issue is that they are down. > > On Tuesday, 21 March 2023 at 00:51:10 UTC+5:30 Brian Candler wrote: > >> If it's the same blackbox exporter you're talking to, and it is returning >> "success" to one probe request (originating from your static_sd_configs) >> but "fail" to another probe request (from your kubernetes_sd_configs), then >> ISTM that the requests that you are sending to blackbox_exporter are >> different in those two cases. >> >> On Monday, 20 March 2023 at 17:06:29 UTC dikshant rai wrote: >> >>> Hi Brian, >>> Thanks for responding. I am using grafana-agent because we are using >>> mimir on top of prometheus. so at first, I thought the same so I went to >>> grafanaa-agent pod using exec command and tried to connect other running >>> services using telnet command. For ex. telnet >>> develop.static-service.dev.svc.cluster.local:8080, it throws 400 error and >>> the same error we are getting from sd-config jobs. Either we are missing >>> something or there may be one more reason that we are using istio for >>> communication that is listening on 15020 port but we tried to connect using >>> that port also but it throws error. >>> >>> On Sunday, 19 March 2023 at 19:45:13 UTC+5:30 Brian Candler wrote: >>> >>>> What you need to do, as I suggested before, is go to the PromQL web >>>> interface and look at Status > Targets to see exactly the scrape URL and >>>> its query parameters, e.g. >>>> >>>> [image: img1.png] >>>> >>>> Then you'll see what's different between the static_configs job and the >>>> kubernetes_sd_configs job, in terms of the generated scrape URL and >>>> params. (And the total number of endpoints being probed, of course). >>>> >>>> Looking at your config, my guess is that the __address__ that you're >>>> getting from kubernetes SD is either a plain address or an address:port >>>> pair, but the target for your http_2xx module requires a complete URL like >>>> http://address:port or https://address:port (maybe with an additional path >>>> too). You'll be able to see if the http:// is missing under Status > >>>> Targets. >>>> >>>> If that's true, then you need to modify your rewriting rules to build >>>> the target appropriately, for example: >>>> >>>> - source_labels: [__address__] >>>> target_label: __param_target >>>> replacement: "http://${1}"; >>>> >>>> That's just a starting point. You need to work on it as appropriate >>>> until your kubernetes SD config gives "target" params which match your >>>> static SD config. >>>> >>>> On Sunday, 19 March 2023 at 13:35:43 UTC dikshant rai wrote: >>>> >>>>> Hi Brian, >>>>> Thanks for the response. >>>>> I have configured static and sd both as different job and when I am >>>>> hitting SD job, I am getting targets as all the services inside the >>>>> cluster >>>>> but they are throwing 400 or 404 error but when I check the logs, the >>>>> status is 200. >>>>> I am attaching job for your reference. Let me know if I am missing >>>>> something. I also tried to check connection by going into >>>>> blackbox-exporter >>>>> pod but it is still throwing same error. I have put all under same >>>>> namespace to avoid any communication failure. >>>>> - job_name: "blackbox-kubernetes-services" >>>>> metrics_path: /probe >>>>> params: >>>>> module: [http_2xx] >>>>> kubernetes_sd_configs: >>>>> - role: service >>>>> relabel_configs: >>>>> # Example relabel to probe only some services that have " >>>>> example.io/should_be_probed = true" annotation >>>>> # - source_labels: >>>>> [__meta_kubernetes_service_annotation_example_io_should_be_probed] >>>>> # action: keep >>>>> # regex: true >>>>> - source_labels: [__address__] >>>>> target_label: __param_target >>>>> - target_label: __address__ >>>>> replacement: prometheus-blackbox-exporter:9115 >>>>> - source_labels: [__param_target] >>>>> target_label: instance >>>>> - action: labelmap >>>>> regex: __meta_kubernetes_service_label_(.+) >>>>> - source_labels: [__meta_kubernetes_namespace] >>>>> target_label: kubernetes_namespace >>>>> - source_labels: [__meta_kubernetes_service_name] >>>>> target_label: kubernetes_service_name >>>>> >>>>> On Friday, 17 March 2023 at 21:04:29 UTC+5:30 Brian Candler wrote: >>>>> >>>>>> > When I am using static config then it shows fine but when I am >>>>>> using service discovery for status of service or pod some are coming up >>>>>> some are down >>>>>> >>>>>> Then clearly you're not scraping/probing the same endpoints as you >>>>>> did with the static config. >>>>>> >>>>>> The easiest way to check is to go to the Prometheus web interface and >>>>>> look at the "Status > Targets" page and see what *actual* targets you're >>>>>> probing: first with the static config, and then secondly with the SD >>>>>> config >>>>>> (actually you can do both at the same time, but configure them as >>>>>> separate >>>>>> jobs so you can distinguish them by the "job" label). The first column >>>>>> shows the URL being scraped and the URL params being passed ("module" >>>>>> and >>>>>> "target" are the ones that need to be set for blackbox_exporter, of >>>>>> course). >>>>>> >>>>>> If you can show your static config, and the service discovery config, >>>>>> the problem might be clearer. My guess is the SD discovery isn't >>>>>> returning >>>>>> data in the form that you expect, or that your rewrite rules aren't >>>>>> working >>>>>> in the way that you expect. >>>>>> >>>>>> On Friday, 17 March 2023 at 15:23:28 UTC dikshant rai wrote: >>>>>> >>>>>>> Hi, >>>>>>> I tried to configure blackbox exporter for checking api status with >>>>>>> prometheus on kubernetes. When I am using static config then it shows >>>>>>> fine >>>>>>> but when I am using service discovery for status of service or pod some >>>>>>> are >>>>>>> coming up some are down. I used telnet to check connectivity as well >>>>>>> that >>>>>>> is fine but still don't know where it is failing. >>>>>>> Please help. >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/77f2e3d7-ad17-4d2e-8993-613a1454c580n%40googlegroups.com.
