Hi,
We are running blackbox exporter as a Nomad allocation, so its ip:port is
dynamic. We had been running a prometheus instance, also in Nomad, and it
was configured with consul-template to dynamically set the __address__:
- target_label: __address__
replacement: [% range $i, $s := service "blackbox-exporter" %][%
if eq $i 0 %][%.Address%]:[%.Port%][% end %][% end %]
Now, we have migrated to using grafana cloud agent instead of a standalone
prometheus. We run the agent on every Nomad client, to collect logs and
metrics with its built in integrations. I want to update the config so
that it will spawn a prometheus instance to scrape any local blackbox
exporter services. However, instead of probing the URL that I expect, I
get these targets: (from curl of http://<grafana agent
ip>:53000/agent/api/v1/targets)
{
"instance": "f5d8368b42ae35f965c73d1be161364c",
"target_group": "blackbox-ba-dev",
"endpoint": "http://<bbexip>:21699/probe?target=<bbexip>%3A21699",
"state": "up",
"labels": {
"instance": "10.25.194.212:21699",
"job": "blackbox-ba-dev"
},
"last_scrape": "2020-11-11T21:31:30.45482873Z",
"scrape_duration_ms": 1,
"scrape_error": ""
},
{
"instance": "f5d8368b42ae35f965c73d1be161364c",
"target_group": "blackbox-ba-dev",
"endpoint":
"http://:/probe?target=mineralsoft.dev.ba.drillinginfo.com";,
"state": "down",
"labels": {
"instance": "mineralsoft.dev.ba.drillinginfo.com",
"job": "blackbox-ba-dev"
},
"last_scrape": "2020-11-11T21:31:29.843730608Z",
"scrape_duration_ms": 0,
"scrape_error": "Get
\"http:///probe?target=mineralsoft.dev.ba.drillinginfo.com\": http: no Host
in request URL"
}
What I need is a single target, with endpoint = http://<bbexip>:<bbex
port>/probe?target=mineralsoft.dev.ba.drillinginfo.com
Here is my prometheus config:
prometheus:
wal_directory: /tmp/wal
global:
scrape_interval: 60s
configs:
- name: 'ba-[% key "global/config/env" %]-prometheus-gca'
host_filter: false
remote_write:
- url:
https://prometheus-us-central1.grafana.net/api/prom/push
basic_auth:
[%with secret "di-secrets/prometheus/credentials"%]
username: [%.Data.data.USERNAME%]
password: [%.Data.data.APIKEY%]
[%end%]
scrape_configs:
- job_name: 'blackbox-ba-[% key "global/config/env" %]'
metrics_path: /probe
static_configs:
- targets:
- mineralsoft.dev.ba.drillinginfo.com # Target to
probe with http.
consul_sd_configs:
- services: ['blackbox-exporter']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- source_labels:
['__meta_consul_address','__meta_consul_service_port']
separator: ':'
target_label: __address__
Does anyone see what I am doing wrong?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com.
