On Wed, 11 Nov 2020 at 21:46, david goodine <[email protected]> wrote:
>
> Hi,
> We are running blackbox exporter as a Nomad allocation, so its ip:port is
> dynamic. We had been running a prometheus instance, also in Nomad, and it
> was configured with consul-template to dynamically set the __address__:
>
> - target_label: __address__
> replacement: [% range $i, $s := service "blackbox-exporter"
> %][% if eq $i 0 %][%.Address%]:[%.Port%][% end %][% end %]
>
> Now, we have migrated to using grafana cloud agent instead of a standalone
> prometheus. We run the agent on every Nomad client, to collect logs and
> metrics with its built in integrations. I want to update the config so
> that it will spawn a prometheus instance to scrape any local blackbox
> exporter services. However, instead of probing the URL that I expect, I
> get these targets: (from curl of http://<grafana agent
> ip>:53000/agent/api/v1/targets)
>
> {
> "instance": "f5d8368b42ae35f965c73d1be161364c",
> "target_group": "blackbox-ba-dev",
> "endpoint": "http://<bbexip>:21699/probe?target=<bbexip>%3A21699",
> "state": "up",
> "labels": {
> "instance": "10.25.194.212:21699",
> "job": "blackbox-ba-dev"
> },
> "last_scrape": "2020-11-11T21:31:30.45482873Z",
> "scrape_duration_ms": 1,
> "scrape_error": ""
> },
> {
> "instance": "f5d8368b42ae35f965c73d1be161364c",
> "target_group": "blackbox-ba-dev",
> "endpoint": "http://:/probe?target=
> mineralsoft.dev.ba.drillinginfo.com",
> "state": "down",
> "labels": {
> "instance": "mineralsoft.dev.ba.drillinginfo.com",
> "job": "blackbox-ba-dev"
> },
> "last_scrape": "2020-11-11T21:31:29.843730608Z",
> "scrape_duration_ms": 0,
> "scrape_error": "Get \"http:///probe?target=
> mineralsoft.dev.ba.drillinginfo.com\": http: no Host in request URL"
> }
>
> What I need is a single target, with endpoint = http://<bbexip>:<bbex
> port>/probe?target=mineralsoft.dev.ba.drillinginfo.com
>
> Here is my prometheus config:
> prometheus:
> wal_directory: /tmp/wal
> global:
> scrape_interval: 60s
> configs:
> - name: 'ba-[% key "global/config/env" %]-prometheus-gca'
> host_filter: false
> remote_write:
> - url:
> https://prometheus-us-central1.grafana.net/api/prom/push
> basic_auth:
> [%with secret "di-secrets/prometheus/credentials"%]
> username: [%.Data.data.USERNAME%]
> password: [%.Data.data.APIKEY%]
> [%end%]
> scrape_configs:
> - job_name: 'blackbox-ba-[% key "global/config/env" %]'
>
Job labels should not vary by environment, you should use external labels
for this.
> metrics_path: /probe
> static_configs:
> - targets:
> - mineralsoft.dev.ba.drillinginfo.com # Target to
> probe with http.
>
consul_sd_configs:
> - services: ['blackbox-exporter']
>
You can't mix SDs like this, they get all merged together. Use this like a
normal exporter, remove the static_configs and put the single URL to be
probed in as a params.
Brian
> relabel_configs:
> - source_labels: [__address__]
> target_label: __param_target
> - source_labels: [__param_target]
> target_label: instance
> - source_labels:
> ['__meta_consul_address','__meta_consul_service_port']
> separator: ':'
> target_label: __address__
>
>
> Does anyone see what I am doing wrong?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Brian Brazil
www.robustperception.io
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAHJKeLq2C-hAqGFJ0YMn_3dkJRi6c-1ANAe8xKyfpfnqY_Su4w%40mail.gmail.com.
