On Wed, 11 Nov 2020 at 21:46, david goodine <davegood...@gmail.com> wrote:
>
> Hi,
> We are running blackbox exporter as a Nomad allocation, so its ip:port is
> dynamic. We had been running a prometheus instance, also in Nomad, and it
> was configured with consul-template to dynamically set the __address__:
>
> - target_label: __address__
> replacement: [% range $i, $s := service "blackbox-exporter"
> %][% if eq $i 0 %][%.Address%]:[%.Port%][% end %][% end %]
>
> Now, we have migrated to using grafana cloud agent instead of a standalone
> prometheus. We run the agent on every Nomad client, to collect logs and
> metrics with its built in integrations. I want to update the config so
> that it will spawn a prometheus instance to scrape any local blackbox
> exporter services. However, instead of probing the URL that I expect, I
> get these targets: (from curl of http://<grafana agent
> ip>:53000/agent/api/v1/targets)
>
> {
> "instance": "f5d8368b42ae35f965c73d1be161364c",
> "target_group": "blackbox-ba-dev",
> "endpoint": "http://<bbexip>:21699/probe?target=<bbexip>%3A21699",
> "state": "up",
> "labels": {
> "instance": "10.25.194.212:21699",
> "job": "blackbox-ba-dev"
> },
> "last_scrape": "2020-11-11T21:31:30.45482873Z",
> "scrape_duration_ms": 1,
> "scrape_error": ""
> },
> {
> "instance": "f5d8368b42ae35f965c73d1be161364c",
> "target_group": "blackbox-ba-dev",
> "endpoint": "http://:/probe?target=
> mineralsoft.dev.ba.drillinginfo.com",
> "state": "down",
> "labels": {
> "instance": "mineralsoft.dev.ba.drillinginfo.com",
> "job": "blackbox-ba-dev"
> },
> "last_scrape": "2020-11-11T21:31:29.843730608Z",
> "scrape_duration_ms": 0,
> "scrape_error": "Get \"http:///probe?target=
> mineralsoft.dev.ba.drillinginfo.com\": http: no Host in request URL"
> }
>
> What I need is a single target, with endpoint = http://<bbexip>:<bbex
> port>/probe?target=mineralsoft.dev.ba.drillinginfo.com
>
> Here is my prometheus config:
> prometheus:
> wal_directory: /tmp/wal
> global:
> scrape_interval: 60s
> configs:
> - name: 'ba-[% key "global/config/env" %]-prometheus-gca'
> host_filter: false
> remote_write:
> - url:
> https://prometheus-us-central1.grafana.net/api/prom/push
> basic_auth:
> [%with secret "di-secrets/prometheus/credentials"%]
> username: [%.Data.data.USERNAME%]
> password: [%.Data.data.APIKEY%]
> [%end%]
> scrape_configs:
> - job_name: 'blackbox-ba-[% key "global/config/env" %]'
>
Job labels should not vary by environment, you should use external labels
for this.
> metrics_path: /probe
> static_configs:
> - targets:
> - mineralsoft.dev.ba.drillinginfo.com # Target to
> probe with http.
>
consul_sd_configs:
> - services: ['blackbox-exporter']
>
You can't mix SDs like this, they get all merged together. Use this like a
normal exporter, remove the static_configs and put the single URL to be
probed in as a params.
Brian
> relabel_configs:
> - source_labels: [__address__]
> target_label: __param_target
> - source_labels: [__param_target]
> target_label: instance
> - source_labels:
> ['__meta_consul_address','__meta_consul_service_port']
> separator: ':'
> target_label: __address__
>
>
> Does anyone see what I am doing wrong?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Brian Brazil
www.robustperception.io
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAHJKeLq2C-hAqGFJ0YMn_3dkJRi6c-1ANAe8xKyfpfnqY_Su4w%40mail.gmail.com.
