Hi,
Did you ever get to the bottom of this? I too have a similar problem
We use prometheus in Kubernetes, and from other pods on same cluster I'm
able to do a curl against the blackbox exporter which works fine.
```
- job_name: prometheus-blackbox-exporter-lon-internal
honor_timestamps: true
params:
module:
- http_2xx
scrape_interval: 20s
scrape_timeout: 10s
metrics_path: /probe
scheme: https
static_configs:
- targets:
- https://www.google.com
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__address__]
separator: ;
regex: (.*)
target_label: __param_target
replacement: $1
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: __address__
replacement: promblackbox-lon.xxx.internal:443
action: replace
```
What I discovered is if I try wget from the pod where promethues is running
I get this error
```
/prometheus $ wget
"https://promblackbox-lon.xxx.internal.live:443/probe?module=http_2xx&target=https%3A%2F%2Fwww.google.com";
Connecting to promblackbox-lon.sea.live:443 (10.53.10.244:443)
wget: note: TLS certificate validation not implemented
wget: short read, have only 0: Connection reset by peer
wget: error getting response: No such file or directory
```
Same command from another pod (ubuntu) works fine
So for whatever reason there is some TLS issue in promethues pod causing
this, anyone have any ideas?
On Wednesday, March 11, 2020 at 3:00:13 PM UTC [email protected] wrote:
> Hi Experts,
>
> I configured blackbox-exporter to monitor kubernetes services. But I
> didn't managed to make it work
> It shows targets down in Prometheus with error connection reset by peer.
> Here is my settings:
>
> *blackbox config:*
> modules:
> http_2xx:
> prober: http
> http:
> method: GET
> preferred_ip_protocol: "ip4"
> valid_status_codes: [200]
> http_post_2xx:
> prober: http
> http:
> method: POST
> http_kubernetes_service:
> prober: http
> timeout: 5s
> http:
> headers:
> Accept: "*/*"
> Accept-Language: "en-US"
> tls_config:
> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> bearer_token_file: /var/run/secrets/
> kubernetes.io/serviceaccount/token
> preferred_ip_protocol: "ip4"
>
> *Prometheus additional-scrape-configs*
>
> - job_name: blackbox-exporter-kubernetes-services
> tls_config:
> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
> metrics_path: /probe
> params:
> module: [http_2xx]
> kubernetes_sd_configs:
> - role: service
> relabel_configs:
> - source_labels:
> [__meta_kubernetes_service_annotation_prometheus_io_probed]
> action: keep
> regex: true
> - source_labels: [__address__]
> target_label: __param_target
> - target_label: __address__
> replacement:
> monitoring-blackbox-exporter.kyma-system.svc.cluster.local:9115
> - source_labels: [__param_target]
> target_label: instance
> - action: labelmap
> regex: __meta_kubernetes_service_label_(.+)
> - source_labels: [__meta_kubernetes_namespace]
> target_label: kubernetes_namespace
> - source_labels: [__meta_kubernetes_service_name]
> target_label: kubernetes_name
>
> *Error In Prometheus Targets*
> Get
> http://monitoring-blackbox-exporter.kyma-system.svc.cluster.local:9115/probe?module=http_2xx&target=kiali.my-ns.svc%3A20001:
>
> read tcp {ip}:40780->{ip}:9115: read: connection reset by peer
>
> I have tried several suggestion and cannot solve this! Anyone could help,
> appreciate it!
>
> Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/b8bdc093-8971-43c3-8633-32c937147a89n%40googlegroups.com.
