I tried mounting local certs and made no difference to wget, the problem still persists with prometheus as well.
I looked in the article for troubleshooting blackbox exporter, however the issue appears to be with prometheus (or busybox container) as I can successfully connect to blackbox from other pods within the same Kubernetes cluster. This is the error as seen from /targets endpoint. Get "https://promblackbox-lon.sea.live:443/probe?module=http_2xx&target=https%3A%2F%2Fgain.uat.clarksons.com": read tcp 10.23.195.103:56740->10.53.10.244:443: read: connection reset by peer This seems to be a common issue yet no solution that I could find (yet!). On Tuesday, August 11, 2020 at 8:31:49 AM UTC+1 Abu Belal wrote: > Hi Christian, > > Thank you for your response :) > > I was thinking of mounting the underlying nodes (managed Azure Kubernetes) > certs to prometheus, do you think that could cause problems? > > On Monday, August 10, 2020 at 9:37:20 PM UTC+1 Christian Hoffmann wrote: > >> On 8/7/20 4:52 PM, Abu Belal wrote: >> > What I discovered is if I try wget from the pod where promethues is >> > running I get this error >> > ``` >> > /prometheus $ wget >> > " >> https://promblackbox-lon.xxx.internal.live:443/probe?module=http_2xx&target=https%3A%2F%2Fwww.google.com >> " >> > Connecting to promblackbox-lon.sea.live:443 (10.53.10.244:443) >> > wget: note: TLS certificate validation not implemented >> > wget: short read, have only 0: Connection reset by peer >> > wget: error getting response: No such file or directory >> > ``` >> > >> > Same command from another pod (ubuntu) works fine >> >> Hrm, had never seen this, but a quick Google search turns up this issue: >> >> https://github.com/docker-library/busybox/issues/80 >> >> And as I think the Prometheus docker images are based on busybox, this >> might explain the wget problem. >> >> I don't think a missing openssl implementation would cause issues for >> blackbox_exporter, as it uses Go's http/tls stack, as far as I >> understand. However, it might still rely on some default certificates. >> >> I suggest trying to get more blackbox_exporter logs and maybe trying to >> place a (relevant) ca bundle in the proper paths. >> >> This article may also help: >> https://www.robustperception.io/debugging-blackbox-exporter-failures >> >> Kind regards, >> Christian >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0c08db7c-0640-45de-a58c-441196693de9n%40googlegroups.com.
