I've verified: - v2.20.1 is the last version where the mTLS scraping works. - It doesn't matter which Docker registry you pull from (Docker Hub or quay.io - I've sometimes seen different "versions" of containers based on registry).
Looking at the release notes for v2.21.0 <https://github.com/prometheus/prometheus/releases/tag/v2.21.0> it appears there's a new version of Go used for compilation which includes some changes on how certificates are handled <https://golang.org/doc/go1.15#commonname>. Unclear if this is what I'm hitting, but it seems worth looking into. On Tuesday, July 6, 2021 at 11:02:56 AM UTC-7 Travis Illig wrote: > I'm deploying Prometheus using the Helm chart > <https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus> > > and I have it configured to scrape Istio mTLS-secured pods using the TLS > settings specified > <https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings> > by the Istio team to do so. Basically what this amounts to is: > > - Add the Istio sidecar to the Prometheus instance but disable all > traffic proxying - you just want to get the certificates from it. > - Mount the certificates into the Prometheus container. > - Set up your scrape configuration to use the certificates when > scraping Istio-enabled pods. > > The YAML for the scrape configuration looks like this: > > - job_name: "kubernetes-pods-istio-secure" > scheme: https > tls_config: > ca_file: /etc/istio-certs/root-cert.pem > cert_file: /etc/istio-certs/cert-chain.pem > key_file: /etc/istio-certs/key.pem > insecure_skip_verify: true > > *This totally works using Prometheus v2.20.1* packaged as > `prom/prometheus` from Docker Hub. > > *This fails on Prometheus v2.28.0* packaged as ` > quay.io/prometheus/prometheus` <http://quay.io/prometheus/prometheus>. > Instead of getting a successful scrape, I get "connection reset by peer." > I've validated the files are there and properly mounted; they have the > expected contents; and there are no Prometheus log messages to indicate > anything is amiss. > > I've been rolling back slowly to see where it starts working again. I've > tried v2.26.0 and it still fails. I thought I'd drop a note in here to see > if anyone knows what's up. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/aadef2ec-bab5-4182-929e-ab0c8baed131n%40googlegroups.com.
