It's not the certificate handling. I tried setting GODEBUG as indicated in the docs and that didn't fix anything. I'm starting to wonder if it's an HTTP/2 issue or something similar but I'm not sure how to determine if that's the problem.
The error message in Prometheus debug logs isn't super helpful, it just seems to indicate a protocol problem. level=debug ts=2021-07-06T20:00:50.996Z caller=scrape.go:1091 component="scrape manager" scrape_pool=kubernetes-pods-istio-secure target=https://10.244.3.10:9102/metrics msg="Scrape failed" err="Get \"https://10.244.3.10:9102/metrics\": read tcp 10.244.4.85:51794->10.244.3.10:9102: read: connection reset by peer" On Tuesday, July 6, 2021 at 12:01:08 PM UTC-7 Travis Illig wrote: > I've verified: > > - v2.20.1 is the last version where the mTLS scraping works. > - It doesn't matter which Docker registry you pull from (Docker Hub or > quay.io - I've sometimes seen different "versions" of containers based > on registry). > > Looking at the release notes for v2.21.0 > <https://github.com/prometheus/prometheus/releases/tag/v2.21.0> it > appears there's a new version of Go used for compilation which includes > some changes on how certificates are handled > <https://golang.org/doc/go1.15#commonname>. Unclear if this is what I'm > hitting, but it seems worth looking into. > > On Tuesday, July 6, 2021 at 11:02:56 AM UTC-7 Travis Illig wrote: > >> I'm deploying Prometheus using the Helm chart >> <https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus> >> >> and I have it configured to scrape Istio mTLS-secured pods using the TLS >> settings specified >> <https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings> >> by the Istio team to do so. Basically what this amounts to is: >> >> - Add the Istio sidecar to the Prometheus instance but disable all >> traffic proxying - you just want to get the certificates from it. >> - Mount the certificates into the Prometheus container. >> - Set up your scrape configuration to use the certificates when >> scraping Istio-enabled pods. >> >> The YAML for the scrape configuration looks like this: >> >> - job_name: "kubernetes-pods-istio-secure" >> scheme: https >> tls_config: >> ca_file: /etc/istio-certs/root-cert.pem >> cert_file: /etc/istio-certs/cert-chain.pem >> key_file: /etc/istio-certs/key.pem >> insecure_skip_verify: true >> >> *This totally works using Prometheus v2.20.1* packaged as >> `prom/prometheus` from Docker Hub. >> >> *This fails on Prometheus v2.28.0* packaged as ` >> quay.io/prometheus/prometheus` <http://quay.io/prometheus/prometheus>. >> Instead of getting a successful scrape, I get "connection reset by peer." >> I've validated the files are there and properly mounted; they have the >> expected contents; and there are no Prometheus log messages to indicate >> anything is amiss. >> >> I've been rolling back slowly to see where it starts working again. I've >> tried v2.26.0 and it still fails. I thought I'd drop a note in here to see >> if anyone knows what's up. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/54830641-bd69-47dc-ad26-389b35b42758n%40googlegroups.com.
