One problem that affected me was that go version 1.15+ removed the ability to use a certificate which has only "commonName" <https://golang.org/doc/go1.15#commonname>. It now requires "subjectAltName" to be present. That, I believe, would have affected prometheus 2.21.0 <https://github.com/prometheus/prometheus/releases/tag/v2.21.0> onwards, so it doesn't explain your change at 2.13.0.
Prometheus 2.12.0 was built with go1.12.8, and 2.13.0 was built with go1.13.1, so it could be a similar issue. Now, looking at the go1.13 release notes <https://go.dev/doc/go1.13#tls_1_3>, I see that TLS 1.3 was enabled by default then - there's more information about go's TLS 1.3 support in the go1.12 release notes <https://go.dev/doc/go1.12#tls_1_3> when it was made available but opt-in only. Perhaps that's the issue - your target server has a broken TLS 1.3 implementation? If so, forcing TLS 1.2 might be a workaround. Unfortunately, it looks like prometheus' tls_config <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config> in scrape jobs allows setting a minimum TLS version, but not a maximum :-( That's weird, because when configuring prometheus' own https endpoint <https://prometheus.io/docs/prometheus/latest/configuration/https/>, you can select both min and max versions. Maybe disabling TLS 1.3 in the exporter itself is an option? On Monday, 12 September 2022 at 09:23:35 UTC+1 [email protected] wrote: > Hi Brian, > thanks for these tips. I already tried those before my post, but they > didn't help. > In the meantime I did some more investigation. The servers creating the > metrics are Payara Micro 5.2021.10 > The Prometheus (actually Thanos) is v2.38.0. Strangly it works with 2.12.0 > (old on prem installation that got replaced), and is broken with every > version starting at 2.13.0. > Should be quite obvious to figure out, but I don't see any changes in > 2.13.0 that I would pinpoint this too. > > I asked the guys running the servers if they could look into their config. > e.g. disabling h2 didn't help either. > > Very strange issue :( > > Cu > Markus > > On Friday, 9 September 2022 at 15:10:46 UTC+2 Brian Candler wrote: > >> Note: the ability to disable http2 via scrape config was only added in >> v2.35.0 <https://github.com/prometheus/prometheus/releases/tag/v2.35.0>. >> >> For an older version, you could try environment variables >> DISABLE_HTTP2=1 >> and/or >> GODEBUG=http2client=0 >> >> On Friday, 9 September 2022 at 13:18:42 UTC+1 Brian Candler wrote: >> >>> What prometheus version are you using? I'd suggest v2.37.0 (2.37 is an >>> LTS release branch) >>> >>> "promtool debug metrics" doesn't seem to have many options, but you >>> could try in your prometheus scrape config: >>> >>> enable_http2: false >>> >>> and/or >>> >>> tls_config: >>> insecure_skip_verify: true >>> >>> to try and narrow down the problem. >>> >>> On Friday, 9 September 2022 at 10:19:40 UTC+1 [email protected] wrote: >>> >>>> promtool debug metrics https://myserver:8181 >>>> >>>> collecting: https://myserver:8181/metrics >>>> >>>> 2022/09/09 11:15:05 http2: Transport failed to get client conn for >>>> myserver:8181: http2: no cached connection was available >>>> >>>> error completing debug command: error executing HTTP request: Get " >>>> https://myserver:8181/metrics": EOF >>>> >>>> >>>> If I open the url in a browser or with curl I get proper response :( >>>> Any hint what I might be missing? testssl.sh works fine too >>>> >>>> >>>> >>>> On Friday, 9 September 2022 at 10:37:13 UTC+2 Markus Glück wrote: >>>> >>>>> Hi guys, >>>>> I currently trying to debug a similar issue. I am also getting EOF >>>>> only as error in debug level from scrape manager. I had the same issue >>>>> locally with curl and LibreSSL on macOS. Updating the curl version to use >>>>> openSSL fixed it. So my assumption it's something related to SSL/TLS. My >>>>> working curl verbose output gives me this in regards to TLS: >>>>> >>>>> * ALPN: offers h2 >>>>> >>>>> * ALPN: offers http/1.1 >>>>> >>>>> } [5 bytes data] >>>>> >>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>> >>>>> } [512 bytes data] >>>>> >>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>> >>>>> { [193 bytes data] >>>>> >>>>> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): >>>>> >>>>> } [1 bytes data] >>>>> >>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>> >>>>> } [512 bytes data] >>>>> >>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>> >>>>> { [155 bytes data] >>>>> >>>>> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): >>>>> >>>>> { [51 bytes data] >>>>> >>>>> * TLSv1.3 (IN), TLS handshake, Certificate (11): >>>>> >>>>> { [4943 bytes data] >>>>> >>>>> * TLSv1.3 (IN), TLS handshake, CERT verify (15): >>>>> >>>>> { [520 bytes data] >>>>> >>>>> * TLSv1.3 (IN), TLS handshake, Finished (20): >>>>> >>>>> { [52 bytes data] >>>>> >>>>> * TLSv1.3 (OUT), TLS handshake, Finished (20): >>>>> >>>>> } [52 bytes data] >>>>> >>>>> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 >>>>> >>>>> * ALPN: server accepted h2 >>>>> >>>>> >>>>> but HOW do I get more info from Prometheus? >>>>> >>>>> Thx, >>>>> >>>>> Markus >>>>> >>>>> On Thursday, 14 July 2022 at 16:25:36 UTC+2 Brian Candler wrote: >>>>> >>>>>> The straightforward way to debug this by doing a scrape by hand: it's >>>>>> just a HTTP request. >>>>>> >>>>>> curl -g 'https://blah.local:9126/metrics' >>>>>> >>>>>> Add flag '-v' for more debugging if required (e.g. response headers >>>>>> may give you an extra clue). Once you're able to scrape the exporter >>>>>> with >>>>>> curl, then prometheus should be able to talk to it too. >>>>>> >>>>>> Given that it's https, if you get a certificate error then you can >>>>>> add flag '-k' to skip certificate verification. If that turns out to be >>>>>> the problem, then there are extra flags you can pass to curl, e.g. to >>>>>> pass >>>>>> the CA root certificate (if the target's cert was signed by a CA that's >>>>>> not >>>>>> in the system trust store). Once you've got all that working, you can >>>>>> make >>>>>> the corresponding changes to prometheus' tls_config. >>>>>> >>>>>> On Thursday, 14 July 2022 at 14:46:24 UTC+1 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Hello Team, >>>>>>> >>>>>>> I have promethus installed on kubernet and add scraping targets from >>>>>>> cmdb using http_sd_config. >>>>>>> >>>>>>> I am getting EOF error while i add targets. And it’s show down. >>>>>>> >>>>>>> Can someone please help me what is the issue. >>>>>>> >>>>>>> For your reference attached screenshot of error. >>>>>>> >>>>>>> Thanks and regards >>>>>>> Ritesh patel >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f5062418-c66a-423d-937e-44c2646f1fa1n%40googlegroups.com.
