You could also prove this with prometheus 2.13.0, by setting environment variable GODEBUG=tls13=0 and see if it starts to work.
(That workaround was removed in later versions of go, so it's not usable in production - but at least it will demonstrate whether this is the problem or not) On Monday, 12 September 2022 at 10:06:54 UTC+1 Brian Candler wrote: > One problem that affected me was that go version 1.15+ removed the ability > to use a certificate which has only "commonName" > <https://golang.org/doc/go1.15#commonname>. It now requires > "subjectAltName" to be present. That, I believe, would have affected > prometheus 2.21.0 > <https://github.com/prometheus/prometheus/releases/tag/v2.21.0> onwards, > so it doesn't explain your change at 2.13.0. > > Prometheus 2.12.0 was built with go1.12.8, and 2.13.0 was built > with go1.13.1, so it could be a similar issue. > > Now, looking at the go1.13 release notes > <https://go.dev/doc/go1.13#tls_1_3>, I see that TLS 1.3 was enabled by > default then - there's more information about go's TLS 1.3 support in the > go1.12 > release notes <https://go.dev/doc/go1.12#tls_1_3> when it was made > available but opt-in only. > > Perhaps that's the issue - your target server has a broken TLS 1.3 > implementation? If so, forcing TLS 1.2 might be a workaround. > Unfortunately, it looks like prometheus' tls_config > <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config> > in > scrape jobs allows setting a minimum TLS version, but not a maximum :-( > > That's weird, because when configuring prometheus' own https endpoint > <https://prometheus.io/docs/prometheus/latest/configuration/https/>, you > can select both min and max versions. > > Maybe disabling TLS 1.3 in the exporter itself is an option? > > On Monday, 12 September 2022 at 09:23:35 UTC+1 [email protected] wrote: > >> Hi Brian, >> thanks for these tips. I already tried those before my post, but they >> didn't help. >> In the meantime I did some more investigation. The servers creating the >> metrics are Payara Micro 5.2021.10 >> The Prometheus (actually Thanos) is v2.38.0. Strangly it works with >> 2.12.0 (old on prem installation that got replaced), and is broken with >> every version starting at 2.13.0. >> Should be quite obvious to figure out, but I don't see any changes in >> 2.13.0 that I would pinpoint this too. >> >> I asked the guys running the servers if they could look into their >> config. e.g. disabling h2 didn't help either. >> >> Very strange issue :( >> >> Cu >> Markus >> >> On Friday, 9 September 2022 at 15:10:46 UTC+2 Brian Candler wrote: >> >>> Note: the ability to disable http2 via scrape config was only added in >>> v2.35.0 <https://github.com/prometheus/prometheus/releases/tag/v2.35.0>. >>> >>> For an older version, you could try environment variables >>> DISABLE_HTTP2=1 >>> and/or >>> GODEBUG=http2client=0 >>> >>> On Friday, 9 September 2022 at 13:18:42 UTC+1 Brian Candler wrote: >>> >>>> What prometheus version are you using? I'd suggest v2.37.0 (2.37 is an >>>> LTS release branch) >>>> >>>> "promtool debug metrics" doesn't seem to have many options, but you >>>> could try in your prometheus scrape config: >>>> >>>> enable_http2: false >>>> >>>> and/or >>>> >>>> tls_config: >>>> insecure_skip_verify: true >>>> >>>> to try and narrow down the problem. >>>> >>>> On Friday, 9 September 2022 at 10:19:40 UTC+1 [email protected] wrote: >>>> >>>>> promtool debug metrics https://myserver:8181 >>>>> >>>>> collecting: https://myserver:8181/metrics >>>>> >>>>> 2022/09/09 11:15:05 http2: Transport failed to get client conn for >>>>> myserver:8181: http2: no cached connection was available >>>>> >>>>> error completing debug command: error executing HTTP request: Get " >>>>> https://myserver:8181/metrics": EOF >>>>> >>>>> >>>>> If I open the url in a browser or with curl I get proper response :( >>>>> Any hint what I might be missing? testssl.sh works fine too >>>>> >>>>> >>>>> >>>>> On Friday, 9 September 2022 at 10:37:13 UTC+2 Markus Glück wrote: >>>>> >>>>>> Hi guys, >>>>>> I currently trying to debug a similar issue. I am also getting EOF >>>>>> only as error in debug level from scrape manager. I had the same issue >>>>>> locally with curl and LibreSSL on macOS. Updating the curl version to >>>>>> use >>>>>> openSSL fixed it. So my assumption it's something related to SSL/TLS. My >>>>>> working curl verbose output gives me this in regards to TLS: >>>>>> >>>>>> * ALPN: offers h2 >>>>>> >>>>>> * ALPN: offers http/1.1 >>>>>> >>>>>> } [5 bytes data] >>>>>> >>>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>>> >>>>>> } [512 bytes data] >>>>>> >>>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>>> >>>>>> { [193 bytes data] >>>>>> >>>>>> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): >>>>>> >>>>>> } [1 bytes data] >>>>>> >>>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>>> >>>>>> } [512 bytes data] >>>>>> >>>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>>> >>>>>> { [155 bytes data] >>>>>> >>>>>> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): >>>>>> >>>>>> { [51 bytes data] >>>>>> >>>>>> * TLSv1.3 (IN), TLS handshake, Certificate (11): >>>>>> >>>>>> { [4943 bytes data] >>>>>> >>>>>> * TLSv1.3 (IN), TLS handshake, CERT verify (15): >>>>>> >>>>>> { [520 bytes data] >>>>>> >>>>>> * TLSv1.3 (IN), TLS handshake, Finished (20): >>>>>> >>>>>> { [52 bytes data] >>>>>> >>>>>> * TLSv1.3 (OUT), TLS handshake, Finished (20): >>>>>> >>>>>> } [52 bytes data] >>>>>> >>>>>> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 >>>>>> >>>>>> * ALPN: server accepted h2 >>>>>> >>>>>> >>>>>> but HOW do I get more info from Prometheus? >>>>>> >>>>>> Thx, >>>>>> >>>>>> Markus >>>>>> >>>>>> On Thursday, 14 July 2022 at 16:25:36 UTC+2 Brian Candler wrote: >>>>>> >>>>>>> The straightforward way to debug this by doing a scrape by hand: >>>>>>> it's just a HTTP request. >>>>>>> >>>>>>> curl -g 'https://blah.local:9126/metrics' >>>>>>> >>>>>>> Add flag '-v' for more debugging if required (e.g. response headers >>>>>>> may give you an extra clue). Once you're able to scrape the exporter >>>>>>> with >>>>>>> curl, then prometheus should be able to talk to it too. >>>>>>> >>>>>>> Given that it's https, if you get a certificate error then you can >>>>>>> add flag '-k' to skip certificate verification. If that turns out to >>>>>>> be >>>>>>> the problem, then there are extra flags you can pass to curl, e.g. to >>>>>>> pass >>>>>>> the CA root certificate (if the target's cert was signed by a CA that's >>>>>>> not >>>>>>> in the system trust store). Once you've got all that working, you can >>>>>>> make >>>>>>> the corresponding changes to prometheus' tls_config. >>>>>>> >>>>>>> On Thursday, 14 July 2022 at 14:46:24 UTC+1 [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Team, >>>>>>>> >>>>>>>> I have promethus installed on kubernet and add scraping targets >>>>>>>> from cmdb using http_sd_config. >>>>>>>> >>>>>>>> I am getting EOF error while i add targets. And it’s show down. >>>>>>>> >>>>>>>> Can someone please help me what is the issue. >>>>>>>> >>>>>>>> For your reference attached screenshot of error. >>>>>>>> >>>>>>>> Thanks and regards >>>>>>>> Ritesh patel >>>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/449b8ce5-b262-4a95-a564-5057bc46d60bn%40googlegroups.com.
