You're da man! :) GODEBUG=tls13=0 in 2.13.0 works. so its an incompatibility between newer GO versions and Payara. Now I have to push the server owners to disable the TLS1.3 for the metrics endpoints, since I need to use a more recent payara (not able to rollback the data and also High CVE in GoVersion of 2.35.0 or so). But thx very much to give the definite hint that proved the causing issue!
CU Markus On Monday, 12 September 2022 at 11:12:43 UTC+2 Brian Candler wrote: > You could also prove this with prometheus 2.13.0, by setting environment > variable GODEBUG=tls13=0 and see if it starts to work. > > (That workaround was removed in later versions of go, so it's not usable > in production - but at least it will demonstrate whether this is the > problem or not) > > On Monday, 12 September 2022 at 10:06:54 UTC+1 Brian Candler wrote: > >> One problem that affected me was that go version 1.15+ removed the >> ability to use a certificate which has only "commonName" >> <https://golang.org/doc/go1.15#commonname>. It now requires >> "subjectAltName" to be present. That, I believe, would have affected >> prometheus 2.21.0 >> <https://github.com/prometheus/prometheus/releases/tag/v2.21.0> onwards, >> so it doesn't explain your change at 2.13.0. >> >> Prometheus 2.12.0 was built with go1.12.8, and 2.13.0 was built >> with go1.13.1, so it could be a similar issue. >> >> Now, looking at the go1.13 release notes >> <https://go.dev/doc/go1.13#tls_1_3>, I see that TLS 1.3 was enabled by >> default then - there's more information about go's TLS 1.3 support in the >> go1.12 >> release notes <https://go.dev/doc/go1.12#tls_1_3> when it was made >> available but opt-in only. >> >> Perhaps that's the issue - your target server has a broken TLS 1.3 >> implementation? If so, forcing TLS 1.2 might be a workaround. >> Unfortunately, it looks like prometheus' tls_config >> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config> >> in >> scrape jobs allows setting a minimum TLS version, but not a maximum :-( >> >> That's weird, because when configuring prometheus' own https endpoint >> <https://prometheus.io/docs/prometheus/latest/configuration/https/>, you >> can select both min and max versions. >> >> Maybe disabling TLS 1.3 in the exporter itself is an option? >> >> On Monday, 12 September 2022 at 09:23:35 UTC+1 [email protected] wrote: >> >>> Hi Brian, >>> thanks for these tips. I already tried those before my post, but they >>> didn't help. >>> In the meantime I did some more investigation. The servers creating the >>> metrics are Payara Micro 5.2021.10 >>> The Prometheus (actually Thanos) is v2.38.0. Strangly it works with >>> 2.12.0 (old on prem installation that got replaced), and is broken with >>> every version starting at 2.13.0. >>> Should be quite obvious to figure out, but I don't see any changes in >>> 2.13.0 that I would pinpoint this too. >>> >>> I asked the guys running the servers if they could look into their >>> config. e.g. disabling h2 didn't help either. >>> >>> Very strange issue :( >>> >>> Cu >>> Markus >>> >>> On Friday, 9 September 2022 at 15:10:46 UTC+2 Brian Candler wrote: >>> >>>> Note: the ability to disable http2 via scrape config was only added in >>>> v2.35.0 <https://github.com/prometheus/prometheus/releases/tag/v2.35.0> >>>> . >>>> >>>> For an older version, you could try environment variables >>>> DISABLE_HTTP2=1 >>>> and/or >>>> GODEBUG=http2client=0 >>>> >>>> On Friday, 9 September 2022 at 13:18:42 UTC+1 Brian Candler wrote: >>>> >>>>> What prometheus version are you using? I'd suggest v2.37.0 (2.37 is >>>>> an LTS release branch) >>>>> >>>>> "promtool debug metrics" doesn't seem to have many options, but you >>>>> could try in your prometheus scrape config: >>>>> >>>>> enable_http2: false >>>>> >>>>> and/or >>>>> >>>>> tls_config: >>>>> insecure_skip_verify: true >>>>> >>>>> to try and narrow down the problem. >>>>> >>>>> On Friday, 9 September 2022 at 10:19:40 UTC+1 [email protected] wrote: >>>>> >>>>>> promtool debug metrics https://myserver:8181 >>>>>> >>>>>> collecting: https://myserver:8181/metrics >>>>>> >>>>>> 2022/09/09 11:15:05 http2: Transport failed to get client conn for >>>>>> myserver:8181: http2: no cached connection was available >>>>>> >>>>>> error completing debug command: error executing HTTP request: Get " >>>>>> https://myserver:8181/metrics": EOF >>>>>> >>>>>> >>>>>> If I open the url in a browser or with curl I get proper response :( >>>>>> Any hint what I might be missing? testssl.sh works fine too >>>>>> >>>>>> >>>>>> >>>>>> On Friday, 9 September 2022 at 10:37:13 UTC+2 Markus Glück wrote: >>>>>> >>>>>>> Hi guys, >>>>>>> I currently trying to debug a similar issue. I am also getting EOF >>>>>>> only as error in debug level from scrape manager. I had the same issue >>>>>>> locally with curl and LibreSSL on macOS. Updating the curl version to >>>>>>> use >>>>>>> openSSL fixed it. So my assumption it's something related to SSL/TLS. >>>>>>> My >>>>>>> working curl verbose output gives me this in regards to TLS: >>>>>>> >>>>>>> * ALPN: offers h2 >>>>>>> >>>>>>> * ALPN: offers http/1.1 >>>>>>> >>>>>>> } [5 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>>>> >>>>>>> } [512 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>>>> >>>>>>> { [193 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): >>>>>>> >>>>>>> } [1 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>>>> >>>>>>> } [512 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>>>> >>>>>>> { [155 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): >>>>>>> >>>>>>> { [51 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (IN), TLS handshake, Certificate (11): >>>>>>> >>>>>>> { [4943 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (IN), TLS handshake, CERT verify (15): >>>>>>> >>>>>>> { [520 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (IN), TLS handshake, Finished (20): >>>>>>> >>>>>>> { [52 bytes data] >>>>>>> >>>>>>> * TLSv1.3 (OUT), TLS handshake, Finished (20): >>>>>>> >>>>>>> } [52 bytes data] >>>>>>> >>>>>>> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 >>>>>>> >>>>>>> * ALPN: server accepted h2 >>>>>>> >>>>>>> >>>>>>> but HOW do I get more info from Prometheus? >>>>>>> >>>>>>> Thx, >>>>>>> >>>>>>> Markus >>>>>>> >>>>>>> On Thursday, 14 July 2022 at 16:25:36 UTC+2 Brian Candler wrote: >>>>>>> >>>>>>>> The straightforward way to debug this by doing a scrape by hand: >>>>>>>> it's just a HTTP request. >>>>>>>> >>>>>>>> curl -g 'https://blah.local:9126/metrics' >>>>>>>> >>>>>>>> Add flag '-v' for more debugging if required (e.g. response headers >>>>>>>> may give you an extra clue). Once you're able to scrape the exporter >>>>>>>> with >>>>>>>> curl, then prometheus should be able to talk to it too. >>>>>>>> >>>>>>>> Given that it's https, if you get a certificate error then you can >>>>>>>> add flag '-k' to skip certificate verification. If that turns out to >>>>>>>> be >>>>>>>> the problem, then there are extra flags you can pass to curl, e.g. to >>>>>>>> pass >>>>>>>> the CA root certificate (if the target's cert was signed by a CA >>>>>>>> that's not >>>>>>>> in the system trust store). Once you've got all that working, you can >>>>>>>> make >>>>>>>> the corresponding changes to prometheus' tls_config. >>>>>>>> >>>>>>>> On Thursday, 14 July 2022 at 14:46:24 UTC+1 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello Team, >>>>>>>>> >>>>>>>>> I have promethus installed on kubernet and add scraping targets >>>>>>>>> from cmdb using http_sd_config. >>>>>>>>> >>>>>>>>> I am getting EOF error while i add targets. And it’s show down. >>>>>>>>> >>>>>>>>> Can someone please help me what is the issue. >>>>>>>>> >>>>>>>>> For your reference attached screenshot of error. >>>>>>>>> >>>>>>>>> Thanks and regards >>>>>>>>> Ritesh patel >>>>>>>>> >>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9894cb02-d153-4282-87ef-e513457c56a9n%40googlegroups.com.
