ok, we found a solution. One of the servers worked all the time. Turns out it had a different JDK installed (OpenJDK Zulu vs normal OpenJDK, both 11). Using the Zulu JDK on the failing ones fixes the issue.
On Monday, 12 September 2022 at 14:26:25 UTC+2 Markus Glück wrote: > You're da man! :) GODEBUG=tls13=0 in 2.13.0 works. so its an > incompatibility between newer GO versions and Payara. > Now I have to push the server owners to disable the TLS1.3 for the metrics > endpoints, since I need to use a more recent payara (not able to rollback > the data and also High CVE in GoVersion of 2.35.0 or so). > But thx very much to give the definite hint that proved the causing issue! > > CU > Markus > > On Monday, 12 September 2022 at 11:12:43 UTC+2 Brian Candler wrote: > >> You could also prove this with prometheus 2.13.0, by setting environment >> variable GODEBUG=tls13=0 and see if it starts to work. >> >> (That workaround was removed in later versions of go, so it's not usable >> in production - but at least it will demonstrate whether this is the >> problem or not) >> >> On Monday, 12 September 2022 at 10:06:54 UTC+1 Brian Candler wrote: >> >>> One problem that affected me was that go version 1.15+ removed the >>> ability to use a certificate which has only "commonName" >>> <https://golang.org/doc/go1.15#commonname>. It now requires >>> "subjectAltName" to be present. That, I believe, would have affected >>> prometheus 2.21.0 >>> <https://github.com/prometheus/prometheus/releases/tag/v2.21.0> >>> onwards, so it doesn't explain your change at 2.13.0. >>> >>> Prometheus 2.12.0 was built with go1.12.8, and 2.13.0 was built >>> with go1.13.1, so it could be a similar issue. >>> >>> Now, looking at the go1.13 release notes >>> <https://go.dev/doc/go1.13#tls_1_3>, I see that TLS 1.3 was enabled by >>> default then - there's more information about go's TLS 1.3 support in the >>> go1.12 >>> release notes <https://go.dev/doc/go1.12#tls_1_3> when it was made >>> available but opt-in only. >>> >>> Perhaps that's the issue - your target server has a broken TLS 1.3 >>> implementation? If so, forcing TLS 1.2 might be a workaround. >>> Unfortunately, it looks like prometheus' tls_config >>> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config> >>> in >>> scrape jobs allows setting a minimum TLS version, but not a maximum :-( >>> >>> That's weird, because when configuring prometheus' own https endpoint >>> <https://prometheus.io/docs/prometheus/latest/configuration/https/>, >>> you can select both min and max versions. >>> >>> Maybe disabling TLS 1.3 in the exporter itself is an option? >>> >>> On Monday, 12 September 2022 at 09:23:35 UTC+1 [email protected] wrote: >>> >>>> Hi Brian, >>>> thanks for these tips. I already tried those before my post, but they >>>> didn't help. >>>> In the meantime I did some more investigation. The servers creating the >>>> metrics are Payara Micro 5.2021.10 >>>> The Prometheus (actually Thanos) is v2.38.0. Strangly it works with >>>> 2.12.0 (old on prem installation that got replaced), and is broken with >>>> every version starting at 2.13.0. >>>> Should be quite obvious to figure out, but I don't see any changes in >>>> 2.13.0 that I would pinpoint this too. >>>> >>>> I asked the guys running the servers if they could look into their >>>> config. e.g. disabling h2 didn't help either. >>>> >>>> Very strange issue :( >>>> >>>> Cu >>>> Markus >>>> >>>> On Friday, 9 September 2022 at 15:10:46 UTC+2 Brian Candler wrote: >>>> >>>>> Note: the ability to disable http2 via scrape config was only added in >>>>> v2.35.0 >>>>> <https://github.com/prometheus/prometheus/releases/tag/v2.35.0>. >>>>> >>>>> For an older version, you could try environment variables >>>>> DISABLE_HTTP2=1 >>>>> and/or >>>>> GODEBUG=http2client=0 >>>>> >>>>> On Friday, 9 September 2022 at 13:18:42 UTC+1 Brian Candler wrote: >>>>> >>>>>> What prometheus version are you using? I'd suggest v2.37.0 (2.37 is >>>>>> an LTS release branch) >>>>>> >>>>>> "promtool debug metrics" doesn't seem to have many options, but you >>>>>> could try in your prometheus scrape config: >>>>>> >>>>>> enable_http2: false >>>>>> >>>>>> and/or >>>>>> >>>>>> tls_config: >>>>>> insecure_skip_verify: true >>>>>> >>>>>> to try and narrow down the problem. >>>>>> >>>>>> On Friday, 9 September 2022 at 10:19:40 UTC+1 [email protected] >>>>>> wrote: >>>>>> >>>>>>> promtool debug metrics https://myserver:8181 >>>>>>> >>>>>>> collecting: https://myserver:8181/metrics >>>>>>> >>>>>>> 2022/09/09 11:15:05 http2: Transport failed to get client conn for >>>>>>> myserver:8181: http2: no cached connection was available >>>>>>> >>>>>>> error completing debug command: error executing HTTP request: Get " >>>>>>> https://myserver:8181/metrics": EOF >>>>>>> >>>>>>> >>>>>>> If I open the url in a browser or with curl I get proper response :( >>>>>>> Any hint what I might be missing? testssl.sh works fine too >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Friday, 9 September 2022 at 10:37:13 UTC+2 Markus Glück wrote: >>>>>>> >>>>>>>> Hi guys, >>>>>>>> I currently trying to debug a similar issue. I am also getting EOF >>>>>>>> only as error in debug level from scrape manager. I had the same issue >>>>>>>> locally with curl and LibreSSL on macOS. Updating the curl version to >>>>>>>> use >>>>>>>> openSSL fixed it. So my assumption it's something related to SSL/TLS. >>>>>>>> My >>>>>>>> working curl verbose output gives me this in regards to TLS: >>>>>>>> >>>>>>>> * ALPN: offers h2 >>>>>>>> >>>>>>>> * ALPN: offers http/1.1 >>>>>>>> >>>>>>>> } [5 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>>>>> >>>>>>>> } [512 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>>>>> >>>>>>>> { [193 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): >>>>>>>> >>>>>>>> } [1 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (OUT), TLS handshake, Client hello (1): >>>>>>>> >>>>>>>> } [512 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (IN), TLS handshake, Server hello (2): >>>>>>>> >>>>>>>> { [155 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): >>>>>>>> >>>>>>>> { [51 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (IN), TLS handshake, Certificate (11): >>>>>>>> >>>>>>>> { [4943 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (IN), TLS handshake, CERT verify (15): >>>>>>>> >>>>>>>> { [520 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (IN), TLS handshake, Finished (20): >>>>>>>> >>>>>>>> { [52 bytes data] >>>>>>>> >>>>>>>> * TLSv1.3 (OUT), TLS handshake, Finished (20): >>>>>>>> >>>>>>>> } [52 bytes data] >>>>>>>> >>>>>>>> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 >>>>>>>> >>>>>>>> * ALPN: server accepted h2 >>>>>>>> >>>>>>>> >>>>>>>> but HOW do I get more info from Prometheus? >>>>>>>> >>>>>>>> Thx, >>>>>>>> >>>>>>>> Markus >>>>>>>> >>>>>>>> On Thursday, 14 July 2022 at 16:25:36 UTC+2 Brian Candler wrote: >>>>>>>> >>>>>>>>> The straightforward way to debug this by doing a scrape by hand: >>>>>>>>> it's just a HTTP request. >>>>>>>>> >>>>>>>>> curl -g 'https://blah.local:9126/metrics' >>>>>>>>> >>>>>>>>> Add flag '-v' for more debugging if required (e.g. response >>>>>>>>> headers may give you an extra clue). Once you're able to scrape the >>>>>>>>> exporter with curl, then prometheus should be able to talk to it too. >>>>>>>>> >>>>>>>>> Given that it's https, if you get a certificate error then you can >>>>>>>>> add flag '-k' to skip certificate verification. If that turns out to >>>>>>>>> be >>>>>>>>> the problem, then there are extra flags you can pass to curl, e.g. to >>>>>>>>> pass >>>>>>>>> the CA root certificate (if the target's cert was signed by a CA >>>>>>>>> that's not >>>>>>>>> in the system trust store). Once you've got all that working, you >>>>>>>>> can make >>>>>>>>> the corresponding changes to prometheus' tls_config. >>>>>>>>> >>>>>>>>> On Thursday, 14 July 2022 at 14:46:24 UTC+1 [email protected] >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello Team, >>>>>>>>>> >>>>>>>>>> I have promethus installed on kubernet and add scraping targets >>>>>>>>>> from cmdb using http_sd_config. >>>>>>>>>> >>>>>>>>>> I am getting EOF error while i add targets. And it’s show down. >>>>>>>>>> >>>>>>>>>> Can someone please help me what is the issue. >>>>>>>>>> >>>>>>>>>> For your reference attached screenshot of error. >>>>>>>>>> >>>>>>>>>> Thanks and regards >>>>>>>>>> Ritesh patel >>>>>>>>>> >>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2f6c8f1f-b391-4dd8-84ba-d8acaba29599n%40googlegroups.com.
