Thanks Bruno,
My first step would be to setup a custom log format that uses log
converters with the appropriate fetches [1]:
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS
%tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[ssl_fc_protocol]
%[ssl_fc_cipher]"
For some reason, the error message doesn't log any of the log-format
fetches. This is all I get in the log.
Sep 1 14:43:22 haproxy haproxy[7843]: ::ffff:<redacted>
[01/Sep/2020:14:43:22.754] www-https/1: SSL handshake failure
May I suggest trying out req.ssl_sni instead of ssl_fc_sni while
troubleshooting the issue [4]?
Tried with the inspect delay. Same result. I'll give req.ssl_sni a test,
but it seems unlikely to be involved here no? If the sni was failing to
match, why would it give a handshake error instead of the usual
strict-sni rejection when seeing a non-existing sni?
Chrome has some strange probing logic that causes false positives and
clogs the logs [5] and has some interesting side-effects [6]. I'd
check the wireshark traffic to see which requests are triggering the
errors. Also, is Chrome the only browser that is affected by this?
Chrome is the only browser I experienced the long delays in connections
with, but I'm getting the handshake errors from a bunch of different
clients (though I expect some of this to be natural/expected, its the
rate its occuring thats concerning as its much higher than the 1.x
series we migrated from)
I'll see if I can wireshark this up, but its hard given session keys and
that I'm unable to reliably reproduce the error on any given client --
but it is showing up reliably on my production traffic load and there's
lots of failures hitting the log.
I take it that means theres no internal debug logging for the tls errors
that we can just expose via logfile?
Thanks for the help,
--
Kevin
On 2020-09-01 10:59 a.m., Bruno Henc wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, September 1, 2020 6:57 PM, Kevin McArthur
<ke...@stormtide.ca> wrote:
Hi haproxy
I'm wondering if there is any way to debug the error message
"www-https/1: SSL handshake failure"? I've tried increasing log
levels to debug etc, but nothing seems to log about why the failure
occurred
My first step would be to setup a custom log format that uses log
converters with the appropriate fetches [1]:
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS
%tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[ssl_fc_protocol]
%[ssl_fc_cipher]"
However, there is no substitute for a tcpdump that captures the TLS
handshake. Wireshark is your best friend, especially if there are
firewalls in front of HAProxy -
in which case I would recommend looking at ssl_fc_session_key option
[3] to be able to
decrypt the session in wireshark. Preferably, consider limiting for
which clients
the key is logged for (to protect regular clients). Something along
these lines may do the trick: http-request set-var(txn.session_key)
ssl_fc_session_key if { src -i 10.10.10.10 }
Then add %[var(txn.session_key)] to the log format.
We've had a strange regression when upgrading from the 1.x series
that presented as very long 'Establishing SSL Connection' times in
Chrome, but the connections would eventually go through and load the
page with an expected cipher etc.
May I suggest trying out req.ssl_sni instead of ssl_fc_sni while
troubleshooting the issue [4]? The latter usually requires a
tcp-request inspect delay 5s or similar line (in tcp mode) to be used
reliably. The former is set after the session has been decrypted so
it's 100% available once you hit the backend rule processing.
Chrome has some strange probing logic that causes false positives and
clogs the logs [5] and has some interesting side-effects [6]. I'd
check the wireshark traffic to see which requests are triggering the
errors. Also, is Chrome the only browser that is affected by this?
Just my 2 cents.
Best regards,
Bruno H.
[1]
https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#7.3.4-ssl_fc_cipher
[2] https://www.haproxy.com/blog/introduction-to-haproxy-logging/
[3] https://www.haproxy.com/blog/announcing-haproxy-2-2/
[4]
https://www.haproxy.com/documentation/hapee/latest/deployment-guides/tls-infrastructure/
[5]
https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4-option%20http-ignore-probes
[6] https://www.theregister.com/2020/08/21/chromiums_dns_network/
