Re: [HAP 2.3.8] Is there a way to see why "" and "SSL handshake failure" happens
On 27.03.21 12:01, Lukas Tribus wrote: Hello, On Sat, 27 Mar 2021 at 11:52, Aleksandar Lazic wrote: Hi. I have a lot of such entries in my logs. ``` Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 [27/Mar/2021:11:48:20.523] https-in~ https-in/ -1/-1/-1/-1/0 0 0 - - PR-- 1041/1011/0/0/0 0/0 "" Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 [27/Mar/2021:11:48:20.523] https-in~ https-in/ -1/-1/-1/-1/0 0 0 - - PR-- 1041/1011/0/0/0 0/0 "" Use show errors on the admin socket: https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.3-show%20errors Thanks. Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166 [27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166 [27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure That's currently a pain point: https://github.com/haproxy/haproxy/issues/693 Thanks. Lukas Regards Alex
Re: [HAP 2.3.8] Is there a way to see why "" and "SSL handshake failure" happens
Hello, On Sat, 27 Mar 2021 at 11:52, Aleksandar Lazic wrote: > > Hi. > > I have a lot of such entries in my logs. > > ``` > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 > [27/Mar/2021:11:48:20.523] https-in~ https-in/ -1/-1/-1/-1/0 0 0 - - > PR-- 1041/1011/0/0/0 0/0 "" > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 > [27/Mar/2021:11:48:20.523] https-in~ https-in/ -1/-1/-1/-1/0 0 0 - - > PR-- 1041/1011/0/0/0 0/0 "" Use show errors on the admin socket: https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.3-show%20errors > Mar 27 11:48:20 lb1 haproxy[14556]: ::ffff::23166 > [27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166 > [27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure That's currently a pain point: https://github.com/haproxy/haproxy/issues/693 Lukas
[HAP 2.3.8] Is there a way to see why "" and "SSL handshake failure" happens
Hi.
I have a lot of such entries in my logs.
```
Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 [27/Mar/2021:11:48:20.523] https-in~
https-in/ -1/-1/-1/-1/0 0 0 - - PR-- 1041/1011/0/0/0 0/0 ""
Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 [27/Mar/2021:11:48:20.523] https-in~
https-in/ -1/-1/-1/-1/0 0 0 - - PR-- 1041/1011/0/0/0 0/0 ""
Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166
[27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure
Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166
[27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure
```
Is there an easy way to see why this happens?
```
root@lb1:~# haproxy -vv
HA-Proxy version 2.3.8-1ppa1~bionic 2021/03/25 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2022.
Known bugs: http://www.haproxy.org/bugs/bugs-2.3.8.html
Running on: Linux 4.15.0-139-generic #143-Ubuntu SMP Tue Mar 16 01:30:17 UTC
2021 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-ot86Gj/haproxy-2.3.8=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1
USE_SYSTEMD=1
DEBUG =
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT
+POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
+GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY
+TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER
+PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2
fcgi : mode=HTTP side=BEmux=FCGI
: mode=HTTP side=FE|BE mux=H1
: mode=TCPside=FE|BE mux=PASS
Available services : prometheus-exporter
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace
```
Regards
Alex
Re: Log the reason for SSL handshake failure
On Wed, Jun 24, 2020 at 01:32:29AM +0200, Marcel Menzel wrote: > Hello list, > > after unsuccessful search in the documentation I am asking here if it's > possible to somehow make HAProxy log the reason why a SSL handshake > failed (especially on a frontend). > I am thinking of logging the SSL alert message, for example logging if > the message came from the server or the client, the AlertLevel and the > alert message: > > "ft_https/1: SSL handshake failure: C>S fatal certificate_unknown" > > We've had to deal with the expired AddTrust certificate and saw a lot of > logged SSL handshake failures, but since HAProxy doesn't log the reason > why a handshake failed we had to use tcpdump to get SSL alert number > leading to an aborted SSL handshake. > > > Kind regards, > > Marcel Menzel Unfortunately it's not possible yet, but we were asked this many time and we will definitively improve that. At the moment the moment what is logged is the error string which is provided by OpenSSL. A ticket was open a few days ago about it on github https://github.com/haproxy/haproxy/issues/693 Regards, -- William Lallemand
Log the reason for SSL handshake failure
Hello list, after unsuccessful search in the documentation I am asking here if it's possible to somehow make HAProxy log the reason why a SSL handshake failed (especially on a frontend). I am thinking of logging the SSL alert message, for example logging if the message came from the server or the client, the AlertLevel and the alert message: "ft_https/1: SSL handshake failure: C>S fatal certificate_unknown" We've had to deal with the expired AddTrust certificate and saw a lot of logged SSL handshake failures, but since HAProxy doesn't log the reason why a handshake failed we had to use tcpdump to get SSL alert number leading to an aborted SSL handshake. Kind regards, Marcel Menzel
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
2015-12-07 13:26 GMT+01:00 Lukas Tribus : > True, but its always a good idea to simplify the configuration when > troubleshooting issues like this. For testing purposes therefor, you > should try with just one server declaration (e.g. what if nginx doesn't > propagate the proxy_protocol directive correctly due to a bug?). So, yeah, It was a bug on Nginx side : https://trac.nginx.org/nginx/ticket/858 > btw: you are using unencrypted backend traffic as well, whats the reason > to encrypt some but not all of the backend traffic? The application need to receive HTTP and HTTPS requests on separate ports. Thank you all :) -- Jonathan Leroy http://www.inikup.com/ Tel: +33 (0)9 74 77 41 72
RE: SSL handshake failure when using "send-proxy" on HTTPS backend
> Both listen directives on port 8443 uses SSL. > With Nginx, listening options must be specified on only one "listen" > directive for each address:port combination. > > So the "listen 10.0.80.1:8443" directive inherit parameters from > "listen 10.0.80.1:8443 default_server ssl proxy_protocol" True, but its always a good idea to simplify the configuration when troubleshooting issues like this. For testing purposes therefor, you should try with just one server declaration (e.g. what if nginx doesn't propagate the proxy_protocol directive correctly due to a bug?). To see if haproxy is behaving correctly tcpdump the failed SSL backend session and check out how it looks on the wire. Then you will have evidence whether haproxy or nginx is behaving incorrectly. btw: you are using unencrypted backend traffic as well, whats the reason to encrypt some but not all of the backend traffic? Regards, Lukas
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
2015-12-06 12:25 GMT+01:00 Lukas Erlacher : > I can't find an obvious error with this. When I tried combining SSL and > proxy protocol in Postfix, it didn't work due to a bug in Postfix. Maybe you > should try to ask an nginx support list instead. Thanks, I'll try that. -- Jonathan Leroy http://www.inikup.com/ Tel: +33 (0)9 74 77 41 72
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
2015-12-06 16:14 GMT+01:00 PiBa-NL :
> Hi,
>
> Ive never used nginx and have little experience with proxy_protocol.. But
> could it be an issue that on the same port your both using and not using
> proxy protocol? What happens if you remove the first server definition
> there?
>
> server {
> listen 10.0.80.1:8443;
> server {
> listen 10.0.80.1:8443 default_server ssl proxy_protocol;
>
> Just a thought..
Hi,
See my previous response to Lukas Tribus. With Nginx, listening
options must be specified only once for the same address+port
combinations.
--
Jonathan Leroy
http://www.inikup.com/
Tel: +33 (0)9 74 77 41 72
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
Hi,
Ive never used nginx and have little experience with proxy_protocol..
But could it be an issue that on the same port your both using and not
using proxy protocol? What happens if you remove the first server
definition there?
server {
listen 10.0.80.1:8443;
server {
listen 10.0.80.1:8443 default_server ssl proxy_protocol;
Just a thought..
Regards,
PiBa-NL
Op 6-12-2015 om 12:25 schreef Lukas Erlacher:
Hi,
On 12/04/2015 04:27 PM, Jonathan Leroy - Inikup wrote:
2015-12-04 13:23 GMT+01:00 Lukas Erlacher :
Please show the nginx config.
Hi Luke,
Here's the Nginx config :
https://gist.githubusercontent.com/jleroy/ab45c328263731c46ec1/raw/69af9edc154329c113aad588ff5f9501edfd61b1/gistfile1.txt
Thanks,
I can't find an obvious error with this. When I tried combining SSL
and proxy protocol in Postfix, it didn't work due to a bug in Postfix.
Maybe you should try to ask an nginx support list instead.
Best,
Luke
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
Hi, On 12/04/2015 04:27 PM, Jonathan Leroy - Inikup wrote: 2015-12-04 13:23 GMT+01:00 Lukas Erlacher : Please show the nginx config. Hi Luke, Here's the Nginx config : https://gist.githubusercontent.com/jleroy/ab45c328263731c46ec1/raw/69af9edc154329c113aad588ff5f9501edfd61b1/gistfile1.txt Thanks, I can't find an obvious error with this. When I tried combining SSL and proxy protocol in Postfix, it didn't work due to a bug in Postfix. Maybe you should try to ask an nginx support list instead. Best, Luke
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
Hi, 2015-12-04 17:02 GMT+01:00 Lukas Tribus : > Well, you will have to update the first config line in nginx: > set_real_ip_from fc00::/7 > > To allow proxy connection from the ULA range. Already done. > As to the original problem: > I don't think you can use both SSL and non-SSL on the same port (8443). > > The non-SSL server block should have a dedicated port, otherwise nginx > will never know what to expect (SSL vs non-SSL, proxy or not proxy). Both listen directives on port 8443 uses SSL. With Nginx, listening options must be specified on only one "listen" directive for each address:port combination. So the "listen 10.0.80.1:8443" directive inherit parameters from "listen 10.0.80.1:8443 default_server ssl proxy_protocol" -- Jonathan Leroy http://www.inikup.com/ Tel: +33 (0)9 74 77 41 72
RE: SSL handshake failure when using "send-proxy" on HTTPS backend
> 2015-12-04 16:27 GMT+01:00 Jonathan Leroy - Inikup : >> Hi Luke, >> >> Here's the Nginx config : >> https://gist.githubusercontent.com/jleroy/ab45c328263731c46ec1/raw/69af9edc154329c113aad588ff5f9501edfd61b1/gistfile1.txt > > Now that I use ULA instead of link-local addresses, send-proxy no > longer works on HTTP backend... Well, you will have to update the first config line in nginx: set_real_ip_from fc00::/7 To allow proxy connection from the ULA range. As to the original problem: I don't think you can use both SSL and non-SSL on the same port (8443). The non-SSL server block should have a dedicated port, otherwise nginx will never know what to expect (SSL vs non-SSL, proxy or not proxy). Regards, Lukas
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
2015-12-04 16:27 GMT+01:00 Jonathan Leroy - Inikup : > Hi Luke, > > Here's the Nginx config : > https://gist.githubusercontent.com/jleroy/ab45c328263731c46ec1/raw/69af9edc154329c113aad588ff5f9501edfd61b1/gistfile1.txt Now that I use ULA instead of link-local addresses, send-proxy no longer works on HTTP backend... -- Jonathan Leroy http://www.inikup.com/ Tel: +33 (0)9 74 77 41 72
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
2015-12-04 13:23 GMT+01:00 Lukas Erlacher : > Please show the nginx config. Hi Luke, Here's the Nginx config : https://gist.githubusercontent.com/jleroy/ab45c328263731c46ec1/raw/69af9edc154329c113aad588ff5f9501edfd61b1/gistfile1.txt Thanks, -- Jonathan Leroy http://www.inikup.com/ Tel: +33 (0)9 74 77 41 72
Re: SSL handshake failure when using "send-proxy" on HTTPS backend
Please show the nginx config.
Best,
Luke
On 12/04/2015 03:30 AM, Jonathan Leroy - Inikup wrote:
Hi,
I have two backends named "nginx-http" and "nginx-https": the first
one handle HTTP connections, the second one HTTPS connections.
The proxy protocol works successfully on nginx-http backend:
server server1 10.0.80.1:8080 send-proxy check check-send-proxy fall
3 inter 2s weight 10
But the same configuration doen't work on nginx-https backend ("SSL
handshake failure"):
server server1 10.0.80.1:8443 ssl send-proxy check check-send-proxy
check-ssl ca-file /etc/ssl/certs/Certum_Trusted_Network_CA.pem cookie
test1 fall 3 inter 2s weight 10
As soon has I remove the "send-proxy" and "check-send-proxy"
directives everything works fine, so I think this is not an
SSL-related issue.
I use latest HAProxy and Nginx stables versions on Debian Jessie. SPDY
is not activated on Nginx side.
Thanks,
smime.p7s
Description: S/MIME Cryptographic Signature
SSL handshake failure when using "send-proxy" on HTTPS backend
Hi,
I have two backends named "nginx-http" and "nginx-https": the first
one handle HTTP connections, the second one HTTPS connections.
The proxy protocol works successfully on nginx-http backend:
server server1 10.0.80.1:8080 send-proxy check check-send-proxy fall
3 inter 2s weight 10
But the same configuration doen't work on nginx-https backend ("SSL
handshake failure"):
server server1 10.0.80.1:8443 ssl send-proxy check check-send-proxy
check-ssl ca-file /etc/ssl/certs/Certum_Trusted_Network_CA.pem cookie
test1 fall 3 inter 2s weight 10
As soon has I remove the "send-proxy" and "check-send-proxy"
directives everything works fine, so I think this is not an
SSL-related issue.
I use latest HAProxy and Nginx stables versions on Debian Jessie. SPDY
is not activated on Nginx side.
Thanks,
--
Jonathan Leroy
http://www.inikup.com/
Tel: +33 (0)9 74 77 41 72
Re: SSL handshake failure when setting up no-tlsv10
ok thanks i will upgrade the OS, even i don't feel comfortable upgrading and patching libraries outside the repo (hence never tried it.)The only thing i wanted to test was if it works with the new version of openssl but even if that is going to cause issues i will focus on upgrading the OS Thanks bryan and lukas once again From: Bryan Talbot To: Lukas Tribus Cc: Amol ; Bryan Talbot ; HAproxy Mailing Lists Sent: Wednesday, May 20, 2015 1:47 PM Subject: Re: SSL handshake failure when setting up no-tlsv10 On Wed, May 20, 2015 at 10:40 AM, Lukas Tribus wrote: > yes i figured since it is a ubuntu 10.04 machine it has old version of > openssl > > so i looked around for upgrading the openssl and found this link > https://sandilands.info/sgordon/upgrade-latest-version-openssl-on-ubuntu > > so can i just upgrade to openssl 1.0.1 and add it to the correct path > and just restart the haproxy service? Please don't. As long as you don't *exactly* know what you are doing, ONLY use your OS internal packaging system and don't follow tips you find on google. This particular blog post for example makes you install a ancient version of openssl (just look at the date of the post), with numerous issues and bugs. Also you would very likely mess up your whole system. Ubuntu 10.04 is EOL, you don't use an EOL'ed OS in production, period. Upgrade to the next Ubuntu LTS edition by following the howto of your OS vendor: https://help.ubuntu.com/community/PreciseUpgrades I agree with Lukas. Unless you're an expert at building and installing customized system software, I would not recommend you do anything like that on a server you want to be stable. Upgrade your OS is your best option for sure. -Bryan
Re: SSL handshake failure when setting up no-tlsv10
On Wed, May 20, 2015 at 10:40 AM, Lukas Tribus wrote: > > yes i figured since it is a ubuntu 10.04 machine it has old version of > > openssl > > > > so i looked around for upgrading the openssl and found this link > > https://sandilands.info/sgordon/upgrade-latest-version-openssl-on-ubuntu > > > > so can i just upgrade to openssl 1.0.1 and add it to the correct path > > and just restart the haproxy service? > > Please don't. > > As long as you don't *exactly* know what you are doing, ONLY use your > OS internal packaging system and don't follow tips you find on google. > This particular blog post for example makes you install a ancient version > of openssl (just look at the date of the post), with numerous issues and > bugs. Also you would very likely mess up your whole system. > > Ubuntu 10.04 is EOL, you don't use an EOL'ed OS in production, period. > > Upgrade to the next Ubuntu LTS edition by following the howto of your > OS vendor: > https://help.ubuntu.com/community/PreciseUpgrades > > > I agree with Lukas. Unless you're an expert at building and installing customized system software, I would not recommend you do anything like that on a server you want to be stable. Upgrade your OS is your best option for sure. -Bryan
RE: SSL handshake failure when setting up no-tlsv10
> yes i figured since it is a ubuntu 10.04 machine it has old version of > openssl > > so i looked around for upgrading the openssl and found this link > https://sandilands.info/sgordon/upgrade-latest-version-openssl-on-ubuntu > > so can i just upgrade to openssl 1.0.1 and add it to the correct path > and just restart the haproxy service? Please don't. As long as you don't *exactly* know what you are doing, ONLY use your OS internal packaging system and don't follow tips you find on google. This particular blog post for example makes you install a ancient version of openssl (just look at the date of the post), with numerous issues and bugs. Also you would very likely mess up your whole system. Ubuntu 10.04 is EOL, you don't use an EOL'ed OS in production, period. Upgrade to the next Ubuntu LTS edition by following the howto of your OS vendor: https://help.ubuntu.com/community/PreciseUpgrades Lukas
Re: SSL handshake failure when setting up no-tlsv10
yes i figured since it is a ubuntu 10.04 machine it has old version of openssl so i looked around for upgrading the openssl and found this link https://sandilands.info/sgordon/upgrade-latest-version-openssl-on-ubuntu so can i just upgrade to openssl 1.0.1 and add it to the correct path and just restart the haproxy service?Do you think that would work i really liked to install haproxy from the repository instead of compiling it myself From: Bryan Talbot To: Amol ; HAproxy Mailing Lists Sent: Wednesday, May 20, 2015 1:18 PM Subject: Re: SSL handshake failure when setting up no-tlsv10 On Wed, May 20, 2015 at 10:10 AM, Amol wrote: here is the output from the commands you requested Built with OpenSSL version : OpenSSL 0.9.8k 25 Mar 2009 Running on OpenSSL version : OpenSSL 0.9.8k 25 Mar 2009 :~$ openssl version OpenSSL 0.9.8k 25 Mar 2009 The openssl version is just too old to support TLS 1.2 as you can see in the supported cipher list below. Best bet would be to upgrade to a newer release of your OS. Another option would be to compile a newer version of openssl and compile your own haproxy and statically link against the newer openssl. -Bryan :~$ openssl ciphers -v DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1 DHE-DSS-AES256-SHA SSLv3 Kx=DH Au=DSS Enc=AES(256) Mac=SHA1 AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 EDH-RSA-DES-CBC3-SHA SSLv3 Kx=DH Au=RSA Enc=3DES(168) Mac=SHA1 EDH-DSS-DES-CBC3-SHA SSLv3 Kx=DH Au=DSS Enc=3DES(168) Mac=SHA1 DES-CBC3-SHA SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1 DES-CBC3-MD5 SSLv2 Kx=RSA Au=RSA Enc=3DES(168) Mac=MD5 DHE-RSA-AES128-SHA SSLv3 Kx=DH Au=RSA Enc=AES(128) Mac=SHA1 DHE-DSS-AES128-SHA SSLv3 Kx=DH Au=DSS Enc=AES(128) Mac=SHA1 AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1 RC2-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC2(128) Mac=MD5 RC4-SHA SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1 RC4-MD5 SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 RC4-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH Au=RSA Enc=DES(56) Mac=SHA1 EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH Au=DSS Enc=DES(56) Mac=SHA1 DES-CBC-SHA SSLv3 Kx=RSA Au=RSA Enc=DES(56) Mac=SHA1 DES-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=DES(56) Mac=MD5 EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512) Au=RSA Enc=DES(40) Mac=SHA1 export EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512) Au=DSS Enc=DES(40) Mac=SHA1 export EXP-DES-CBC-SHA SSLv3 Kx=RSA(512) Au=RSA Enc=DES(40) Mac=SHA1 export EXP-RC2-CBC-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 export EXP-RC2-CBC-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 export EXP-RC4-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 export EXP-RC4-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 export :~$ From: Bryan Talbot To: Amol ; HAproxy Mailing Lists Sent: Wednesday, May 20, 2015 1:04 PM Subject: Re: SSL handshake failure when setting up no-tlsv10 On Wed, May 20, 2015 at 9:39 AM, Amol wrote: Thanks you for responding and i wanted to share some more from my findings when i set ssl-default-bind-options no-sslv3 force-tlsv12 $ sudo vi /etc/haproxy/haproxy.cfg :~$ sudo /etc/init.d/haproxy restart * Restarting haproxy haproxy [ALERT] 139/122930 (8602) : parsing [/etc/haproxy/haproxy.cfg:22] : 'ssl-default-bind-options' 'force-tlsv12': library does not support protocol TLSv1.2 [ALERT] 139/122930 (8602) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 139/122930 (8602) : Fatal errors found in configuration. Yes, it sounds like your openssl lib must be pretty old or is oddly configured. What does "haproxy -vv" and "openssl version" report? You can see a list of supported ciphers and protocols using "openssl ciphers -v" as well. -Bryan
Re: SSL handshake failure when setting up no-tlsv10
On Wed, May 20, 2015 at 10:10 AM, Amol wrote: > here is the output from the commands you requested > > Built with OpenSSL version : OpenSSL 0.9.8k 25 Mar 2009 > Running on OpenSSL version : OpenSSL 0.9.8k 25 Mar 2009 > > > :~$ openssl version > OpenSSL 0.9.8k 25 Mar 2009 > > > The openssl version is just too old to support TLS 1.2 as you can see in the supported cipher list below. Best bet would be to upgrade to a newer release of your OS. Another option would be to compile a newer version of openssl and compile your own haproxy and statically link against the newer openssl. -Bryan > :~$ openssl ciphers -v > DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1 > DHE-DSS-AES256-SHA SSLv3 Kx=DH Au=DSS Enc=AES(256) Mac=SHA1 > AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 > EDH-RSA-DES-CBC3-SHASSLv3 Kx=DH Au=RSA Enc=3DES(168) Mac=SHA1 > EDH-DSS-DES-CBC3-SHASSLv3 Kx=DH Au=DSS Enc=3DES(168) Mac=SHA1 > DES-CBC3-SHASSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1 > DES-CBC3-MD5SSLv2 Kx=RSA Au=RSA Enc=3DES(168) Mac=MD5 > DHE-RSA-AES128-SHA SSLv3 Kx=DH Au=RSA Enc=AES(128) Mac=SHA1 > DHE-DSS-AES128-SHA SSLv3 Kx=DH Au=DSS Enc=AES(128) Mac=SHA1 > AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1 > RC2-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC2(128) Mac=MD5 > RC4-SHA SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1 > RC4-MD5 SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 > RC4-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 > EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH Au=RSA Enc=DES(56) Mac=SHA1 > EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH Au=DSS Enc=DES(56) Mac=SHA1 > DES-CBC-SHA SSLv3 Kx=RSA Au=RSA Enc=DES(56) Mac=SHA1 > DES-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=DES(56) Mac=MD5 > EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512) Au=RSA Enc=DES(40) Mac=SHA1 > export > EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512) Au=DSS Enc=DES(40) Mac=SHA1 > export > EXP-DES-CBC-SHA SSLv3 Kx=RSA(512) Au=RSA Enc=DES(40) Mac=SHA1 > export > EXP-RC2-CBC-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 > export > EXP-RC2-CBC-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 > export > EXP-RC4-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 > export > EXP-RC4-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 > export > :~$ > > > -- > *From:* Bryan Talbot > *To:* Amol ; HAproxy Mailing Lists < > haproxy@formilux.org> > *Sent:* Wednesday, May 20, 2015 1:04 PM > > *Subject:* Re: SSL handshake failure when setting up no-tlsv10 > > On Wed, May 20, 2015 at 9:39 AM, Amol wrote: > > Thanks you for responding and i wanted to share some more from my findings > > when i set > *ssl-default-bind-options no-sslv3 force-tlsv12* > > $ sudo vi /etc/haproxy/haproxy.cfg > :~$ sudo /etc/init.d/haproxy restart > * Restarting haproxy > haproxy > [ALERT] 139/122930 (8602) : parsing [/etc/haproxy/haproxy.cfg:22] : > 'ssl-default-bind-options' 'force-tlsv12': library does not support > protocol TLSv1.2 > [ALERT] 139/122930 (8602) : Error(s) found in configuration file : > /etc/haproxy/haproxy.cfg > [ALERT] 139/122930 (8602) : Fatal errors found in configuration. > > > > Yes, it sounds like your openssl lib must be pretty old or is oddly > configured. What does "haproxy -vv" and "openssl version" report? You can > see a list of supported ciphers and protocols using "openssl ciphers -v" as > well. > > > > -Bryan > > > >
Re: SSL handshake failure when setting up no-tlsv10
here is the output from the commands you requested :~$ haproxy -vvHA-Proxy version 1.5.9 2014/11/25 Copyright 2000-2014 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 0.9.8k 25 Mar 2009 Running on OpenSSL version : OpenSSL 0.9.8k 25 Mar 2009 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. :~$ openssl version OpenSSL 0.9.8k 25 Mar 2009 :~$ openssl ciphers -v DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1 DHE-DSS-AES256-SHA SSLv3 Kx=DH Au=DSS Enc=AES(256) Mac=SHA1 AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 EDH-RSA-DES-CBC3-SHA SSLv3 Kx=DH Au=RSA Enc=3DES(168) Mac=SHA1 EDH-DSS-DES-CBC3-SHA SSLv3 Kx=DH Au=DSS Enc=3DES(168) Mac=SHA1 DES-CBC3-SHA SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1 DES-CBC3-MD5 SSLv2 Kx=RSA Au=RSA Enc=3DES(168) Mac=MD5 DHE-RSA-AES128-SHA SSLv3 Kx=DH Au=RSA Enc=AES(128) Mac=SHA1 DHE-DSS-AES128-SHA SSLv3 Kx=DH Au=DSS Enc=AES(128) Mac=SHA1 AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1 RC2-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC2(128) Mac=MD5 RC4-SHA SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1 RC4-MD5 SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 RC4-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5 EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH Au=RSA Enc=DES(56) Mac=SHA1 EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH Au=DSS Enc=DES(56) Mac=SHA1 DES-CBC-SHA SSLv3 Kx=RSA Au=RSA Enc=DES(56) Mac=SHA1 DES-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=DES(56) Mac=MD5 EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512) Au=RSA Enc=DES(40) Mac=SHA1 export EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512) Au=DSS Enc=DES(40) Mac=SHA1 export EXP-DES-CBC-SHA SSLv3 Kx=RSA(512) Au=RSA Enc=DES(40) Mac=SHA1 export EXP-RC2-CBC-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 export EXP-RC2-CBC-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 export EXP-RC4-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 export EXP-RC4-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 export :~$ From: Bryan Talbot To: Amol ; HAproxy Mailing Lists Sent: Wednesday, May 20, 2015 1:04 PM Subject: Re: SSL handshake failure when setting up no-tlsv10 On Wed, May 20, 2015 at 9:39 AM, Amol wrote: Thanks you for responding and i wanted to share some more from my findings when i set ssl-default-bind-options no-sslv3 force-tlsv12 $ sudo vi /etc/haproxy/haproxy.cfg :~$ sudo /etc/init.d/haproxy restart * Restarting haproxy haproxy [ALERT] 139/122930 (8602) : parsing [/etc/haproxy/haproxy.cfg:22] : 'ssl-default-bind-options' 'force-tlsv12': library does not support protocol TLSv1.2 [ALERT] 139/122930 (8602) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 139/122930 (8602) : Fatal errors found in configuration. Yes, it sounds like your openssl lib must be pretty old or is oddly configured. What does "haproxy -vv" and "openssl version" report? You can see a list of supported ciphers and protocols using "openssl ciphers -v" as well. -Bryan
Re: SSL handshake failure when setting up no-tlsv10
On Wed, May 20, 2015 at 9:39 AM, Amol wrote: > Thanks you for responding and i wanted to share some more from my findings > > when i set > *ssl-default-bind-options no-sslv3 force-tlsv12* > > $ sudo vi /etc/haproxy/haproxy.cfg > :~$ sudo /etc/init.d/haproxy restart > * Restarting haproxy > haproxy > [ALERT] 139/122930 (8602) : parsing [/etc/haproxy/haproxy.cfg:22] : > 'ssl-default-bind-options' 'force-tlsv12': library does not support > protocol TLSv1.2 > [ALERT] 139/122930 (8602) : Error(s) found in configuration file : > /etc/haproxy/haproxy.cfg > [ALERT] 139/122930 (8602) : Fatal errors found in configuration. > Yes, it sounds like your openssl lib must be pretty old or is oddly configured. What does "haproxy -vv" and "openssl version" report? You can see a list of supported ciphers and protocols using "openssl ciphers -v" as well. -Bryan
Re: SSL handshake failure when setting up no-tlsv10
Thanks you for responding and i wanted to share some more from my findings when i set ssl-default-bind-options no-sslv3 force-tlsv12 $ sudo vi /etc/haproxy/haproxy.cfg :~$ sudo /etc/init.d/haproxy restart * Restarting haproxy haproxy [ALERT] 139/122930 (8602) : parsing [/etc/haproxy/haproxy.cfg:22] : 'ssl-default-bind-options' 'force-tlsv12': library does not support protocol TLSv1.2 [ALERT] 139/122930 (8602) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 139/122930 (8602) : Fatal errors found in configuration. when i set ssl-default-bind-options no-sslv3 force-tlsv11 :~$ sudo /etc/init.d/haproxy restart * Restarting haproxy haproxy [ALERT] 139/122945 (8649) : parsing [/etc/haproxy/haproxy.cfg:22] : 'ssl-default-bind-options' 'force-tlsv11': library does not support protocol TLSv1.1 [ALERT] 139/122945 (8649) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 139/122945 (8649) : Fatal errors found in configuration. when i set ssl-default-bind-options no-sslv3 force-tlsv10 :~$ sudo /etc/init.d/haproxy restart * Restarting haproxy haproxy [ OK ] does that mean the it is an issue with the ubuntu installation of haproxy?is there a way to resolve this? From: Bryan Talbot To: Amol Cc: HAproxy Mailing Lists Sent: Monday, May 11, 2015 5:29 PM Subject: Re: SSL handshake failure when setting up no-tlsv10 On Mon, May 11, 2015 at 1:46 PM, Amol wrote: Hi I am using Haproxy (1.5.9) and trying to resolve a PCI compliance issue with TLS v1.0, but when i set the following options in global section of the haproxy.cfg i am getting an error in my haproxy.log and the webpage does not showup. ssl-default-bind-options no-sslv3 no-tlsv10 error in haproxy.log May 11 16:37:39 load-lb haproxy[2680]: xx.xx.xx.xx:56787 [11/May/2015:16:37:39.626] www-https/1: SSL handshake failure here is the snippet of the actual SSL settings ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4: EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 ssl-default-bind-options no-sslv3 no-tlsv10 tune.ssl.default-dh-param 4096 Please let me know if i am missing anything? Works for me. $ ./haproxy -vvHA-Proxy version 1.5.12-2 2015/05/11Copyright 2000-2015 Willy Tarreau Build options : TARGET = generic CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=0 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): noBuilt with zlib version : 1.2.5Compression algorithms supported : identity, deflate, gzipBuilt with OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015Running on OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015OpenSSL library supports TLS extensions : yesOpenSSL library supports SNI : yesOpenSSL library supports prefer-server-ciphers : yesBuilt without PCRE support (using libc's regex instead) Available polling systems : poll : pref=200, test result OK select : pref=150, test result OKTotal: 2 (2 usable), will use poll. $ cat haproxy.cfgglobal tune.ssl.default-dh-param 4096 ssl-default-bind-options no-sslv3 no-tlsv10 ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 defaults timeout client 5s timeout server 5s mode http listen foo bind :4433 ssl crt ./test.pem $ ./haproxy -f ./haproxy.cfg -cConfiguration file is valid $ openssl versionOpenSSL 1.0.2a 19 Mar 2015 $ echo | openssl s_client -connect 127.0.0.1:4433 ...SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384... Maybe it's an issue with your client? -Bryan
Re: SSL handshake failure when setting up no-tlsv10
On Mon, May 11, 2015 at 1:46 PM, Amol wrote: > Hi > I am using Haproxy (1.5.9) and trying to resolve a PCI compliance issue > with TLS v1.0, but when i set the following options in global section of > the haproxy.cfg i am getting an error in my haproxy.log and the webpage > does not showup. > > ssl-default-bind-options no-sslv3 *no-tlsv10* > > *error in haproxy.log* > > May 11 16:37:39 load-lb haproxy[2680]: xx.xx.xx.xx:56787 > [11/May/2015:16:37:39.626] www-https/1: SSL handshake failure > > > here is the snippet of the actual SSL settings > > ssl-default-bind-ciphers > EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4: > EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 > ssl-default-bind-options no-sslv3 *no-tlsv10* > tune.ssl.default-dh-param 4096 > > > Please let me know if i am missing anything? > > > Works for me. $ ./haproxy -vv HA-Proxy version 1.5.12-2 2015/05/11 Copyright 2000-2015 Willy Tarreau Build options : TARGET = generic CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=0 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): no Built with zlib version : 1.2.5 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015 Running on OpenSSL version : OpenSSL 1.0.2a 19 Mar 2015 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built without PCRE support (using libc's regex instead) Available polling systems : poll : pref=200, test result OK select : pref=150, test result OK Total: 2 (2 usable), will use poll. $ cat haproxy.cfg global tune.ssl.default-dh-param 4096 ssl-default-bind-options no-sslv3 no-tlsv10 ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 defaults timeout client 5s timeout server 5s mode http listen foo bind :4433 ssl crt ./test.pem $ ./haproxy -f ./haproxy.cfg -c Configuration file is valid $ openssl version OpenSSL 1.0.2a 19 Mar 2015 $ echo | openssl s_client -connect 127.0.0.1:4433 ... SSL-Session: Protocol : TLSv1.2 Cipher: ECDHE-RSA-AES256-GCM-SHA384 ... Maybe it's an issue with your client? -Bryan
Re: SSL handshake failure when setting up no-tlsv10
On 11/05/2015 10:46 μμ, Amol wrote: > Hi > I am using Haproxy (1.5.9) and trying to resolve a PCI compliance issue > with TLS v1.0, but when i set the following options in global section of > the haproxy.cfg i am getting an error in my haproxy.log and the webpage > does not showup. > > ssl-default-bind-options no-sslv3 *no-tlsv10* > > *error in haproxy.log* > > May 11 16:37:39 load-lb haproxy[2680]: xx.xx.xx.xx:56787 > [11/May/2015:16:37:39.626] www-https/1: SSL handshake failure > > I guess your client tried to use TLS1.0 which is disabled. > here is the snippet of the actual SSL settings > > ssl-default-bind-ciphers > EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4: > EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 > ssl-default-bind-options no-sslv3 *no-tlsv10* > tune.ssl.default-dh-param 4096 > > > Please let me know if i am missing anything? > > > Try openssl s_client -connect site:443 -tls1_1 and should work and openssl s_client -connect site:443 -tls1 it shouldn't work as you disabled that version. Cheers, Pavlos signature.asc Description: OpenPGP digital signature
SSL handshake failure when setting up no-tlsv10
Hi I am using Haproxy (1.5.9) and trying to resolve a PCI compliance issue with TLS v1.0, but when i set the following options in global section of the haproxy.cfg i am getting an error in my haproxy.log and the webpage does not showup. ssl-default-bind-options no-sslv3 no-tlsv10 error in haproxy.log May 11 16:37:39 load-lb haproxy[2680]: xx.xx.xx.xx:56787 [11/May/2015:16:37:39.626] www-https/1: SSL handshake failure here is the snippet of the actual SSL settings ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4: EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 ssl-default-bind-options no-sslv3 no-tlsv10 tune.ssl.default-dh-param 4096 Please let me know if i am missing anything?
Re: SSL handshake failure
On 9/10/2014 11:43 PM, Willy Tarreau wrote: > It is also possible that they have stored locally a copy of your old cert > or maybe they have your CA's certs and you changed to a new CA to sign this > new cert. It's the same CA and intermediate cert. We suspect that they have configured it to only validate a specific certificate -- the broken one. > If they're willing to do some tests, you could probably use openssl itself > to wait for an incoming connection. Check "openssl s_server -accept " > for this. You'd get a lot more detailed info from the openssl tool itself. > Maybe you'l discover that their client is old and cannot negociate a key of > the size you're using for example. Next step could be to have them connect > to your openssl using "openssl s_client -connect" and see on their side if > anything looks wrong. I can make this suggestion, although one thing I do know is that they've got an aspx application running on Windows 2003 ... they probably know less about SSL than my immediate supervisor ... and that's saying something. I'm getting backend servers going down due to Layer6 timeout after five seconds. Is this during SSL handshaking, by chance? I can start a new thread on this issue if that's advisable. This may be the entire problem with these connections -- the Mule process on the back end may have some SSL issues. We've discussed turning SSL off on the back end .. if switching to haproxy proves useful, we may do that. Sep 11 05:21:57 localhost.localdomain haproxy[8434]: Backup Server services-ai-search-backend/aladdin is DOWN, reason: Layer6 timeout, check duration: 5001ms. 0 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue. Sep 11 05:24:28 localhost.localdomain haproxy[8434]: Server services-ai-request-backend/fiesta is DOWN, reason: Layer6 timeout, check duration: 5004ms. 0 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue. Thanks, Shawn
Re: SSL handshake failure
On Wed, Sep 10, 2014 at 07:09:13PM -0600, Shawn Heisey wrote: > > having two different versions, we cannot rule out a problem there. > > I did manage to do that. My captures (of my test requests) don't show an > improvement in wireshark's ability to decrypt. > > I suspect that the actual handshake problem with the customer is on their > end. The certificate we were using in production was expired and had the > wrong host name in the subject, so we got a new one with the correct name. > They couldn't connect to that either. I now have placed that expired and > incorrect cert in haproxy's configuration, and I bet they'll be able to > connect to it now. I think their client is probably very stupid. It is also possible that they have stored locally a copy of your old cert or maybe they have your CA's certs and you changed to a new CA to sign this new cert. If they're willing to do some tests, you could probably use openssl itself to wait for an incoming connection. Check "openssl s_server -accept " for this. You'd get a lot more detailed info from the openssl tool itself. Maybe you'l discover that their client is old and cannot negociate a key of the size you're using for example. Next step could be to have them connect to your openssl using "openssl s_client -connect" and see on their side if anything looks wrong. > Because they're in Japan, it takes pretty much a full day for every little > tweak we make to get tested. I hope we can get a more interactive testing > session going. Yeah I know this situation as well. On the other hand you don't waste as much time as you think because it forces you to prepare everything and review everything before going to sleep, and generally you can spot a number of issues on your side. Willy
Re: SSL handshake failure
> having two different versions, we cannot rule out a problem there. I did manage to do that. My captures (of my test requests) don't show an improvement in wireshark's ability to decrypt. I suspect that the actual handshake problem with the customer is on their end. The certificate we were using in production was expired and had the wrong host name in the subject, so we got a new one with the correct name. They couldn't connect to that either. I now have placed that expired and incorrect cert in haproxy's configuration, and I bet they'll be able to connect to it now. I think their client is probably very stupid. Because they're in Japan, it takes pretty much a full day for every little tweak we make to get tested. I hope we can get a more interactive testing session going. Thanks, Shawn
Re: SSL handshake failure
On Wed, Sep 10, 2014 at 12:20:00PM -0600, Shawn Heisey wrote: > On 9/9/2014 11:45 PM, Willy Tarreau wrote: > > It is possible that the more recent openssl lib above defined a few extra > > fields that are not supported by the older one used at runtime, resulting > > in undefined behaviour. If you cannot upgrade the production version, I > > suggest that instead you rebuild haproxy with a static openssl lib, ideally > > with the same version first, then with a more recent one (eg: 1.0.1h) if it > > continues to fail. > > > >> If I force haproxy to use sslv3, then wireshark can decrypt the packets > >> properly (when checked with a browser), but then our testing tools can't > >> connect to it. > > > > TLS supports a wide number of extensions, it is possible that wireshark > > doesn't know them all. But it's also possible that some garbage is being > > sent due to the incompatibility between the build and runtime version, > > which would explain why Wireshark cannot decrypt it. > > I managed to get a dev environment on the server where I'm running it > and get it recompiled. Unfortunately that didn't change the info in the > capture, so I'm guessing that it's something that wireshark just can't > deal with yet. The request that shows up in the capture with "Ignored > Unknown Record" on the TLSv1 packets worked perfectly from a browser -- > it was a request for a jpg image. > > This means that if I can't force sslv3, I won't be able to fully decode > the packet capture. I *can* see the requests, which may be enough. Could you try with the same version for building and runtime ? Also, can you try with a recent openssl version in this dev environment ? I'm not dismissing 0.9.8 which must work of course, but since you're having two different versions, we cannot rule out a problem there. > I'd really like to completely reload these with an OS newer than CentOS > 5. I don't know what the linux2628 target has that's not in linux26, > but I bet it's pretty nice. You'll see them in the makefile : splice(), accept4(), cpu affinity, transparent proxy. That's already nice indeed :-) But that's not compatible with RHEL5 which comes with a heavily patched 2.6.18. Willy
Re: SSL handshake failure
On 9/9/2014 11:45 PM, Willy Tarreau wrote: > It is possible that the more recent openssl lib above defined a few extra > fields that are not supported by the older one used at runtime, resulting > in undefined behaviour. If you cannot upgrade the production version, I > suggest that instead you rebuild haproxy with a static openssl lib, ideally > with the same version first, then with a more recent one (eg: 1.0.1h) if it > continues to fail. > >> If I force haproxy to use sslv3, then wireshark can decrypt the packets >> properly (when checked with a browser), but then our testing tools can't >> connect to it. > > TLS supports a wide number of extensions, it is possible that wireshark > doesn't know them all. But it's also possible that some garbage is being > sent due to the incompatibility between the build and runtime version, > which would explain why Wireshark cannot decrypt it. I managed to get a dev environment on the server where I'm running it and get it recompiled. Unfortunately that didn't change the info in the capture, so I'm guessing that it's something that wireshark just can't deal with yet. The request that shows up in the capture with "Ignored Unknown Record" on the TLSv1 packets worked perfectly from a browser -- it was a request for a jpg image. This means that if I can't force sslv3, I won't be able to fully decode the packet capture. I *can* see the requests, which may be enough. I'd really like to completely reload these with an OS newer than CentOS 5. I don't know what the linux2628 target has that's not in linux26, but I bet it's pretty nice. Thanks, Shawn
Re: SSL handshake failure
Hi Shawn,
On Tue, Sep 09, 2014 at 03:47:30PM -0600, Shawn Heisey wrote:
> I do not think this is a problem with haproxy (running 1.5.4), but I'm
> hoping haproxy can help me debug it.
>
> When I get SSL handshake failure, can haproxy be configured to log debug
> messages about WHY it failed?
Normally it will log one line indicating that a handshake has failed, with
a cause when it could determine it. The following SSL errors are diagnosed
and logged (from include/proto/connection.h) :
case CO_ER_SSL_EMPTY: return "Connection closed during SSL
handshake";
case CO_ER_SSL_ABORT: return "Connection error during SSL
handshake";
case CO_ER_SSL_TIMEOUT: return "Timeout during SSL handshake";
case CO_ER_SSL_TOO_MANY: return "Too many SSL connections";
case CO_ER_SSL_NO_MEM:return "Out of memory when initializing an
SSL connection";
case CO_ER_SSL_RENEG: return "Rejected a client-initiated SSL
renegociation attempt";
case CO_ER_SSL_CA_FAIL: return "SSL client CA chain cannot be
verified";
case CO_ER_SSL_CRT_FAIL: return "SSL client certificate not trusted";
case CO_ER_SSL_HANDSHAKE: return "SSL handshake failure";
case CO_ER_SSL_HANDSHAKE_HB: return "SSL handshake failure after
heartbeat";
case CO_ER_SSL_KILLED_HB: return "Stopped a TLSv1 heartbeat attack
(CVE-2014-0160)";
case CO_ER_SSL_NO_TARGET: return "Attempt to use SSL on an unknown
target (internal error)";
> We don't have any visibility into the
> client -- it's at a customer site in Japan, I'm in the US.
If you only get the "SSL handshake failure" message in the logs, it's very
likely that it was not possible to agree on a cipher. A sniffer in the middle
wlil help you diagnose it I guess.
> There is another question, but it's on an unrelated product. I've got
> the latest version of Wireshark (1.12.0), configured with my
> certificate's private key for SSL decrypting. The problem is that
> Wireshark is telling me that there is something wrong with the TLSv1
> frames ("Ignored Unknown Record"). I do not have decrypted responses,
> only decrypted requests, and I assume that is because of those TLSv1
> problems. The question: Is wireshark buggy, or are those TLSv1 frames
> actually problematic? The program was compiled against
> openssl-0.9.8e-27.el5_10.1 and it's running on a system with
> openssl-0.9.8e-7.el5 installed -- the production systems don't have a
> compiler or dev libraries installed, and when I attempted to install
> them, yum wouldn't work.
It is possible that the more recent openssl lib above defined a few extra
fields that are not supported by the older one used at runtime, resulting
in undefined behaviour. If you cannot upgrade the production version, I
suggest that instead you rebuild haproxy with a static openssl lib, ideally
with the same version first, then with a more recent one (eg: 1.0.1h) if it
continues to fail.
> If I force haproxy to use sslv3, then wireshark can decrypt the packets
> properly (when checked with a browser), but then our testing tools can't
> connect to it.
TLS supports a wide number of extensions, it is possible that wireshark
doesn't know them all. But it's also possible that some garbage is being
sent due to the incompatibility between the build and runtime version,
which would explain why Wireshark cannot decrypt it.
Regards,
Willy
SSL handshake failure
I do not think this is a problem with haproxy (running 1.5.4), but I'm
hoping haproxy can help me debug it.
When I get SSL handshake failure, can haproxy be configured to log debug
messages about WHY it failed? We don't have any visibility into the
client -- it's at a customer site in Japan, I'm in the US.
There is another question, but it's on an unrelated product. I've got
the latest version of Wireshark (1.12.0), configured with my
certificate's private key for SSL decrypting. The problem is that
Wireshark is telling me that there is something wrong with the TLSv1
frames ("Ignored Unknown Record"). I do not have decrypted responses,
only decrypted requests, and I assume that is because of those TLSv1
problems. The question: Is wireshark buggy, or are those TLSv1 frames
actually problematic? The program was compiled against
openssl-0.9.8e-27.el5_10.1 and it's running on a system with
openssl-0.9.8e-7.el5 installed -- the production systems don't have a
compiler or dev libraries installed, and when I attempted to install
them, yum wouldn't work.
If I force haproxy to use sslv3, then wireshark can decrypt the packets
properly (when checked with a browser), but then our testing tools can't
connect to it.
Thanks,
Shawn
Re: haproxy-1.5-dev23 and ssl handshake failure
> > Markus, please follow Willy's advise and remove all force-* configurations > from your bind line, you should use no-sslv3/no-tlsv1[0-2] keywords to > configure specific TLS version, but in this case, as long as you > troubleshooting this, I strongly suggest to not configure any specific TLS > settings. > i have removed the force-options. so i just have frontend https bind 46.16.74.36:443 ssl crt /opt/haproxy/haproxy.ssl.crt ciphers ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:!RC4+RSA:+HIGH:+MEDIUM with dev23 and dev24 i don't see any handshake error messages anymore. thats good. the error messages in the browser came very seldom. so its hard to confirm, that they are gone. but i would suppose, that they are fixed. i will have a look on this and report... thanxs markus
Re: haproxy-1.5-dev23 and ssl handshake failure
Hello, Here the configuration: global daemon pidfile /var/run/haproxy-3.pid maxconn 25 tune.bufsize8024 log 127.0.0.1 local0 defaults log global mode http option httplog #option dontlognull option dontlog-normal no option httpclose option http-server-close no option forceclose option forwardfor balance roundrobin option redispatch retries 3 timeout client 50s timeout http-keep-alive 30s timeout server 50s timeout connect 10s frontend https_rtb_lamoda25_ru maxconn 3 bindxxx.xxx.xxx.xxx:443 ssl crt /opt/crt/domain.com.pem reqadd X-Scheme:\ https acl is_value path_beg /some/path/ use_backend some_bacjend if is_value acl is_value_2 path_beg /some/path-2/ use_backend some_backend_2 if is_value_2 other backends backend some_backend option http-server-close server server1.1 xxx.xxx.xxx.xxx:8101 weight 1 maxconn 100 check port 8101 server server1.2 xxx.xxx.xxx.xxx:8102 weight 1 maxconn 100 check port 8102 I guess yes, i started see then in April. And ssl load balancing was working about for last 6 months. With Regards, Stefan
Re: haproxy-1.5-dev23 and ssl handshake failure
>> my problem is, that i sometimes see an error message in my browser. i >> also got one response from a user saying that he can't access our >> ssl-pages and gets an error. > > There are 2 issues here: > - the fact that you sometimes (?) see this error in the browser > - the fact that one user can't open the ssl-page at all (likely he has > a browser or SSL middlebox incompatible with your SSL settings) > i try to confirm this (as it happens randomly its not that easy). > > Markus, please follow Willy's advise and remove all force-* configurations > from your bind line, you should use no-sslv3/no-tlsv1[0-2] keywords to > configure specific TLS version, but in this case, as long as you > troubleshooting this, I strongly suggest to not configure any specific TLS > settings. i have now removed them. my thought was to prevent use of "weaker" ssl-versions (like sslv2), but i found in the docs that this is deactivated per default. so no real need to force "newer", as sslv3 and tlsv1x are used per default. > > Also, we need the haproxy -vv output. You said you started running SSL > on haproxy April, 8 th, but dev23 was only released these days. So what > release did you run previsouly, and did you have the same problems (in > the browsers, not the log)? > i have activated ssl loadbalancing on 8th of april (not because of heartbleed). so i have only numbers starting at 8th of april. while testing i used ssl loadbalancing before and saw a few errors, that stopped me from activating ssl load balancing in haproxy in the first run. i have used all versions starting from 1.5 dev19 to now dev23. ./haproxy -vv HA-Proxy version 1.5-dev23-8317b28 2014/04/23 Copyright 2000-2014 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_OPENSSL=yes Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1 14 Mar 2012 Running on OpenSSL version : OpenSSL 1.0.1 14 Mar 2012 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built without PCRE support (using libc's regex instead) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. > > [1] https://www.ssllabs.com/ssltest/ > everything is OK, i see sslv2 is disabled ;-) just what i wanted when first using force-
RE: haproxy-1.5-dev23 and ssl handshake failure
Hi, > I've checked my own logs and found SSL handshake failures starting > on April 8th, or the day after Heartbleed was disclosed, as can be > seen below with the number of errors per day : Yes, please everyone specify whether there are actually users reporting this behavior, or if this is a log error only. We will see a lot of automated Heartbleet exploiting the next months, I'm sure. Check if a test @ssllabls [1] and others generates such an error. > my problem is, that i sometimes see an error message in my browser. i > also got one response from a user saying that he can't access our > ssl-pages and gets an error. There are 2 issues here: - the fact that you sometimes (?) see this error in the browser - the fact that one user can't open the ssl-page at all (likely he has a browser or SSL middlebox incompatible with your SSL settings) Markus, please follow Willy's advise and remove all force-* configurations from your bind line, you should use no-sslv3/no-tlsv1[0-2] keywords to configure specific TLS version, but in this case, as long as you troubleshooting this, I strongly suggest to not configure any specific TLS settings. Also, we need the haproxy -vv output. You said you started running SSL on haproxy April, 8 th, but dev23 was only released these days. So what release did you run previsouly, and did you have the same problems (in the browsers, not the log)? Exact browser and OS release informations are needed as well. [1] https://www.ssllabs.com/ssltest/
Re: haproxy-1.5-dev23 and ssl handshake failure
Hi all, On 22:59 Wed 23 Apr , Willy Tarreau wrote: > Hi again Markus, > > I've checked my own logs and found SSL handshake failures starting > on April 8th, or the day after Heartbleed was disclosed, as can be > seen below with the number of errors per day : > > # err date > 2 Mar 27 > 2 Mar 28 > 1 Mar 29 > 2 Mar 30 > 3 Mar 31 > 3 Apr 1 > 7 Apr 2 > 1 Apr 3 > 2 Apr 4 > 8 Apr 5 > 24 Apr 6 > 2 Apr 7 > 619 Apr 8 > 2 Apr 9 > 2 Apr 10 > 158 Apr 11 > 6 Apr 12 > 2 Apr 13 > 158 Apr 14 > 157 Apr 15 > 168 Apr 16 > 109 Apr 17 > 7 Apr 18 > 7 Apr 19 > 7 Apr 20 > 110 Apr 21 > 497 Apr 22 > 123 Apr 23 > > Interestingly, my version was neither upgraded nor restarted during this > period, so it cannot be caused by a code change, and is very likely caused > by bots trying the attack. So I think it's also possible that you're > experiencing the same things and that you didn't notice them before > upgrading and checking your logs. > > Hoping this helps, > Willy > > We see similar results with -dev19: 20140401378 20140402922 20140403346 20140404370 20140405807 20140406501 20140407445 20140408 3509 20140409360 20140410 1143 20140411 1525 20140412989 20140413991 20140414 1217 20140415 1139 20140416 1141 ... Note the spike on the 8th of April, matching the Heartbleed hypothesis. These can be all sorts of failures occurring before the handshake is completed. I sampled a couple of requests using tcpdump: one of them was a plain HTTP request on the HTTPS port and in the other one the client sent a close-notify TLS alert, 250 ms after receiving the certificate (indicating perhaps a network issue). To put things in perspective, on the 8th of April we had a total of 1.38 million SSL connections¹ so these failures account for roughly 0.25%. Granted that on that day we were expecting a lot of unfinished handshakes probing the heartbeat vulnerability, I wouldn't worry much. ¹ actually unique source IP:source port entries Regards, Apollon
Re: haproxy-1.5-dev23 and ssl handshake failure
Am 24.04.14 03:19, schrieb Stefan: > We also have a lot of "SSL handshake failure" records in log file > > Here some details on configs: > > - haproxy -vv: > HA-Proxy version 1.5-dev23-8317b28 2014/04/23 > Copyright 2000-2014 Willy Tarreau > > Build options : > TARGET = linux2628 > CPU = native > CC = gcc > CFLAGS = -m64 -march=x86-64 -O2 -march=native -g -fno-strict-aliasing > OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_ZLIB=1 > USE_OPENSSL=1 USE_STATIC_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built with zlib version : 1.2.8 > Compression algorithms supported : identity, deflate, gzip > Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 > Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 8.33 2013-05-28 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built with transparent proxy support using: > IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND > > Available polling systems : > epoll : pref=300, test result OK >poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > > > could you send our ssl config in haproxy? did you see those errors after 8th of april like willy and me (i have activated ssl loadbalancing on 8th of april, so i can't compare before heartbleed) markus
Re: haproxy-1.5-dev23 and ssl handshake failure
Am 23.04.14 22:59, schrieb Willy Tarreau: > Hi again Markus, > > I've checked my own logs and found SSL handshake failures starting > on April 8th, or the day after Heartbleed was disclosed, as can be > seen below with the number of errors per day : > > # err date > 2 Mar 27 > 2 Mar 28 > 1 Mar 29 > 2 Mar 30 > 3 Mar 31 > 3 Apr 1 > 7 Apr 2 > 1 Apr 3 > 2 Apr 4 > 8 Apr 5 > 24 Apr 6 > 2 Apr 7 > 619 Apr 8 > 2 Apr 9 > 2 Apr 10 > 158 Apr 11 > 6 Apr 12 > 2 Apr 13 > 158 Apr 14 > 157 Apr 15 > 168 Apr 16 > 109 Apr 17 > 7 Apr 18 > 7 Apr 19 > 7 Apr 20 > 110 Apr 21 > 497 Apr 22 > 123 Apr 23 > > Interestingly, my version was neither upgraded nor restarted during this > period, so it cannot be caused by a code change, and is very likely caused > by bots trying the attack. So I think it's also possible that you're > experiencing the same things and that you didn't notice them before > upgrading and checking your logs. > > Hoping this helps, > Willy > > thats really interesting. i can't compare with my numbers as i have activated ssl loadbalancing on 8th of april. i just checked all of my log files and data, because i first doubt this. so i can't compare my "old" numbers. so heartbleed could really be the cause of the high numbers. my problem is, that i sometimes see an error message in my browser. i also got one response from a user saying that he can't access our ssl-pages and gets an error. markus
Re: haproxy-1.5-dev23 and ssl handshake failure
We also have a lot of "SSL handshake failure" records in log file Here some details on configs: - haproxy -vv: HA-Proxy version 1.5-dev23-8317b28 2014/04/23 Copyright 2000-2014 Willy Tarreau Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -march=native -g -fno-strict-aliasing OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.33 2013-05-28 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll.
Re: haproxy-1.5-dev23 and ssl handshake failure
Hello, We also have the same issue. A lot of "SSL handshake failure" records in log file.
Re: haproxy-1.5-dev23 and ssl handshake failure
Hi again Markus, I've checked my own logs and found SSL handshake failures starting on April 8th, or the day after Heartbleed was disclosed, as can be seen below with the number of errors per day : # err date 2 Mar 27 2 Mar 28 1 Mar 29 2 Mar 30 3 Mar 31 3 Apr 1 7 Apr 2 1 Apr 3 2 Apr 4 8 Apr 5 24 Apr 6 2 Apr 7 619 Apr 8 2 Apr 9 2 Apr 10 158 Apr 11 6 Apr 12 2 Apr 13 158 Apr 14 157 Apr 15 168 Apr 16 109 Apr 17 7 Apr 18 7 Apr 19 7 Apr 20 110 Apr 21 497 Apr 22 123 Apr 23 Interestingly, my version was neither upgraded nor restarted during this period, so it cannot be caused by a code change, and is very likely caused by bots trying the attack. So I think it's also possible that you're experiencing the same things and that you didn't notice them before upgrading and checking your logs. Hoping this helps, Willy
Re: haproxy-1.5-dev23 and ssl handshake failure
Hi Markus, On Wed, Apr 23, 2014 at 08:00:21PM +0200, Markus Rietzler wrote: > today i have switch to dev23. everything is working very well in our > environment. haproxy works perfect in http mode. > load balancing our two backend servers with master/slave and backup setup. > > i also use haproxy for ssl terminiation. exakt: haproxy takes ssl requests to > our shop and then do ssl to the backend > servers with backup setup. > > so far everything works very good. > > only problem is that i see > > xx.xx.xx.xx:50281 [23/Apr/2014:19:49:03.771] https/1: SSL handshake failure > > those error messages in the log file. what happens here? sometimes i get an > error message in the browser, firefox gives > the error message: ssl_error_illegal_parameter_alert. but not always. > > this is the ssl config for haproxy What version were you running previously ? Could you please check with haproxy -vv ? This will also give us the exact openssl version in use. I'm seeing something strange in your config with "force-sslv3 force-tlsv10", because each "force" parameter should be unique, so in practice I think you're forcing TLSv1.0. Could you please try to remove these two statements to see if that fixes anything ? I'm leaving your config below for reference in case someone else has any idea. Willy > global > daemon > maxconn 2000 > stats socket/opt/haproxy/var/socket mode 0600 level admin > user www > group www > pidfile /opt/haproxy/var/pid > > defaults > mode http > log global > balance roundrobin > option httplog > option dontlognull > > retries 3 > option redispatch > option http-server-close > # option http-keep-alive > option forwardfor > > timeout connect 5000ms > timeout client 5ms > timeout server 5ms > > log 127.0.0.1 local0 > > frontend https > bind xx.xx.xx.xx:443 ssl crt /opt/haproxy/haproxy.ssl.crt force-sslv3 > force-tlsv10 ciphers > ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:!RC4+RSA:+HIGH:+MEDIUM > capture request header Host len 32 > default_backend lbhttps > monitor-uri /ok > reqadd X-Forwarded-Proto:\ https > > > backend lbhttps > server master yy.yy.yy.yy:443 ssl maxconn 50 check weight 1 inter 5s > rise 3 fall 2 verify none > server slave zz.zz.zz.zz:443 ssl maxconn 50 check backup weight 1 inter > 5s rise 3 fall 2 verify none >
haproxy-1.5-dev23 and ssl handshake failure
today i have switch to dev23. everything is working very well in our environment. haproxy works perfect in http mode. load balancing our two backend servers with master/slave and backup setup. i also use haproxy for ssl terminiation. exakt: haproxy takes ssl requests to our shop and then do ssl to the backend servers with backup setup. so far everything works very good. only problem is that i see xx.xx.xx.xx:50281 [23/Apr/2014:19:49:03.771] https/1: SSL handshake failure those error messages in the log file. what happens here? sometimes i get an error message in the browser, firefox gives the error message: ssl_error_illegal_parameter_alert. but not always. this is the ssl config for haproxy global daemon maxconn 2000 stats socket/opt/haproxy/var/socket mode 0600 level admin user www group www pidfile /opt/haproxy/var/pid defaults mode http log global balance roundrobin option httplog option dontlognull retries 3 option redispatch option http-server-close # option http-keep-alive option forwardfor timeout connect 5000ms timeout client 5ms timeout server 5ms log 127.0.0.1 local0 frontend https bind xx.xx.xx.xx:443 ssl crt /opt/haproxy/haproxy.ssl.crt force-sslv3 force-tlsv10 ciphers ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:!RC4+RSA:+HIGH:+MEDIUM capture request header Host len 32 default_backend lbhttps monitor-uri /ok reqadd X-Forwarded-Proto:\ https backend lbhttps server master yy.yy.yy.yy:443 ssl maxconn 50 check weight 1 inter 5s rise 3 fall 2 verify none server slave zz.zz.zz.zz:443 ssl maxconn 50 check backup weight 1 inter 5s rise 3 fall 2 verify none
RE: SSL handshake failure
Hi, > Lukas, > > The folks that access our service via the ColdFusion CFHTTP method > report errors, because those calls fail, and thus they are not getting > the requested data. It fails sporadically for some clients or it never works for specific clients? I would suggest: - collect as much informations as possible about those clients (OS, platform, ssllib, browser, sw/hw firewall + other security products) - try to find something affected clients all have in common and if possible: - try reproducing it - post the output of haproxy -vv, so we can see what openssl release you are running - catch a failed request with tcpdump (-s0 to avoid truncating the packets; write it to a pcap file, not just stdout output) This is most likely client related. Regards, Lukas
Re: SSL handshake failure
Lukas, The folks that access our service via the ColdFusion CFHTTP method report errors, because those calls fail, and thus they are not getting the requested data. -- Thomas Best, Thomas Amsler http://gplus.to/tamsler On Thu, Oct 17, 2013 at 10:41 AM, Lukas Tribus wrote: > Hi Thomas! > > > We are using HAProxy v1.5-dev19, and are seeing a lot of the following > > errors in our haproxy logs: > > > > <-- snip --> > > Oct 16 02:24:22 localhost haproxy[2473]: :44950 > > [16/Oct/2013:02:24:22.643] https-in/1: SSL handshake failure > > Oct 16 02:30:47 localhost haproxy[2473]: :37530 > > [16/Oct/2013:02:30:47.436] https-in/1: SSL handshake failure > > Oct 16 02:32:09 localhost haproxy[2473]: :32930 > > [16/Oct/2013:02:32:08.709] https-in/1: SSL handshake failure > > Oct 16 02:32:28 localhost haproxy[2473]: :38069 > > [16/Oct/2013:02:32:27.731] https-in/1: SSL handshake failure > > <-- snip --> > > > > This error occurs at a rate of 0.7%. It most often happens via > > ColdFusion CFHTTP connections. Could there be any issues with HAProxy > > or is this a client connection issue? > > Did you have an actual customer reporting issues or are you just seeing > this errors in the log? > > Until you have a reported and confirmed issue, I wouldn't worry too much. > Clients may do bogus things, have transient network issues or simply die. > All those things may generate the error above. > > Other software simply doesn't show you those errors. > > > > Lukas
RE: SSL handshake failure
Hi Thomas! > We are using HAProxy v1.5-dev19, and are seeing a lot of the following > errors in our haproxy logs: > > <-- snip --> > Oct 16 02:24:22 localhost haproxy[2473]: :44950 > [16/Oct/2013:02:24:22.643] https-in/1: SSL handshake failure > Oct 16 02:30:47 localhost haproxy[2473]: :37530 > [16/Oct/2013:02:30:47.436] https-in/1: SSL handshake failure > Oct 16 02:32:09 localhost haproxy[2473]: :32930 > [16/Oct/2013:02:32:08.709] https-in/1: SSL handshake failure > Oct 16 02:32:28 localhost haproxy[2473]: :38069 > [16/Oct/2013:02:32:27.731] https-in/1: SSL handshake failure > <-- snip --> > > This error occurs at a rate of 0.7%. It most often happens via > ColdFusion CFHTTP connections. Could there be any issues with HAProxy > or is this a client connection issue? Did you have an actual customer reporting issues or are you just seeing this errors in the log? Until you have a reported and confirmed issue, I wouldn't worry too much. Clients may do bogus things, have transient network issues or simply die. All those things may generate the error above. Other software simply doesn't show you those errors. Lukas
Re: SSL handshake failure
Baptiste, Please see my inline comments below: > It could be related to opened but unused connections from some > browsers (chrome). > We know that the "SSL handshake failures" are related to connections made by the ColdFusion CFHTTP client. So in this specific case, there are no browsers involved. > The only way to confirm this behavior, is to capture some traffic > using tcpdump and see if you get connections reseted by HAProxy due to > client timeout with no client traffic or some real SSL handshake > failure. > Do you know what the various error conditions are in the HAProxy source code, that can generate this specific "SSL handshake failure" error message? > > Do you have an IPS in front of HAProxy? > What is an IPS? We are running our service on Amazon's AWS. Traffic hits HAProxy and it load balances requests to Node.js instances. Thank you very much for your quick response and help. Best, -- Thomas Amsler > Baptiste > > > On Wed, Oct 16, 2013 at 4:45 AM, Thomas Amsler wrote: > > Hello, > > > > We are using HAProxy v1.5-dev19, and are seeing a lot of the following > > errors in our haproxy logs: > > > > <-- snip --> > > Oct 16 02:24:22 localhost haproxy[2473]: :44950 > > [16/Oct/2013:02:24:22.643] https-in/1: SSL handshake failure > > Oct 16 02:30:47 localhost haproxy[2473]: :37530 > > [16/Oct/2013:02:30:47.436] https-in/1: SSL handshake failure > > Oct 16 02:32:09 localhost haproxy[2473]: :32930 > > [16/Oct/2013:02:32:08.709] https-in/1: SSL handshake failure > > Oct 16 02:32:28 localhost haproxy[2473]: :38069 > > [16/Oct/2013:02:32:27.731] https-in/1: SSL handshake failure > > <-- snip --> > > > > This error occurs at a rate of 0.7%. It most often happens via ColdFusion > > CFHTTP connections. Could there be any issues with HAProxy or is this a > > client connection issue? > > > > Our server infrastructure handles REST as well as Socket.io (WetSocket) > > connections. > > > > > > Our config file: > > > > > > global > > nbproc 1 > > daemon > > maxconn 8192 > > log 127.0.0.1 local0 > > user ec2-user > > group ec2-user > > chroot /var/lib/haproxy > > > > defaults > > mode http > > option httplog > > log global > > # Add x-forwarded-for header. > > option forwardfor > > option http-server-close > > timeout connect 5s > > timeout client 30s > > timeout server 30s > > # Long timeout for WebSocket connections. > > timeout tunnel 1h > > > > # Redirect HTTP to HTTPS > > frontend http-in > > bind *:80 > > acl is_aggiefeed hdr_end(host) -i aggiefeed.ucdavis.edu > > redirect prefix https://aggiefeed.ucdavis.edu code 301 if > is_aggiefeed > > > > # HTTPS > > frontend https-in > > bind *:443 ssl crt /home/ec2-user/ssl/aggiefeed.pem > > default_backend servers > > errorfile 503 /home/ec2-user/errorfiles/503.http > > > > backend servers > > balance roundrobin > > cookie SERVERID insert indirect nocache > > server server1 10.0.1.100:8080 cookie server1 weight 1 maxconn 4096 > > check > > server server2 10.0.1.101:8080 cookie server2 weight 1 maxconn 4096 > > check > > > > > > > > Best, > > Thomas Amsler > > http://gplus.to/tamsler >
Re: SSL handshake failure
Hi Thomas, It could be related to opened but unused connections from some browsers (chrome). The only way to confirm this behavior, is to capture some traffic using tcpdump and see if you get connections reseted by HAProxy due to client timeout with no client traffic or some real SSL handshake failure. Do you have an IPS in front of HAProxy? Baptiste On Wed, Oct 16, 2013 at 4:45 AM, Thomas Amsler wrote: > Hello, > > We are using HAProxy v1.5-dev19, and are seeing a lot of the following > errors in our haproxy logs: > > <-- snip --> > Oct 16 02:24:22 localhost haproxy[2473]: :44950 > [16/Oct/2013:02:24:22.643] https-in/1: SSL handshake failure > Oct 16 02:30:47 localhost haproxy[2473]: :37530 > [16/Oct/2013:02:30:47.436] https-in/1: SSL handshake failure > Oct 16 02:32:09 localhost haproxy[2473]: :32930 > [16/Oct/2013:02:32:08.709] https-in/1: SSL handshake failure > Oct 16 02:32:28 localhost haproxy[2473]: :38069 > [16/Oct/2013:02:32:27.731] https-in/1: SSL handshake failure > <-- snip --> > > This error occurs at a rate of 0.7%. It most often happens via ColdFusion > CFHTTP connections. Could there be any issues with HAProxy or is this a > client connection issue? > > Our server infrastructure handles REST as well as Socket.io (WetSocket) > connections. > > > Our config file: > > > global > nbproc 1 > daemon > maxconn 8192 > log 127.0.0.1 local0 > user ec2-user > group ec2-user > chroot /var/lib/haproxy > > defaults > mode http > option httplog > log global > # Add x-forwarded-for header. > option forwardfor > option http-server-close > timeout connect 5s > timeout client 30s > timeout server 30s > # Long timeout for WebSocket connections. > timeout tunnel 1h > > # Redirect HTTP to HTTPS > frontend http-in > bind *:80 > acl is_aggiefeed hdr_end(host) -i aggiefeed.ucdavis.edu > redirect prefix https://aggiefeed.ucdavis.edu code 301 if is_aggiefeed > > # HTTPS > frontend https-in > bind *:443 ssl crt /home/ec2-user/ssl/aggiefeed.pem > default_backend servers > errorfile 503 /home/ec2-user/errorfiles/503.http > > backend servers > balance roundrobin > cookie SERVERID insert indirect nocache > server server1 10.0.1.100:8080 cookie server1 weight 1 maxconn 4096 > check > server server2 10.0.1.101:8080 cookie server2 weight 1 maxconn 4096 > check > > > > Best, > Thomas Amsler > http://gplus.to/tamsler
SSL handshake failure
Hello, We are using HAProxy v1.5-dev19, and are seeing a lot of the following errors in our haproxy logs: <-- snip --> Oct 16 02:24:22 localhost haproxy[2473]: :44950 [16/Oct/2013:02:24:22.643] https-in/1: SSL handshake failure Oct 16 02:30:47 localhost haproxy[2473]: :37530 [16/Oct/2013:02:30:47.436] https-in/1: SSL handshake failure Oct 16 02:32:09 localhost haproxy[2473]: :32930 [16/Oct/2013:02:32:08.709] https-in/1: SSL handshake failure Oct 16 02:32:28 localhost haproxy[2473]: :38069 [16/Oct/2013:02:32:27.731] https-in/1: SSL handshake failure <-- snip --> This error occurs at a rate of 0.7%. It most often happens via ColdFusion CFHTTP connections. Could there be any issues with HAProxy or is this a client connection issue? Our server infrastructure handles REST as well as Socket.io (WetSocket) connections. Our config file: globalnbproc 1daemonmaxconn 8192log 127.0.0.1 local0 user ec2-usergroup ec2-userchroot /var/lib/haproxydefaults mode httpoption httploglog global# Add x-forwarded-for header.option forwardforoption http-server-closetimeout connect 5stimeout client 30stimeout server 30s# Long timeout for WebSocket connections.timeout tunnel 1h# Redirect HTTP to HTTPSfrontend http-inbind *:80acl is_aggiefeed hdr_end(host) -i aggiefeed.ucdavis.eduredirect prefix https://aggiefeed.ucdavis.edu code 301 if is_aggiefeed# HTTPSfrontend https-inbind *:443 ssl crt /home/ec2-user/ssl/aggiefeed.pem default_backend serverserrorfile 503 /home/ec2-user/errorfiles/503.httpbackend serversbalance roundrobincookie SERVERID insert indirect nocacheserver server1 10.0.1.100:8080 cookie server1 weight 1 maxconn 4096 check server server2 10.0.1.101:8080 cookie server2 weight 1 maxconn 4096 check Best, Thomas Amsler http://gplus.to/tamsler
Re: 'SSL handshake failure' errors
Hi Merton, It is a good way to capture the packets during SSL handshake by tcpdump or wireshark from your client to find out what error happens. I have used this method in debugging SSL feature in haproxy. FYI. Best Regards, Godbach On 2013/6/20 1:46, Merton Lister wrote: Thank you Lukas. We will see whether SSLv3 improves things. Best, Merton On Wed, Jun 19, 2013 at 1:15 AM, Lukas Tribus mailto:luky...@hotmail.com>> wrote: Hi Merton! don't forget to CC the mailing-list :) > Out of the 5 possible causes you listed, we probably can't do much > about the other ones. But we can control the above two from our end. I > suppose that most 'modern' browsers nowadays should be able to do TLS > v1.0, and SSLv3 is considered as a weaker choice (But I wonder if it > will make more compatible for clients to support both TLSv1.0 and > SSLv3?). The specific ciphers we've chosen is to speed up the SSL but > it might 'screen out' some clients. The issue is probably with embedded, mobile and outdated browsers. If you have a 5 year old Windows CE phone, chances are it will connect in SSLv3 only (for example). > We also see in the log that some clients connected/handshake > successfully initially on some page, but failed the handshake in > subsequent requests to other parts of the site. Keep in mind that a browsers may open a connection to accelerate a _possible_ future HTTP transaction - and if the users doesn't request another page, the connection will just be dropped. Those optimizations in browsers can trigger warnings on the server-side. > Any suggestion on making SSL as much compatible as possible while > keeping it fast? You may enable SSLv3 again and monitor the log. Regards, Lukas
Re: 'SSL handshake failure' errors
Thank you Lukas. We will see whether SSLv3 improves things. Best, Merton On Wed, Jun 19, 2013 at 1:15 AM, Lukas Tribus wrote: > Hi Merton! > > > don't forget to CC the mailing-list :) > > > > Out of the 5 possible causes you listed, we probably can't do much > > about the other ones. But we can control the above two from our end. I > > suppose that most 'modern' browsers nowadays should be able to do TLS > > v1.0, and SSLv3 is considered as a weaker choice (But I wonder if it > > will make more compatible for clients to support both TLSv1.0 and > > SSLv3?). The specific ciphers we've chosen is to speed up the SSL but > > it might 'screen out' some clients. > > The issue is probably with embedded, mobile and outdated browsers. > If you have a 5 year old Windows CE phone, chances are it will connect > in SSLv3 only (for example). > > > > > We also see in the log that some clients connected/handshake > > successfully initially on some page, but failed the handshake in > > subsequent requests to other parts of the site. > > Keep in mind that a browsers may open a connection to accelerate a > _possible_ future HTTP transaction - and if the users doesn't request > another page, the connection will just be dropped. > > Those optimizations in browsers can trigger warnings on the server-side. > > > > > Any suggestion on making SSL as much compatible as possible while > > keeping it fast? > > You may enable SSLv3 again and monitor the log. > > > > Regards, > > Lukas
RE: 'SSL handshake failure' errors
Hi Merton! don't forget to CC the mailing-list :) > Out of the 5 possible causes you listed, we probably can't do much > about the other ones. But we can control the above two from our end. I > suppose that most 'modern' browsers nowadays should be able to do TLS > v1.0, and SSLv3 is considered as a weaker choice (But I wonder if it > will make more compatible for clients to support both TLSv1.0 and > SSLv3?). The specific ciphers we've chosen is to speed up the SSL but > it might 'screen out' some clients. The issue is probably with embedded, mobile and outdated browsers. If you have a 5 year old Windows CE phone, chances are it will connect in SSLv3 only (for example). > We also see in the log that some clients connected/handshake > successfully initially on some page, but failed the handshake in > subsequent requests to other parts of the site. Keep in mind that a browsers may open a connection to accelerate a _possible_ future HTTP transaction - and if the users doesn't request another page, the connection will just be dropped. Those optimizations in browsers can trigger warnings on the server-side. > Any suggestion on making SSL as much compatible as possible while > keeping it fast? You may enable SSLv3 again and monitor the log. Regards, Lukas
RE: 'SSL handshake failure' errors
Hi Merton! > We are seeing a fair amount of 'SSL handshake failure' errors in our > log, and we are running HAProxy 1.5-dev18. I suggest to update to dev19. There are a lot of bug fixes, including a security fix since dev18, which you want to have if this is a production box. It will not make those warnings disappear however. > Any idea what causes these 'SSL handshake failure' errors? SSL/TLS clients not completing the SSL handshake due to: - not being at least TLSv1.0 compatible (you disabled SSLv3) - not matching your ciphers (you have a pretty specific configuration) - a user hit STOP in the browser - a transient network problem - a bug somewhere > SSL handshake failure Don't look at this from a server perspective. Look at every single client with problems and troubleshoot it from there. > Given our whole site uses SSL, this is impacting usability for users. Do you have actual user reports of failing SSL handshakes or is this a conclusion you based on the log? If the former, then collect all the informations you can get from those users, like OS, SSL stack, browser version, tcpdumps of the failed handshake, etc. If the latter is the case, then don't panic - this is normal if you have a internet facing server. Check "option dontlognull" if you want to mute those warnings. Cheers, Lukas
'SSL handshake failure' errors
Hi,
We are seeing a fair amount of 'SSL handshake failure' errors in our log,
and we are running HAProxy 1.5-dev18.
The pattern of errors is:
Jun 17 20:00:28 localhost.localdomain haproxy[26060]: 68.xxx.xx.216:56030
[17/Jun/2013:20:00:28.002] public/2: SSL handshake failure
The following are the relevant frontend settings in our config:
frontend public
modehttp
bind0.0.0.0:80 <http://0.0.0.0/>
bind0.0.0.0:443 ssl crt /etc/haproxy/ssl_wc/site.pem no-sslv3
ciphers RC4:HIGH:!EXP:!LOW:!RC2:!3DES:!SEED:!aNULL:!eNULL:!MD5:!EDH
option forwardfor except 127.0.0.1
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
redirect scheme https if !{ ssl_fc }
Any idea what causes these 'SSL handshake failure' errors? Given our whole
site uses SSL, this is impacting usability for users.
Best regards,
Merton
Re: SSL handshake failure
Hi Samat, On Mon, Feb 11, 2013 at 08:23:54PM +0400, Samat Galimov wrote: > First connectin, gracefully started & closed: > 0001:https.accept(0005)=0007 from [5.9.11.40:25188] > 0002:decipher.accept(0006)=0009 from [5.9.11.40:25188] > 0002:decipher.clicls[0009:] > 0002:decipher.closed[0009:] > 0001:https.srvcls[0007:0008] > 0001:https.clicls[0007:0008] > 0001:https.closed[0007:0008] > Second one, broken > 0003:https.accept(0005)=0007 from [5.9.11.40:64633] > 0003:https.clicls[0007:0008] > 0003:https.closed[0007:0008] Yesterday we fixed an issue which happens when the openssl lib used at runtime differs from the one used to build haproxy, and which can cause such things depending on the variations between the versions. Could you please retest with latest snapshot and also send the output of "haproxy -vv" ? Thanks, Willy
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
$ openssl ciphers -v 'ALL:!SSLv2:!eNULL:!aNULL:!LOW:!EXPORT:!kECDH:!MD5:@STRENGTH' \ | while read C dumb; do echo -n "# $C " openssl s_client -connect 176.31.104.63:443 -cipher $C < /dev/null > /dev/null 2>&1 \ && echo OK \ || echo FAIL \ done \ | sort -k 3 \ | column -t Replace 'echo FAIL \' with 'echo FAIL'
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi, If it can help, I've been in touch with Emeric about SSL handshake failure since some times now but it's maybe preferable to use the ML to share experience. I'm using the following cipher filter list : 'ALL:!SSLv2:!eNULL:!aNULL:!LOW:!EXPORT:!kECDH:!MD5:@STRENGTH' The PEM file I used is composed by the following : -BEGIN CERTIFICATE- <= Leaf cert -BEGIN CERTIFICATE- <= Intermediate cert -BEGIN CERTIFICATE- <= Root cert -BEGIN DH PARAMETERS- <= "openssl dhparam 4096" result -BEGIN DSA PARAMETERS- <= "openssl dsaparam 4096" result -BEGIN EC PARAMETERS- <= "openssl ecparam -name prime256v1" result -BEGIN RSA PRIVATE KEY- <= Dumbo jacket Here is the result on trying to use each cipher on the list : $ openssl ciphers -v 'ALL:!SSLv2:!eNULL:!aNULL:!LOW:!EXPORT:!kECDH:!MD5:@STRENGTH' \ | while read C dumb; do echo -n "# $C " openssl s_client -connect 176.31.104.63:443 -cipher $C < /dev/null > /dev/null 2>&1 \ && echo OK \ || echo FAIL \ done \ | sort -k 3 \ | column -t # DHE-DSS-AES128-GCM-SHA256 FAIL # DHE-DSS-AES128-SHA256 FAIL # DHE-DSS-AES128-SHA FAIL # DHE-DSS-AES256-GCM-SHA384 FAIL # DHE-DSS-AES256-SHA256 FAIL # DHE-DSS-AES256-SHA FAIL # DHE-DSS-CAMELLIA128-SHAFAIL # DHE-DSS-CAMELLIA256-SHAFAIL # DHE-DSS-SEED-SHA FAIL # ECDHE-ECDSA-AES128-GCM-SHA256 FAIL # ECDHE-ECDSA-AES128-SHA256 FAIL # ECDHE-ECDSA-AES128-SHA FAIL # ECDHE-ECDSA-AES256-GCM-SHA384 FAIL # ECDHE-ECDSA-AES256-SHA384 FAIL # ECDHE-ECDSA-AES256-SHA FAIL # ECDHE-ECDSA-DES-CBC3-SHA FAIL # ECDHE-ECDSA-RC4-SHAFAIL # EDH-DSS-DES-CBC3-SHA FAIL # PSK-3DES-EDE-CBC-SHA FAIL # PSK-AES128-CBC-SHA FAIL # PSK-AES256-CBC-SHA FAIL # PSK-RC4-SHAFAIL # SRP-DSS-3DES-EDE-CBC-SHA FAIL # SRP-DSS-AES-128-CBC-SHAFAIL # SRP-DSS-AES-256-CBC-SHAFAIL # SRP-RSA-3DES-EDE-CBC-SHA FAIL # SRP-RSA-AES-128-CBC-SHAFAIL # SRP-RSA-AES-256-CBC-SHAFAIL # AES128-GCM-SHA256 OK # AES128-SHA256 OK # AES128-SHA OK # AES256-GCM-SHA384 OK # AES256-SHA256 OK # AES256-SHA OK # CAMELLIA128-SHAOK # CAMELLIA256-SHAOK # DES-CBC3-SHA OK # DHE-RSA-AES128-GCM-SHA256 OK # DHE-RSA-AES128-SHA256 OK # DHE-RSA-AES128-SHA OK # DHE-RSA-AES256-GCM-SHA384 OK # DHE-RSA-AES256-SHA256 OK # DHE-RSA-AES256-SHA OK # DHE-RSA-CAMELLIA128-SHAOK # DHE-RSA-CAMELLIA256-SHAOK # DHE-RSA-SEED-SHA OK # ECDHE-RSA-AES128-GCM-SHA256OK # ECDHE-RSA-AES128-SHA256OK # ECDHE-RSA-AES128-SHA OK # ECDHE-RSA-AES256-GCM-SHA384OK # ECDHE-RSA-AES256-SHA384OK # ECDHE-RSA-AES256-SHA OK # ECDHE-RSA-DES-CBC3-SHA OK # ECDHE-RSA-RC4-SHA OK # EDH-RSA-DES-CBC3-SHA OK # IDEA-CBC-SHA OK # RC4-SHAOK # SEED-SHA OK On the client (openssl s_client) it give : error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:741: --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE On the server : SSL handshake failure Also, while debugging haproxy v1.5-dev18-34-gf27af0d I can see that PEM_read_bio_DHparams() called in ssl_sock_load_dh_params() return the origin (in PEM file) DH parameter and that ssl_sock_load_dh_params() return 1. beber On 2013-04-19 20:53, Connelly, Zachary (CGI Federal) wrote: HAProxy list, I am currently working to implement SSL within HAProxy using the 1.5-dev18 version. Much like the thread started by Samat Galimov [1] on 2/5/2013, I am seeing the same behavior where the first time I send a request via SSL the request is serviced and everything is fine; the next time the same request is attempted I receive 'ERROR:Exception in request: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake.' I noticed the attached code in the thread was not put into the dev18 version (I believe). Did that code end up resolving the issue or is the issue still being reviewed? I can supply my config file if that would help. Is there any way to get more info out of HAProxy to see what it is doing while it handles the SSL Handshake (the log does not seem to write anything when the request fails)? Any assistance would be appreciated. Thanks, Zack Connelly Links: -- [1] http://search.gmane.org/?author=Samat+Galimov&sort=date
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
On Fri, Apr 26, 2013 at 06:22:57PM +, Connelly, Zachary (CGI Federal) wrote: > Two things: > > > > 1. After taking the two patches, ran version and am definitely getting > different versions. I'll have to look into how this could be with the admins > some more. > > Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 > > Running on OpenSSL version : OpenSSL 0.9.8y 5 Feb 2013 (VERSIONS DIFFER!) > > 2. The fix for the SSL dereference looks to have solved the issue! I am > no longer getting a crash and SSL seems to be handling the offload correctly > without issue. > > > > Thank you so much again, I really appreciate the quick support and patience > (as you can probably guess I am not the strongest Linux/C developer)! Thank you for the tests and reports, Zack. I tend to be confident that if it runs OK for you it should be stable, but I have no idea how your lib might handle some 1.0-only extensions. Cheers, Willy
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Two things: 1. After taking the two patches, ran version and am definitely getting different versions. I'll have to look into how this could be with the admins some more. Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 Running on OpenSSL version : OpenSSL 0.9.8y 5 Feb 2013 (VERSIONS DIFFER!) 2. The fix for the SSL dereference looks to have solved the issue! I am no longer getting a crash and SSL seems to be handling the offload correctly without issue. Thank you so much again, I really appreciate the quick support and patience (as you can probably guess I am not the strongest Linux/C developer)! Zack
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Thanks Willy/Emeric! I will try and track down the OpenSSL and we have and ensure we got the right versions. I did add the ADDINC parameter to the build to explicitly point to the include linked with the lib and same error occurred. I will also download the two fixes from today and see if the dereference especially helps resolve the issue. Zack
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
On Fri, Apr 26, 2013 at 06:25:38PM +0200, Willy Tarreau wrote: > We've checked with Emeric and I can confirm that the SSL struct changed > between the two versions, which exactly explains the 8 bytes offset we > found for ssl->sid_ctx_length which pointed to some wrong location. > > I have added a second control in the sources to report both the build > version and the runtime version. Just an update, Emeric has removed the dereference on the SSL struct so that this issue does not happen in the future. But still you need to fix your build issue anyway, and check with haproxy -vv that build and runtime versions are the same. Willy
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
At least on Debian, you need to hack around the includes a bit if you compile with USE_PCRE=1 and have the libssl-dev package installed. Because both the PCRE headers and the system-provided openssl headers are located in /usr/include and USE_PCRE adds an include for that directory, the openssl headers there are earlier in the search path than the ones defined by ADD_LIB. I have incorporated that "hack" which "hijacks" the PCRE_DIR argument to load the openssl library into my Chef cookbook to compile HAProxy (and optionally OpenSSL) from source at https://github.com/meineerde-cookbooks/haproxy/blob/master/recipes/source.rb#L125-L144 Maybe the Makefile can be adapted to allow an easier override of the OpenSSL path but I'm not completely sure how. Maybe I can have a look later. --Holger Connelly, Zachary (CGI Federal) wrote: Emeric, I'm not sure about that either actually. We definitely only have 0.9.8~ versions on the box and I explicitly reference the 0.9.8y library when I compile the executable: TARGET=linux26 USE_PCRE=1 USE_OPENSSL=1 ADDLIB=-L/usr/local/openssl-0.9.8y/lib LDFLAGS+=-ldl** Zack -Original Message- From: Emeric Brun [mailto:eb...@exceliance.fr] Sent: Friday, April 26, 2013 6:04 AM To: Connelly, Zachary (CGI Federal) Cc: Lukas Tribus; Baptiste; haproxy@formilux.org Subject: Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi don't understand: You said using openssl version 0.9.8y, but haproxy -vv shows OpenSSL 1.0.0a. Emeric On 04/25/2013 04:45 PM, Connelly, Zachary (CGI Federal) wrote: Lukas (et al), Here’s what I have so far: 1.use latest snapshot from [1] – *I’ll* *work on this today* 2.provide the output of haproxy –vv – *Output below* Sharing sig_handlers with pipe Sharing pendconn with pipe HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau mailto:w...@1wt.eu>> Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -g -O0 OPTIONS = USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. 3.can you tell us OS, kernel and openssl version? *Linux 5.5, 2.6.18-164.11.1.el5, openssl version 0.9.8y* 4.compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] *Done* 5.catch a backtrace of the crash with gdb (see [2] if you need details) – *Will work on this once #1 is complete from above* Thanks for the assistance so far, Zack *From:*Lukas Tribus [mailto:luky...@hotmail.com] *Sent:* Wednesday, April 24, 2013 12:36 PM *To:* Connelly, Zachary (CGI Federal); Baptiste *Cc:* haproxy@formilux.org <mailto:haproxy@formilux.org> *Subject:* RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi! > Please also note that the second SOAP call made that fails the > handshake also causes the HAProxy server to crash. Could you: - use latest snapshot from [1] - provide the output of haproxy -vv - can you tell us OS, kernel and openssl version? - compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] - catch a backtrace of the crash with gdb (see [2] if you need details) Regards, Lukas [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ [2] http://www.mail-archive.com/haproxy@formilux.org/msg09472.html
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Zack, On Fri, Apr 26, 2013 at 02:12:46PM +, Connelly, Zachary (CGI Federal) wrote: > Emeric, > > I'm not sure about that either actually. We definitely only have 0.9.8~ > versions on the box and I explicitly reference the 0.9.8y library when I > compile the executable: > > TARGET=linux26 USE_PCRE=1 USE_OPENSSL=1 > ADDLIB=-L/usr/local/openssl-0.9.8y/lib LDFLAGS+=-ldl You're missing a ADDINC somewhere, so I guess you're building with another version's headers and with this version's lib, which explains why haproxy -vv reports 1.0.0a (it reports the version used for building). We've checked with Emeric and I can confirm that the SSL struct changed between the two versions, which exactly explains the 8 bytes offset we found for ssl->sid_ctx_length which pointed to some wrong location. I have added a second control in the sources to report both the build version and the runtime version. Cheers, Willy
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Emeric, I'm not sure about that either actually. We definitely only have 0.9.8~ versions on the box and I explicitly reference the 0.9.8y library when I compile the executable: TARGET=linux26 USE_PCRE=1 USE_OPENSSL=1 ADDLIB=-L/usr/local/openssl-0.9.8y/lib LDFLAGS+=-ldl Zack -Original Message- From: Emeric Brun [mailto:eb...@exceliance.fr] Sent: Friday, April 26, 2013 6:04 AM To: Connelly, Zachary (CGI Federal) Cc: Lukas Tribus; Baptiste; haproxy@formilux.org Subject: Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi don't understand: You said using openssl version 0.9.8y, but haproxy -vv shows OpenSSL 1.0.0a. Emeric On 04/25/2013 04:45 PM, Connelly, Zachary (CGI Federal) wrote: > Lukas (et al), > > Here's what I have so far: > > 1.use latest snapshot from [1] - *I'll* *work on this today* > > 2.provide the output of haproxy -vv - *Output below* > > Sharing sig_handlers with pipe > > Sharing pendconn with pipe > > HA-Proxy version 1.5-dev18 2013/04/03 > > Copyright 2000-2013 Willy Tarreau mailto:w...@1wt.eu>> > > Build options : > >TARGET = linux26 > >CPU = generic > >CC = gcc > >CFLAGS = -g -O0 > >OPTIONS = USE_OPENSSL=1 USE_PCRE=1 > > Default settings : > >maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = > 200 > > Encrypted password support via crypt(3): yes > > Built without zlib support (USE_ZLIB not set) > > Compression algorithms supported : identity > > Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 > > OpenSSL library supports TLS extensions : yes > > OpenSSL library supports SNI : yes > > OpenSSL library supports prefer-server-ciphers : yes > > Available polling systems : > >epoll : pref=300, test result OK > > poll : pref=200, test result OK > > select : pref=150, test result OK > > Total: 3 (3 usable), will use epoll. > > 3.can you tell us OS, kernel and openssl version? *Linux 5.5, > 2.6.18-164.11.1.el5, openssl version 0.9.8y* > > 4.compile haproxy with debug and without compiler optimizations: make > DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] *Done* > > 5.catch a backtrace of the crash with gdb (see [2] if you need > details) - *Will work on this once #1 is complete from above* > > Thanks for the assistance so far, > > Zack > > *From:*Lukas Tribus [mailto:luky...@hotmail.com] > *Sent:* Wednesday, April 24, 2013 12:36 PM > *To:* Connelly, Zachary (CGI Federal); Baptiste > *Cc:* haproxy@formilux.org<mailto:haproxy@formilux.org> > *Subject:* RE: Follow-up on thread 'SSL handshake failure' from > 2/5/2013 > > Hi! > > >> Please also note that the second SOAP call made that fails the >> handshake also causes the HAProxy server to crash. > > Could you: > - use latest snapshot from [1] > - provide the output of haproxy -vv > - can you tell us OS, kernel and openssl version? > - compile haproxy with debug and without compiler optimizations: > make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] > - catch a backtrace of the crash with gdb (see [2] if you need > details) > > > Regards, > Lukas > > [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ > [2] http://www.mail-archive.com/haproxy@formilux.org/msg09472.html >
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi don't understand: You said using openssl version 0.9.8y, but haproxy -vv shows OpenSSL 1.0.0a. Emeric On 04/25/2013 04:45 PM, Connelly, Zachary (CGI Federal) wrote: Lukas (et al), Here’s what I have so far: 1.use latest snapshot from [1] – *I’ll* *work on this today* 2.provide the output of haproxy –vv – *Output below* Sharing sig_handlers with pipe Sharing pendconn with pipe HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -g -O0 OPTIONS = USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. 3.can you tell us OS, kernel and openssl version? *Linux 5.5, 2.6.18-164.11.1.el5, openssl version 0.9.8y* 4.compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] *Done* 5.catch a backtrace of the crash with gdb (see [2] if you need details) – *Will work on this once #1 is complete from above* Thanks for the assistance so far, Zack *From:*Lukas Tribus [mailto:luky...@hotmail.com] *Sent:* Wednesday, April 24, 2013 12:36 PM *To:* Connelly, Zachary (CGI Federal); Baptiste *Cc:* haproxy@formilux.org *Subject:* RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi! Please also note that the second SOAP call made that fails the handshake also causes the HAProxy server to crash. Could you: - use latest snapshot from [1] - provide the output of haproxy -vv - can you tell us OS, kernel and openssl version? - compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] - catch a backtrace of the crash with gdb (see [2] if you need details) Regards, Lukas [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ [2] http://www.mail-archive.com/haproxy@formilux.org/msg09472.html
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi! > report the exact snapshot you used. He is at current HEAD by using 20130425 with c621d36ba applied manually on it (linux 2.6.18 without tproxy support). He also saw the crashes in -dev18, but I had him update the code. Thanks, Lukas
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi Zack, On Thu, Apr 25, 2013 at 08:46:57PM +, Connelly, Zachary (CGI Federal) wrote: > Lukas (et al), > > I pulled down the latest code and compiled (thanks for the build fix). I'm > still getting the same problem with the latest code. Despite compiling with > the debug options as specified we're not seeing a core dump come out yet to > then do the backtrace against. I'm still looking around for it unless anyone > knows what I might be missing. > > Let me know if there is anything else I can run and/or provide. Thanks for the trace you sent me. It would really help if you could send me the core file with the equivalent executable, and report the exact snapshot you used. I think that this crash explains why it randomly fails. Something is not working as expected, and when we're lucky, it crashes, but when we're unlucky, it probably silently uses an incorrect memory location explaining why the rest of the processing fails. The patch that was discussed in the thread you mentionned had no reason for changing anything, instead of clearing the errors unconditionally, it only cleared them if there were any. It's more of an optimization and the fact that it changed something indicates that we have a random bug somewhere. And your trace proves that ! Thanks, Willy
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi Zack, your OS probably doesn't generate coredumps by default. It can be tricky to generate them. Make sure: - haproxy is started as root - it doesn't use chrooting - don't use the init script, but start haproxy manually right after you used the ulimit command - otherwise follow the description here: http://www.mail-archive.com/haproxy@formilux.org/msg09472.html The ´cat /proc/sys/kernel/core_pattern´ setting specifies in what directory the coredump is saved and By following the above suggestion you make your setup less secure, so use it with care and revert those things after you obtained the coredump. The coredump itself and the gdb output as well will contain things like ip addresses, etc. Please be aware of that when you post it to the mailing list. If you cannot publish such data you may just sent the gdb output to Willy Tarreau instead of the mailing list. I'm CC'ing the list, maybe someone else finds this useful. Regards, Lukas > From: zachary.conne...@cgifederal.com > To: luky...@hotmail.com > Subject: RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 > Date: Thu, 25 Apr 2013 20:41:47 + > > > Lukas, > > > > Thanks, the change fixed the compile problem. Testing out the latest > version now... > > > > Do you happen to know where the core dump would end up? So far we > haven't found it but the server has definitely come down after the > second SOAP request. > > > > Thanks, > Zack > > > > -Original Message- > From: Lukas Tribus [mailto:luky...@hotmail.com] > Sent: Thursday, April 25, 2013 4:19 PM > To: Connelly, Zachary (CGI Federal) > Subject: RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 > > > > Hi Zack, > > > > in fact there was a build breakage in the development snapshots. > > > > > > Please apply the following patch or wait for tonights snapshot, which > will include the patch as well: > > > > http://haproxy.1wt.eu/git?p=haproxy.git;a=commitdiff_plain;h=c621d36ba3ff8b4ca2907f4dc966cfb13cbf3451<http://haproxy.1wt.eu/git?p=haproxy.git%3ba=commitdiff_plain%3bh=c621d36ba3ff8b4ca2907f4dc966cfb13cbf3451> > > > > > > > > > Regards, > > Lukas
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Lukas (et al), I pulled down the latest code and compiled (thanks for the build fix). I'm still getting the same problem with the latest code. Despite compiling with the debug options as specified we're not seeing a core dump come out yet to then do the backtrace against. I'm still looking around for it unless anyone knows what I might be missing. Let me know if there is anything else I can run and/or provide. Thanks for the assistance so far, Zack From: Connelly, Zachary (CGI Federal) Sent: Thursday, April 25, 2013 10:46 AM To: 'Lukas Tribus'; Baptiste Cc: haproxy@formilux.org Subject: RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Lukas (et al), Here's what I have so far: 1. use latest snapshot from [1] - I'll work on this today 2. provide the output of haproxy -vv - Output below Sharing sig_handlers with pipe Sharing pendconn with pipe HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau mailto:w...@1wt.eu>> Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -g -O0 OPTIONS = USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. 3. can you tell us OS, kernel and openssl version? Linux 5.5, 2.6.18-164.11.1.el5, openssl version 0.9.8y 4. compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] Done 5. catch a backtrace of the crash with gdb (see [2] if you need details) - Will work on this once #1 is complete from above Thanks for the assistance so far, Zack From: Lukas Tribus [mailto:luky...@hotmail.com] Sent: Wednesday, April 24, 2013 12:36 PM To: Connelly, Zachary (CGI Federal); Baptiste Cc: haproxy@formilux.org<mailto:haproxy@formilux.org> Subject: RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi! > Please also note that the second SOAP call made that fails > the handshake also causes the HAProxy server to crash. Could you: - use latest snapshot from [1] - provide the output of haproxy -vv - can you tell us OS, kernel and openssl version? - compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] - catch a backtrace of the crash with gdb (see [2] if you need details) Regards, Lukas [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ [2] http://www.mail-archive.com/haproxy@formilux.org/msg09472.html
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Lukas (et al), Here's what I have so far: 1. use latest snapshot from [1] - I'll work on this today 2. provide the output of haproxy -vv - Output below Sharing sig_handlers with pipe Sharing pendconn with pipe HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -g -O0 OPTIONS = USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.0a 1 Jun 2010 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. 3. can you tell us OS, kernel and openssl version? Linux 5.5, 2.6.18-164.11.1.el5, openssl version 0.9.8y 4. compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] Done 5. catch a backtrace of the crash with gdb (see [2] if you need details) - Will work on this once #1 is complete from above Thanks for the assistance so far, Zack From: Lukas Tribus [mailto:luky...@hotmail.com] Sent: Wednesday, April 24, 2013 12:36 PM To: Connelly, Zachary (CGI Federal); Baptiste Cc: haproxy@formilux.org Subject: RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi! > Please also note that the second SOAP call made that fails > the handshake also causes the HAProxy server to crash. Could you: - use latest snapshot from [1] - provide the output of haproxy -vv - can you tell us OS, kernel and openssl version? - compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] - catch a backtrace of the crash with gdb (see [2] if you need details) Regards, Lukas [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ [2] http://www.mail-archive.com/haproxy@formilux.org/msg09472.html
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi! > Please also note that the second SOAP call made that fails > the handshake also causes the HAProxy server to crash. Could you: - use latest snapshot from [1] - provide the output of haproxy -vv - can you tell us OS, kernel and openssl version? - compile haproxy with debug and without compiler optimizations: make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=[...] - catch a backtrace of the crash with gdb (see [2] if you need details) Regards, Lukas [1] http://haproxy.1wt.eu/download/1.5/src/snapshot/ [2] http://www.mail-archive.com/haproxy@formilux.org/msg09472.html
RE: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Baptiste, Thanks for the advice. I am trying to receive an SSL request into HAProxy then pass along to the back-end server as http. The back-end server is a simple SOAP service that responds on http and we want HAProxy to respond to the client on https. We are not redirecting on the back-end in anyway, just receiving http from HAProxy after the SSL offload and responding with http to HAProxy. When that occurs we are seeing the error described here: http://comments.gmane.org/gmane.comp.web.haproxy/10830. I was wondering if the code change described in this thread was implemented and/or successful. Please also note that the second SOAP call made that fails the handshake also causes the HAProxy server to crash. Here are the front and back end sections for reference: frontend http-in bind xx.xx.xx.xx:80 #actual IP removed bind xx.xx.xx.xx:443 ssl crt /usr/local/cdx/apache/ssl/combined.pem id 100 #actual IP removed option http-server-close default_backend devngn1 capture response header Location len 32 capture response header Set-Cookie len 32 backend devngn1 balance roundrobin reqrep ^([^\ :]*)\ /generic(.*) \1\ /specific-path-location\2 #actual path removed server app1 xx.xx.xx.xx:80 Thanks, Zack -Original Message- From: Baptiste [mailto:bed...@gmail.com] Sent: Monday, April 22, 2013 2:43 AM To: Connelly, Zachary (CGI Federal) Cc: haproxy@formilux.org<mailto:haproxy@formilux.org> Subject: Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013 Hi Zachary, It sounds your application server is not aware the connections was made over a SSL socket on HAProxy frontend and tries to redirect the user on the same socket but on HTTP protocol. To figure out if this is really the case, and to know how to fix it, you can read this blog article: http://blog.exceliance.fr/2013/02/26/ssl-offloading-impact-on-web-applications/ Baptiste On Fri, Apr 19, 2013 at 8:53 PM, Connelly, Zachary (CGI Federal) mailto:zachary.conne...@cgifederal.com>> wrote: > HAProxy list, > > > > I am currently working to implement SSL within HAProxy using the > 1.5-dev18 version. Much like the thread started by Samat Galimov on > 2/5/2013, I am seeing the same behavior where the first time I send a > request via SSL the request is serviced and everything is fine; the > next time the same request is attempted I receive 'ERROR:Exception in request: > javax.net.ssl.SSLHandshakeException: Remote host closed connection > during handshake.' I noticed the attached code in the thread was not > put into the > dev18 version (I believe). Did that code end up resolving the issue or > is the issue still being reviewed? I can supply my config file if that > would help. Is there any way to get more info out of HAProxy to see > what it is doing while it handles the SSL Handshake (the log does not > seem to write anything when the request fails)? > > > > Any assistance would be appreciated. Thanks, > > Zack Connelly
Re: Follow-up on thread 'SSL handshake failure' from 2/5/2013
Hi Zachary, It sounds your application server is not aware the connections was made over a SSL socket on HAProxy frontend and tries to redirect the user on the same socket but on HTTP protocol. To figure out if this is really the case, and to know how to fix it, you can read this blog article: http://blog.exceliance.fr/2013/02/26/ssl-offloading-impact-on-web-applications/ Baptiste On Fri, Apr 19, 2013 at 8:53 PM, Connelly, Zachary (CGI Federal) wrote: > HAProxy list, > > > > I am currently working to implement SSL within HAProxy using the 1.5-dev18 > version. Much like the thread started by Samat Galimov on 2/5/2013, I am > seeing the same behavior where the first time I send a request via SSL the > request is serviced and everything is fine; the next time the same request > is attempted I receive ‘ERROR:Exception in request: > javax.net.ssl.SSLHandshakeException: Remote host closed connection during > handshake.’ I noticed the attached code in the thread was not put into the > dev18 version (I believe). Did that code end up resolving the issue or is > the issue still being reviewed? I can supply my config file if that would > help. Is there any way to get more info out of HAProxy to see what it is > doing while it handles the SSL Handshake (the log does not seem to write > anything when the request fails)? > > > > Any assistance would be appreciated. Thanks, > > Zack Connelly
Re: SSL handshake failure
First connectin, gracefully started & closed: 0001:https.accept(0005)=0007 from [5.9.11.40:25188] 0002:decipher.accept(0006)=0009 from [5.9.11.40:25188] 0002:decipher.clicls[0009:] 0002:decipher.closed[0009:] 0001:https.srvcls[0007:0008] 0001:https.clicls[0007:0008] 0001:https.closed[0007:0008] Second one, broken 0003:https.accept(0005)=0007 from [5.9.11.40:64633] 0003:https.clicls[0007:0008] 0003:https.closed[0007:0008] On Mon, Feb 11, 2013 at 1:26 PM, Emeric Brun wrote: > On 02/08/2013 08:27 PM, Samat Galimov wrote: > >> You are right. >> After I start haproxy with patch it establishes first connection >> normally, as it should. >> All following connections are randomly dropping an error as before. >> >> >> > Could you check your logs, Do you see HTTP redirects (status codes 30x) > > I suspect your servers send redirects with a clear scheme in location > (http:// instead https://). > > > >
Re: SSL handshake failure
On 02/08/2013 08:27 PM, Samat Galimov wrote: You are right. After I start haproxy with patch it establishes first connection normally, as it should. All following connections are randomly dropping an error as before. Could you check your logs, Do you see HTTP redirects (status codes 30x) I suspect your servers send redirects with a clear scheme in location (http:// instead https://).
Re: SSL handshake failure
On Sat, Feb 09, 2013 at 08:45:50PM +0400, Samat Galimov wrote: > Can I help in any way? Not yes I think, I'll check with Emeric if we can find other locations where errors might remain uncaught. Thanks! Willy
Re: SSL handshake failure
Can I help in any way? On Fri, Feb 8, 2013 at 11:29 PM, Willy Tarreau wrote: > On Fri, Feb 08, 2013 at 11:27:31PM +0400, Samat Galimov wrote: > > You are right. > > After I start haproxy with patch it establishes first connection > normally, > > as it should. > > All following connections are randomly dropping an error as before. > > OK, so maybe there are still some places where the change is needed. We'll > audit the code again to try to find something else. > > Thanks, > Willy > >
Re: SSL handshake failure
On Fri, Feb 08, 2013 at 11:27:31PM +0400, Samat Galimov wrote: > You are right. > After I start haproxy with patch it establishes first connection normally, > as it should. > All following connections are randomly dropping an error as before. OK, so maybe there are still some places where the change is needed. We'll audit the code again to try to find something else. Thanks, Willy
Re: SSL handshake failure
You are right. After I start haproxy with patch it establishes first connection normally, as it should. All following connections are randomly dropping an error as before. On Thu, Feb 7, 2013 at 11:09 PM, Willy Tarreau wrote: > On Thu, Feb 07, 2013 at 09:22:37PM +0400, Samat Galimov wrote: > > Funny, with patch applied it establishes first connection after start > > normally. > > Then old thing continues. > > I'm unsure what you mean, do you mean the patch has slightly improved the > situation but not completely ? > > Willy > >
Re: SSL handshake failure
On Thu, Feb 07, 2013 at 09:22:37PM +0400, Samat Galimov wrote: > Funny, with patch applied it establishes first connection after start > normally. > Then old thing continues. I'm unsure what you mean, do you mean the patch has slightly improved the situation but not completely ? Willy
Re: SSL handshake failure
Funny, with patch applied it establishes first connection after start normally. Then old thing continues. On Thu, Feb 7, 2013 at 6:58 PM, Willy Tarreau wrote: > On Thu, Feb 07, 2013 at 06:49:14PM +0400, Samat Galimov wrote: > > Thank you very much, overlooked your email due to filters, sorry for > delay. > > I am very happy to help, sure I would accept a patch. > > Server is available from outside world but is not heavily used ??? we > dont > > point load to it because of this SSL errors. > > > > By the way, I am using default haproxy-devel port in FreeBSD tree, so > > > http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev17.tar.gzsource > > is being used. > > OK, please find it attached. You should apply it on top of your current > source tree and rebuild. > > Thanks, > Willy > >
Re: SSL handshake failure
On Thu, Feb 07, 2013 at 06:49:14PM +0400, Samat Galimov wrote:
> Thank you very much, overlooked your email due to filters, sorry for delay.
> I am very happy to help, sure I would accept a patch.
> Server is available from outside world but is not heavily used ??? we dont
> point load to it because of this SSL errors.
>
> By the way, I am using default haproxy-devel port in FreeBSD tree, so
> http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev17.tar.gzsource
> is being used.
OK, please find it attached. You should apply it on top of your current
source tree and rebuild.
Thanks,
Willy
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 87eff2b..07b1ca7 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -128,7 +128,8 @@ int ssl_sock_verifycbk(int ok, X509_STORE_CTX *x_store)
}
if (objt_listener(conn->target)->bind_conf->ca_ignerr & (1ULL
<< err)) {
- ERR_clear_error();
+ if (ERR_peek_error())
+ ERR_clear_error();
return 1;
}
@@ -141,7 +142,8 @@ int ssl_sock_verifycbk(int ok, X509_STORE_CTX *x_store)
/* check if certificate error needs to be ignored */
if (objt_listener(conn->target)->bind_conf->crt_ignerr & (1ULL << err))
{
- ERR_clear_error();
+ if (ERR_peek_error())
+ ERR_clear_error();
return 1;
}
@@ -885,6 +887,9 @@ int ssl_sock_handshake(struct connection *conn, unsigned
int flag)
if ((conn->flags & CO_FL_CONNECTED) &&
SSL_renegotiate_pending(conn->xprt_ctx)) {
char c;
+ if (unlikely(ERR_peek_error()))
+ ERR_clear_error();
+
ret = SSL_peek(conn->xprt_ctx, &c, 1);
if (ret <= 0) {
/* handshake may have not been completed, let's find
why */
@@ -942,6 +947,9 @@ int ssl_sock_handshake(struct connection *conn, unsigned
int flag)
goto reneg_ok;
}
+ if (unlikely(ERR_peek_error()))
+ ERR_clear_error();
+
ret = SSL_do_handshake(conn->xprt_ctx);
if (ret != 1) {
/* handshake did not complete, let's find why */
@@ -1008,7 +1016,8 @@ reneg_ok:
out_error:
/* Clear openssl global errors stack */
- ERR_clear_error();
+ if (ERR_peek_error())
+ ERR_clear_error();
/* free resumed session if exists */
if (objt_server(conn->target) &&
objt_server(conn->target)->ssl_ctx.reused_sess) {
@@ -1062,6 +1071,9 @@ static int ssl_sock_to_buf(struct connection *conn,
struct buffer *buf, int coun
* EINTR too.
*/
while (try) {
+ if (unlikely(ERR_peek_error()))
+ ERR_clear_error();
+
ret = SSL_read(conn->xprt_ctx, bi_end(buf), try);
if (conn->flags & CO_FL_ERROR) {
/* CO_FL_ERROR may be set by ssl_sock_infocbk */
@@ -1084,7 +1096,8 @@ static int ssl_sock_to_buf(struct connection *conn,
struct buffer *buf, int coun
conn->flags |= CO_FL_ERROR;
/* Clear openssl global errors stack */
- ERR_clear_error();
+ if (ERR_peek_error())
+ ERR_clear_error();
}
goto read0;
}
@@ -1118,7 +1131,8 @@ static int ssl_sock_to_buf(struct connection *conn,
struct buffer *buf, int coun
return done;
out_error:
/* Clear openssl global errors stack */
- ERR_clear_error();
+ if (ERR_peek_error())
+ ERR_clear_error();
conn->flags |= CO_FL_ERROR;
return done;
@@ -1158,6 +1172,9 @@ static int ssl_sock_from_buf(struct connection *conn,
struct buffer *buf, int fl
if (buf->data + try > buf->p)
try = buf->data + try - buf->p;
+ if (unlikely(ERR_peek_error()))
+ ERR_clear_error();
+
ret = SSL_write(conn->xprt_ctx, bo_ptr(buf), try);
if (conn->flags & CO_FL_ERROR) {
/* CO_FL_ERROR may be set by ssl_sock_infocbk */
@@ -1201,7 +1218,8 @@ static int ssl_sock_from_buf(struct connection *conn,
struct buffer *buf, int fl
out_error:
/* Clear openssl global errors stack */
- ERR_clear_error();
+ if (ERR_peek_error())
+ ERR_clear_error();
conn->flags |= CO_FL_ERROR;
return done;
@@ -1226,7 +1244,8 @@ static void ssl_sock_shutw(struct connection *conn, int
clean)
/* no handshake was in progress, try a clean ssl shutdown */
if (clean && (SSL_shutdown(conn->xprt_ctx) <= 0)) {
/* Clear openssl global errors stack */
- ERR_clear_error()
Re: SSL handshake failure
Thank you very much, overlooked your email due to filters, sorry for delay.
I am very happy to help, sure I would accept a patch.
Server is available from outside world but is not heavily used — we dont
point load to it because of this SSL errors.
By the way, I am using default haproxy-devel port in FreeBSD tree, so
http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev17.tar.gzsource
is being used.
On Wed, Feb 6, 2013 at 10:56 AM, Willy Tarreau wrote:
> Hello Samat,
>
> On Tue, Feb 05, 2013 at 12:39:20PM +0400, Samat Galimov wrote:
> > Hello,
> >
> > I have very strange behaviour of HA-Proxy version 1.5-dev17 2012/12/28 on
> > FreeBSD 9.0-Stable
> >
> > % openssl s_client -debug -servername dharma.zvq.me -connect
> > dharma.zvq.me:443 /usr/local/etc
> > CONNECTED(0003)
> > write to 0x801407160 [0x801525000] (128 bytes => 128 (0x80))
> > - 16 03 01 00 7b 01 00 00-77 03 01 51 10 6a 26 66 {...w..Q.j&f
> > 0010 - e8 2b 77 63 f9 ea 25 e8-b7 cb 51 84 0a d7 0d 7c .+wc..%...Q???.|
> > 0020 - 58 2c 32 6f 0f 54 94 c6-29 57 c4 00 00 34 00 39 X,2o.T..)W???4.9
> > 0030 - 00 38 00 35 00 88 00 87-00 84 00 16 00 13 00 0a .8.5..??
> > 0040 - 00 33 00 32 00 2f 00 45-00 44 00 41 00 05 00 04 .3.2./.E.D.A???.
> > 0050 - 00 15 00 12 00 09 00 14-00 11 00 08 00 06 00 03 .??.
> > 0060 - 00 ff 01 00 00 1a 00 00-00 12 00 10 00 00 0d 64 .??d
> > 0070 - 68 61 72 6d 61 2e 7a 76-71 2e 6d 65 00 23 harma.zvq.me.#
> > 0080 -
> > read from 0x801407160 [0x801577000] (7 bytes => 0 (0x0))
> > 42642:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
> >
> failure:/mnt/jq032hgn/usr/src/secure/lib/libssl/../../../crypto/openssl/ssl/s23_lib.c:182:
> > OpenSSL is 0.9.8q 2 Dec 2010
> >
> > It's randomly gives such a weird error, 50% chance, as I see.
>
> Are you the only one to access this service or is it in production and
> used by other people ? I'm asking because we had a similar report a few
> weeks ago of 0.9.8 on solaris experiencing random errors, and we suspected
> that the error queue was probably sometimes filled by some SSL calls
> without returning an error, and thus was not flushed.
>
> Would you accept to try a patch ? We have one to change the behaviour
> that we have still not merged due to the lack of testers experiencing
> the issue !
>
> > On server side (i run haproxy with -d) i get:
> > 000c:https.accept(0005)=0007 from [5.9.11.40:43423]
> > 000c:https.clicls[0007:0008]
> > 000c:https.closed[0007:0008]
> >
> > Here is my config:
> (...)
>
> I see nothing wrong in your configuration, and a config should not cause
> a random behaviour anyway. Also you're not in a chroot so it cannot be
> caused by a lack of entropy caused by the inability to access /dev/urandom.
>
> Willy
>
>
Re: SSL handshake failure
Hello Samat,
On Tue, Feb 05, 2013 at 12:39:20PM +0400, Samat Galimov wrote:
> Hello,
>
> I have very strange behaviour of HA-Proxy version 1.5-dev17 2012/12/28 on
> FreeBSD 9.0-Stable
>
> % openssl s_client -debug -servername dharma.zvq.me -connect
> dharma.zvq.me:443 /usr/local/etc
> CONNECTED(0003)
> write to 0x801407160 [0x801525000] (128 bytes => 128 (0x80))
> - 16 03 01 00 7b 01 00 00-77 03 01 51 10 6a 26 66 {...w..Q.j&f
> 0010 - e8 2b 77 63 f9 ea 25 e8-b7 cb 51 84 0a d7 0d 7c .+wc..%...Q???.|
> 0020 - 58 2c 32 6f 0f 54 94 c6-29 57 c4 00 00 34 00 39 X,2o.T..)W???4.9
> 0030 - 00 38 00 35 00 88 00 87-00 84 00 16 00 13 00 0a .8.5..??
> 0040 - 00 33 00 32 00 2f 00 45-00 44 00 41 00 05 00 04 .3.2./.E.D.A???.
> 0050 - 00 15 00 12 00 09 00 14-00 11 00 08 00 06 00 03 .??.
> 0060 - 00 ff 01 00 00 1a 00 00-00 12 00 10 00 00 0d 64 .??d
> 0070 - 68 61 72 6d 61 2e 7a 76-71 2e 6d 65 00 23 harma.zvq.me.#
> 0080 -
> read from 0x801407160 [0x801577000] (7 bytes => 0 (0x0))
> 42642:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
> failure:/mnt/jq032hgn/usr/src/secure/lib/libssl/../../../crypto/openssl/ssl/s23_lib.c:182:
> OpenSSL is 0.9.8q 2 Dec 2010
>
> It's randomly gives such a weird error, 50% chance, as I see.
Are you the only one to access this service or is it in production and
used by other people ? I'm asking because we had a similar report a few
weeks ago of 0.9.8 on solaris experiencing random errors, and we suspected
that the error queue was probably sometimes filled by some SSL calls
without returning an error, and thus was not flushed.
Would you accept to try a patch ? We have one to change the behaviour
that we have still not merged due to the lack of testers experiencing
the issue !
> On server side (i run haproxy with -d) i get:
> 000c:https.accept(0005)=0007 from [5.9.11.40:43423]
> 000c:https.clicls[0007:0008]
> 000c:https.closed[0007:0008]
>
> Here is my config:
(...)
I see nothing wrong in your configuration, and a config should not cause
a random behaviour anyway. Also you're not in a chroot so it cannot be
caused by a lack of entropy caused by the inability to access /dev/urandom.
Willy
