RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko, I did a couple of new tests. But as my self-compiled version 1.5.5 did behave weird sometimes I decided to have a try with the 1.5.2 from Red Hat which was packaged with RHEL 6.6. I will do a re-test tomorrow but: - the self-compiled version tried to use TLS 1.2 a couple of times although the ‘no-tlsv12’ option was set. When you are compiling, are you *always running make clean before the actual build? Could it be that you forgot it for your 1.5.5 build? While testing I got another idea we had trouble with some time ago. The certificate we use is a wildcard cert by DigiCert. I replaced it with a self-signed web certificate but it did not change anything. The results are identical. Do you have any further idea what might cause the problem? You have to use a haproxy binary that you can trust and the all the previous tests you have done with binary that ignores force-tlsv10 must be disregarded. So, if the reason of this was a missing make clean (that is just a wild, but valid guess), then please recompile latest v.1.5.6 cleanly and retry with dh-params set to 1024bit. On other thing that cames to my mind is intermediate certificates. Are intermediate certificates correctly installed on haproxy? Currently, riege.com correctly delivers [1] the intermediate certificate DigiCert High Assurance CA-3, but I guess thats still a stunnel setup? Please double check with haproxy and the openssl s_client command [2]. Regards, Lukas [1] https://www.ssllabs.com/ssltest/analyze.html?d=riege.com [2] http://blog.yimingliu.com/2008/02/04/testing-https-with-openssl/
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas, On other thing that cames to my mind is intermediate certificates. Are intermediate certificates correctly installed on haproxy? Here we got the problem and now it is clear to me — all the behavior we saw in the tcpdumps. The browsers have the intermediate cert installed and that´s why it looked good there. After generating a new PEM file with all the certificates from the chain the web socket connection works fine. Even with: tune.ssl.default-dh-param 2048 Thank you very, very much for your help! Best regards, Heiko
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
On other thing that cames to my mind is intermediate certificates. Are intermediate certificates correctly installed on haproxy? Here we got the problem and now it is clear to me — all the behavior we saw in the tcpdumps. The browsers have the intermediate cert installed and that´s why it looked good there. Even then, a browser would have emitted an error message that would've lead to this discovery much sooner, but the Java client really hid the problem. After generating a new PEM file with all the certificates from the chain the web socket connection works fine. Even with: tune.ssl.default-dh-param 2048 Thank you very, very much for your help! You're welcome! Lukas
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko, Also, please try the bind keywords no-tlsv12, no-tlsv11 and ciphers TLS_RSA_WITH_RC4_128_SHA. If this makes it work, please apply the attached debug patch and just run it with force-tlsv10, I would like to know if that call fails. I added the parameters except TLS_RSA_WITH_RC4_128_SHA as it is not available in openssl. This one seems to be the equivalent here: RC4-SHA Well, with this configuration there is now a real TLS handshake problem. Please just try with no-tlsv12, without the cipher configuration and redo those test. I suspect there is no real SSL/TLS handshake issue here, but that the issue is on the application layer (I misread the first capture you sent me: the actual handshake seems ok, its the client that is sending the FIN+ACK). Lukas
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas, Thanks for your reply. You can find my additional information in your text below. Am 17.10.2014 um 01:32 schrieb Lukas Tribus luky...@hotmail.com: Gonna need to see your configuration to be able to help you, especially ssl and http related parts. ##- # Global settings #- global log 127.0.0.1 local2 debug alert chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4096 userhaproxy group haproxy daemon nbproc 1 #ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA #- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #- defaults modehttp log global option httplog option logasap option redispatch balance roundrobin timeout connect 1 # default 10 second time out if a backend is not found timeout client 8640 timeout server 30 timeout queue 5000 maxconn 6 retries 3 default_backend deny_backend #- # test configuration #- frontend test1 bind *:443 ssl crt /etc/pki/tls/certs/domain.com-haproxy.pem #ciphers HIGH:RC4-SHA:!ADH #ciphers AES:RC4:ALL:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!ECDH reqadd X-Forwarded-Proto:\ https acl nourlurl / acl baseurl url /java-app acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket path_beg /java-app/r3 acl is_download path_beg /java-app/ acl admin_urlpath_beg /admin acl is_admin src 192.168.1.0/24 redirect location http://www.domain.com/ if nourl redirect location https://test.domain.com/java-app/ if baseurl use_backend test1_socket_backend if is_websocket use_backend test1_website_backend if is_download ! is_websocket use_backend private_monitoring if is_admin admin_url #-Backends backend deny_backend option httpclose reqideny .* backend private_monitoring stats enable stats uri /admin?stats stats refresh 10s # Test1 backend test1_website_backend option httpclose server test1 internal hostname:8080 backend test1_socket_backend no option httpclose server test1 internal hostname:8080 #- — changing bind *:443 ssl crt /etc/pki/tls/certs/domain.com-haproxy.pem to bind *:81 (and removing the https line next to it) makes it run with stunnel. Out of the back of my mind I recall Java has problems with DHE cihpers when the dh size is more than 1024 bits. Could that be your case? I read about it anywhere and was aware of it (a known problem with Java 7). I tested with the default of 1024 bit but also tried with other values. There was noch change in behavior. Can you share a tcpdump capture of the failed handshake (don't forget -s0 otherwise packets will be truncated). 43 29.413080 CLIENT HAPROXYTCP 80 62199→443 [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=16 TSval=666771533 TSecr=0 SACK_PERM=1 44 29.413184 HAPROXYCLIENT TCP 76 443→62199 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1791798027 TSecr=666771533 WS=64 45 29.430494 CLIENT HAPROXYTCP 68 62199→443 [ACK] Seq=1 Ack=1 Win=132480 Len=0 TSval=666771557 TSecr=1791798027 46 29.435106 CLIENT HAPROXYSSLv2213Client Hello 47 29.440451 HAPROXYCLIENT TLSv1.2 1508 Server Hello 48 29.440473 HAPROXYCLIENT TLSv1.2 1134 Certificate 49 29.461553 CLIENT HAPROXYTCP 68 62199→443 [ACK]
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko, #- # test configuration #- frontend test1 bind *:443 ssl crt /etc/pki/tls/certs/domain.com-haproxy.pem #ciphers HIGH:RC4-SHA:!ADH #ciphers AES:RC4:ALL:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!ECDH reqadd X-Forwarded-Proto:\ https acl nourl url / acl baseurl url /java-app acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket path_beg /java-app/r3 acl is_download path_beg /java-app/ acl admin_url path_beg /admin acl is_admin src 192.168.1.0/24 redirect location http://www.domain.com/ if nourl redirect location https://test.domain.com/java-app/ if baseurl use_backend test1_socket_backend if is_websocket use_backend test1_website_backend if is_download ! is_websocket use_backend private_monitoring if is_admin admin_url #-Backends backend deny_backend option httpclose reqideny .* backend private_monitoring stats enable stats uri /admin?stats stats refresh 10s # Test1 backend test1_website_backend option httpclose server test1 internal hostname:8080 backend test1_socket_backend no option httpclose Better switch to option http-tunnel here. Out of the back of my mind I recall Java has problems with DHE cihpers when the dh size is more than 1024 bits. Could that be your case? I read about it anywhere and was aware of it (a known problem with Java 7). I tested with the default of 1024 bit but also tried with other values. There was no change in behavior. How exactly did you modify this value? Do you have those dh-params in your pem file? 46 29.435106 CLIENT HAPROXY SSLv2 213 Client Hello 47 29.440451 HAPROXY CLIENT TLSv1.2 1508 Server Hello 48 29.440473 HAPROXY CLIENT TLSv1.2 1134 Certificate Can you try force-tlsv10 on the bind line, to see if that changes anything? If that doesn't help, can you send me the full tcpdump capture file offlist (tcpdump -ns0 -w capture.cap tcp port 443 or something like that)? Do you have any way to enable the java debug console and see what it says? Regards, Lukas
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Used the bind parameter before which did / does not help and created a tcpdump with the mentioned settings (DH = 1024 Bit and force tls) with your requested parameters. Something doesn't add up. The handshake you sent me is still negotiating TLSv1.2 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027), but: - Java 7 doesn't support TLSv1.2 - Java 7 doesn't support this cipher - HAProxy with force-tlsv10 is supposed to force TLSv1.0 Are you sure this is a Java 7 JRE connecting? Are you sure forcetls is configured and HAProxy has been properly restarted? Please also provide the output of haproxy -vv. Thanks, Lukas
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas, Am 17.10.2014 um 18:11 schrieb Lukas Tribus luky...@hotmail.com: Used the bind parameter before which did / does not help and created a tcpdump with the mentioned settings (DH = 1024 Bit and force tls) with your requested parameters. Something doesn't add up. The handshake you sent me is still negotiating TLSv1.2 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027), but: - Java 7 doesn't support TLSv1.2 - Java 7 doesn't support this cipher - HAProxy with force-tlsv10 is supposed to force TLSv1.0 Are you sure this is a Java 7 JRE connecting? Are you sure forcetls is configured and HAProxy has been properly restarted? The client used for the test has Oracle Java 1.7.0_72 (JDK) installed on Mac OS X 10.9. The bind is done this way: — bind *:443 ssl crt /etc/pki/tls/certs/domain-haproxy.pem force-tlsv10 — Currently, I start haproxy manually with this command (in the same shell I edit the config file, thus I have to stop haproxy with CTRL-C for changes): — haproxy -d -f /etc/haproxy/haproxy.cfg — Please also provide the output of haproxy -vv. HA-Proxy version 1.5.5 2014/10/07 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Thanks again and best regards, Heiko --- Heiko Burghardt IT Infrastructure -- .. Riege Software International GmbH Phone: +49 2159 91480 Mollsfeld 10 Fax: +49 2159 914811 40670 MeerbuschWeb: www.riege.com GermanyE-Mail: burgha...@riege.com -- -- Commercial Register: Managing Directors: Amtsgericht Neuss HRB-NR 4207 Christian Riege VAT Reg No.: DE120585842 Gabriele Riege Johannes Riege TobiasRiege .. YOU CARE FOR FREIGHT, WE CARE FOR YOU
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko, Currently, I start haproxy manually with this command (in the same shell I edit the config file, thus I have to stop haproxy with CTRL-C for changes): — haproxy -d -f /etc/haproxy/haproxy.cfg — I see. Can you run it through strace -tt, Not that I expect to see why the TLS handshake fails, just to confirm that its indeed haproxy that accepts the connection (just prepend your command above with strace -tt). Attach the strace output to a txt file to the mail, as it will be long. Also, please try the bind keywords no-tlsv12, no-tlsv11 and ciphers TLS_RSA_WITH_RC4_128_SHA. If this makes it work, please apply the attached debug patch and just run it with force-tlsv10, I would like to know if that call fails. Anyway, I still think we don't see the whole picture. You don't have any SSL/TLS intercepting middleboxes between your client and your server, correct? Regards, Lukas debugforcetlsv10.patch Description: Binary data
Re: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hello Lukas,
Thanks for your update.
Currently, I start haproxy manually with this command (in the same shell I
edit the config file, thus I have to stop haproxy with CTRL-C for changes):
—
haproxy -d -f /etc/haproxy/haproxy.cfg
—
I see. Can you run it through strace -tt, Not that I expect to see why the TLS
handshake fails, just to confirm that its indeed haproxy that accepts the
connection (just prepend your command above with strace -tt). Attach the
strace output to a txt file to the mail, as it will be long.
I will send the entire strace output separately to you as it very long but the
short form for the mailing list:
—
00:15:55.786195 gettimeofday({1413584155, 786244}, NULL) = 0
00:15:55.786280 accept4(4, {sa_family=AF_INET, sin_port=htons(57248),
sin_addr=inet_addr(„MY IPV4 ADDRESS ON INTERNET)}, [16], SOCK_NONBLOCK) = 6
00:15:55.786548 accept4(4, 0x7fff2fbe9880, [128], SOCK_NONBLOCK) = -1 EAGAIN
(Resource temporarily unavailable)
00:15:55.786629 gettimeofday({1413584155, 786643}, NULL) = 0
00:15:55.786772 read(6, 0x119b343, 5) = -1 EAGAIN (Resource temporarily
unavailable)
00:15:55.786890 epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|0x2000, {u32=6,
u64=6}}) = 0
00:15:55.786943 gettimeofday({1413584155, 786956}, NULL) = 0
00:15:55.786983 epoll_wait(3, {{EPOLLIN, {u32=6, u64=6}}}, 200, 1000) = 1
00:15:55.815985 gettimeofday({1413584155, 816008}, NULL) = 0
00:15:55.816038 gettimeofday({1413584155, 816051}, NULL) = 0
00:15:55.816092 read(6, \200\217\1\3\3, 5) = 5
00:15:55.816154 recvfrom(6, NULL, 2147483647,
MSG_TRUNC|MSG_DONTWAIT|MSG_NOSIGNAL, NULL, NULL) = 140
00:15:55.816208 recvfrom(6, 0, 2147483647, 16480, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
00:15:55.816312 sendto(5, 150Oct 18 00:15:55 haproxy[100..., 116,
MSG_DONTWAIT|MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(514),
sin_addr=inet_addr(127.0.0.1)}, 16) = 116
00:15:55.816565 close(6)= 0
—
This is the connect to haproxy where the web socket upgrade shall take place.
Also, please try the bind keywords no-tlsv12, no-tlsv11 and
ciphers TLS_RSA_WITH_RC4_128_SHA. If this makes it work, please apply
the attached debug patch and just run it with force-tlsv10, I would like
to know if that call fails.
I added the parameters except TLS_RSA_WITH_RC4_128_SHA as it is not available
in openssl. This one seems to be the equivalent here: RC4-SHA
Thus, my bind looks like this:
bind *:443 ssl crt /etc/pki/tls/certs/domain-haproxy.pem force-tlsv10 no-tlsv12
no-tlsv11 ciphers RC4-SHA
But again, the web socket upgrade does not take place. haproxy log still says:
Connection closed during SSL handshake.
Am I right that the patch is only useful for you on a positive result?
Anyway, I still think we don't see the whole picture. You don't have
any SSL/TLS intercepting middleboxes between your client and your
server, correct?
All my tests were established this way:
Java client - Internet router doing SNAT to Internet IP - INTERNET -
Company firewall doing DNAT to internal, private IP - haproxy server
I always re-checked the company firewall but this connection is not logged in
any way.
Best regards,
Heiko
RE: Switching Java client to Websocket with SSL // Connection closed during SSL handshake
Hi Heiko! Due to connection limit problems I´d like to remove stunnel from a configuration in front of haproxy. The original setup was: - stunnel was responsible for the SSL(https) connection - using localhost the web traffic was transferred to haproxy - haproxy divided traffic into web page requests and the Java software tunnel to an application server via websocket. I updated haproxy from version 1.4.2 to 1.5.5 on a Red Hat Enterprise Linux 6.5 host and the mentioned setup still worked fine. Using a test system I tried to add the SSL functionality directly to haproxy and removed stunnel from the setup. The web pages are still working with any crypto protocols and ciphers but the upgrade to websocket does not work anymore. I can see that the Java client sends initial packets to start the encryption but drops the connection with a FIN+ACK after haproxy sends a TLSv1.2 proposal. The haproxy log then tells: Connection closed during SSL handshake Additionally, I testet all the crypto protocol options in the Java control panel from SSLv3 up to TLSv1.2 — all with the same result. There is no additional crypto library implemented in the client software, so it depends completely on the Java settings. I used a very recent version of Java 7 for my tests. Does somebody have further ideas what I might have overseen? Gonna need to see your configuration to be able to help you, especially ssl and http related parts. Out of the back of my mind I recall Java has problems with DHE cihpers when the dh size is more than 1024 bits. Could that be your case? Can you share a tcpdump capture of the failed handshake (don't forget -s0 otherwise packets will be truncated). Regards, Lukas
