Randomly added byte in GET request line with HAProxy 1.5 + OpenSSL
Hello all, we see some strange errors in our logs after having introduced HAProxy 1.5 snapshot 20130611 before our nginx. It seems like HAProxy sometimes (seldom) inserts a rather random byte as the second byte of a GET request line on SSL requests. Some (anonymized) log lines follow: 1.1.1.1:30893 [13/Jun/2013:08:41:50.443] front~ master/gemini 369/0/0/500/869 500 817 - - 3/2/0/0/0 0/0 GNET /login HTTP/1.1 2.2.2.2:50771 [13/Jun/2013:16:03:17.488] front~ special/gemini 184/0/0/-1/184 502 4410 - - PH-- 0/0/0/0/0 0/0 G3ET /foo HTTP/1.1 3.3.3.3:37310 [13/Jun/2013:16:13:52.495] front~ master/gemini 911/0/0/-1/911 502 4410 - - PH-- 0/0/0/0/0 0/0 GqET / HTTP/1.1 and more of that. Inserted characters that I have seen include A J H I U Q N 3 % ~ + ! $ . ' o z q They are always inserted before the E in GET. We have only seen this behavior on GET requests. All other HTTP verbs are completely unaffected. I can reproduce this error every time with the following conditions: * HAProxy is compiled with a self-compiled openssl 1.0.1d * The client is an IE on Windows 7 Other browsers don't show this issue. Also, when I compile HAProxy against the default OpenSSL 0.9.8o in Debian Squeeze, it works fine too. I can reproduce the issue with even the most simple (ssl-) configs, on the current snapshot, dev18 and dev17. I'm a bit worried that this might be the symptom of a larger issue. But it might just be that I'm not competent enough to compile my own OpenSSL. I would appreciate, if someone could give me some input here. # uname -a Linux gemini 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux # cat /etc/debian_version 6.0.7 I compiled openssl 1.0.1d with ./config no-idea no-mdc2 no-rc5 zlib enable-tlsext no-ssl2 --openssldir=/opt/haproxy/openssl make make test make install Haproxy is compiled as follows (using https://github.com/meineerde-cookbooks/haproxy/blob/master/recipes/source.rb): # haproxy -vv HA-Proxy version 1.5-dev18 2013/04/03 Copyright 2000-2013 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3.4 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1d 5 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1d 5 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.02 2010-03-19 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. The full make line is this: make TARGET=linux2628 USE_PCRE=1 CPU=generic ARCH=x86_64 PREFIX=/opt/haproxy/haproxy USE_OPENSSL=1 USE_ZLIB=1 PCREDIR=/opt/haproxy/openssl/lib -L/usr DEFINE= SILENT_DEFINE=-I/opt/haproxy/openssl/include ADDLIB=-lz -ldl ADDINC= Any hints or help would be greatly appreciated. Regards, Holger
RE: Randomly added byte in GET request line with HAProxy 1.5 + OpenSSL
Hi Holger, sounds like a tricky issue ... a few questions here: - has the Windows 7 box all the latest patches from MS? - any reason not to use openssl1.0.1e? - any security software (suites, software firewalls, anti-virus) which may intercept the SSL/TLS session (basically: do you see your real certificate in the browser or do you see a certificate of a security product)? - could you reproduce this with a self-signed certificate you *don't* use in production (so that the private key can be disclosed for troubleshooting), tcpdump the ssl session and provide the capture, including the private server certificate? Thanks, Lukas
Re: Randomly added byte in GET request line with HAProxy 1.5 + OpenSSL
Hi Lukas, Lukas Tribus wrote: sounds like a tricky issue ... indeed :) - has the Windows 7 box all the latest patches from MS? Yes. - any reason not to use openssl1.0.1e? I couldn't get it to compile, or in fact, I could compile it, but it would break at the `make test` step and I hadn't yet found time to get to the bottom of this. - any security software (suites, software firewalls, anti-virus) which may intercept the SSL/TLS session (basically: do you see your real certificate in the browser or do you see a certificate of a security product)? There is a simple iptables on the box. By policy, we don't deploy any magic security snake oil, so no, nothing of that kind between the client and HAProxy. The browser is talking directly to HAProxy. - could you reproduce this with a self-signed certificate you *don't* use in production (so that the private key can be disclosed for troubleshooting), tcpdump the ssl session and provide the capture, including the private server certificate? I'll have to reconstruct this on a local VM to anonymize the data a bit. I'll get back to you as soon as possible. Thanks for your support. --Holger
RE: Randomly added byte in GET request line with HAProxy 1.5 + OpenSSL
Hi Holger, There is a simple iptables on the box. By policy, we don't deploy any magic security snake oil I didn't mean the server or intermediate devices, I meant directly on the windows 7 client: *windows* software security solutions, which intercept SSL/TLS traffic on your local windows machine before they even touch the wire; there are a lot of those products intercepting outgoing https traffic by installing their own certificate in your IE/schannel certificate store. When you surf on your HTTPS site from IE 10 you can view the security report by clicking on the lock and with show certificate you see all the details about that certificate. When those things like sha1 fingerprint, serial, valid up/until dates and other things match with the certificate you installed on your server, then fine. If on the other hand you see Kaspersky, Avast or other names which have nothing to do with your certificate or CA, then probably one of those security products (or a MITM ...) is intercepting your HTTPS traffic, which *may* be causing issues if that software is buggy. I couldn't get it to compile, or in fact, I could compile it, but it would break at the `make test` step and I hadn't yet found time to get to the bottom of this. Do you run make depend also? openssl ask me that after config when I use your parameters: Since you've disabled or enabled at least one algorithm, you need to do the following before building: make depend I'll have to reconstruct this on a local VM to anonymize the data a bit. I'll get back to you as soon as possible. It is kind of a last resort option, perhaps someone has a better idea ... Regards, Lukas
