Hi Luca and Igor,
I know there is not a simple way. In this network trace I verified an IE11 /
Edge bug with preconnect sessions.
This is a known problem, also if not so documented.
As you can see Windows Client TCP Stack correctly send an ACK to a FIN request
from HAProxy, but IE
does not instruct TCP Stack to send its own FIN to HAProxy to close the TCP
connection gracefully, so IE at later time
erroneously try to use the TCP Connection, correctly HAProxy send a RST. In the
HTTP buffer of the client there is
what HAProxy sent before closing the connection (a 408 Timeout HTTP Status
message), so IE erroneously read this
message and wrong again, instead of closing HTTP session and retry with another
session, send the 408
Timeout to the Browser, so client see 408, thinking the server does not work
well…
[cid:image001.png@01D1BE4E.ED11D370]
[cid:image002.png@01D1BE4E.ED11D370]
In this case:
- 408 is not sent from backend server, so not all the traffic can be
collected on backend server to analyze the issue
- HAProxy send 408, but I don’t see HTTP flow, I see only HTTPS
(SSL/TLS) Flow as all is encrypted from client to haproxy and haproxy do SSL
termination
- In this particular case we can reproduce the problem on a test
machine, as we know and manage the machine, we can use the wireshark
unencypting capabilities, but I think this method is not so robust as involves
many changing parts Cipher/haproxy/…
I have to put togheter many info (haproxy log, tcpdump trace, ecc ) to have a
complete picture (ie: it would be simple if I can see and search 408 http
status code in the tcpdump trace instead of know it is here from browser and
form haproxy log, but not shown in wireshark as incapsulated in TLS
communication. Haproxy just do different not properly application level thing
for SSL termination, so I’m uncertain is haproxy or not to manage and provide
unencrypted traffic, all other solution as we are seeing are prone to
difficulties and instability of the trace process.
I like robust solution and if possible solution independent as soon as possible
from implicit (instead explicit) support by components, that is: haproxy work
using unencrypting with client session keys, but I know on some version this
mechanism does not work, so this introduce instability.
But I’ll try FPS with client session keys solution also if I think this is
complex as we have to capture all keys for all TLS sessions to see the complete
traffic we take as many SSL handshake take place in that trace.
I think that at the moment the best solution is to temporarily disable PFS
ciphers suite, I think this is a simple reconfiguration of haproxy.cfg and
haproxy daemon restart, for the test period, not so critical for us.
I need for this some hints as I’m not so informed about cipher suites, in
particular I didn’t found a clear specification how to identify cipher show /
used by haproxy and how identify what support or not PFS.
ie: if I check from a SSL tester site I see this protocols and chiper suite for
haproxy SSL termination:
Protocols
TLS 1.2 Yes
TLS 1.1 Yes
TLS 1.0 Yes
SSL 3 No
SSL 2 No
Cipher Suites
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) ECDH secp256r1 (eq.
3072 bits RSA) FS 256
TLS_RSA_WITH_RC4_128_SHA (0x5)
INSECURE 128
TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011) ECDH secp256r1 (eq.
3072 bits RSA) FS INSECURE 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) ECDH secp256r1 (eq.
3072 bits RSA) FS 256
TLS_RSA_WITH_AES_256_CBC_SHA256 (0x3d)
256
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)
256
TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (0x84)
256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) ECDH secp256r1 (eq.
3072 bits RSA) FS 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH secp256r1 (eq.
3072 bits RSA) FS 128
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012) ECDH secp256r1 (eq.
3072 bits RSA) FS 112
TLS_RSA_WITH_AES_128_CBC_SHA256 (0x3c)
128
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)
128
TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (0x41)
128
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa)
112
In haproxy.cfg I used these cipher I found recommended:
ciphers ECDHE-RSA-AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM
but I do not understand well the mapping with those recognized by test (FS = I
think are Forward Secrecy) an these configured and how to find if they support
or not PFS. Your help will be appreciated, so I can change haproxy
configuration to quickly disable/enable cipher so I can use certificates
private keys that means a simple trace analysis, I need to have more control on
chiper management.
Roberto
From: Igor Cicimov [mailto:ig...@encompasscorporation.com]
Sent: sabato 4 giugno 2016 02.14
To: Lukas Tribus <lu...@gmx.net>
Cc: mlist <ml...@apsystems.it>; haproxy@formilux.org
Subject: Re: tcpdump and Haproxy SSL Offloading
Hi Lukas,
On Sat, Jun 4, 2016 at 3:03 AM, Lukas Tribus
<lu...@gmx.net<mailto:lu...@gmx.net>> wrote:
Hello,
you can dump the symmetric keys from the browser and import them in wireshark
to decrypt PFS protected TLS sessions [1]
Yes in case you want to troubleshoot something generic this is a good approach
but if you want to troubleshoot sessions not initiated by your self, ie
particular client connection, this is practically impossible.
or downgrade your ciphers settings to non-PF ciphers. Properly decrypting the
TLS session is the only way to really make sure you see what happens, even if
there is a TLS related bug in the client or server (haproxy).
Some other idea's are:
- if your backend traffic is unencrypted, you may want to capture the traffic
there.
Not practical though if you have tens of backend servers. Much better if you
have to troubleshoot on 2 instead of dozen of servers. At least that's how I
understand the question related to tcpdump usage on the haproxy servers them
self.
- if haproxy is rejecting the request, check "show errors" on the admin socket.
As you said, the best solution is to not depend on haproxy specific features,
as you don't want to modify existing infrastructure in a troubleshooting
case.Maybe something
Outside haproxy maybe something like mitmproxy or sslstrip might help. Not sure
though have never used them myself.
Another proxy layer means that you decrypt TLS on the front-end proxy, while
you sniff the plaintext traffic between the front-end and the second tier
proxy. You can probably do this with a single haproxy instance recirculating
the traffic through a unix socket and capture the traffic on it, but it would
require some trial and error and definitely some testing.
This will probably be faster but can't use tcpdump in that case.
I believe the SSLKEYLOGFILE approach [1] to be the most efficient and simplest
approach.
cheers,
lukas
[1]
https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-wireshark-the-easy-way/
--
Il messaggio e' stato analizzato alla ricerca di virus o
contenuti pericolosi da MailScanner<http://www.mailscanner.info/>, ed e'
risultato non infetto.
