Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
On 2017-06-01 12:23, Michael Wojcik wrote: On the other hand, this doesn't really answer Florin's question of why the server sees so many clients falling back. If the load is bursty, it might be listen-queue dumping. I don't know if Nginx lets you configure the listen queue depth, but at some point the stack will start dropping connections (either silently, in the BSD or "correct" manner; or with a RST, in the Winsock or "patently wrong" manner) if the sustained load is too high. But unless this is a pretty busy server or has a really high variance in its load distribution, it's probably an intermediary node that's to blame. Aggregate traffic has a smooth profile, with daily increase towards the middle of the day, and decrease towards midnight. Client sessions are typically short (way under 1 minute, perhaps most of them under 15 seconds). I don't see any errors related to what you're saying. Or the clients have a really short timeout, or the connection's being aborted by the user and the client is incorrectly falling back when that happens. Though then I'd assume Florin would see that in the Nginx logs, assuming it logs ECONNRESET. Well, I'm digging through the logs. The one thing I see often is: client X.Y.Z.K closed keepalive connection Not much other than that, and these are really "severity:info" events, I see them even with TLS termination at the ALB. I see some amount of... peer closed connection in SSL handshake while SSL handshaking ...also a "severity:info" event, at a rate about 4x less than the “inappropriate fallback” stuff. (shrug) Seems normal. As a more informal argument: We're using whatever Amazon deemed appropriate for their TLS policy for load balancers, in terms of protocol versions and ciphers. Surely if there was a problem with versions or ciphers, we'd hear about it, or they would change it quickly, just based on the amount of traffic they receive every day. That was the main reason why I've found hard to accept that this rate of TLS errors is somehow normal; but now I think I start to understand this aspect of the protocol thanks to the excellent explanations I've seen early in this discussion. -- Florin Andrei http://florin.myip.org/ -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf > Of Salz, Rich via openssl-users > Sent: Thursday, June 01, 2017 14:44 > To: openssl-users@openssl.org > Subject: Re: [openssl-users] SSL error “inappropriate fallback” and > TLS_FALLBACK_SCSV > > > Would clients actually attempt to send TLS_FALLBACK_SCSV even if the > > previous connection attempt failed for reasons other than TLS? If, say, the > > initial connection attempt failed at the TCP level? That sounds a little > > strange > > to me. > > Yes they do. And they should, *if* they have actually fallen back to an older protocol version, because an attacker with the appropriate affordance (e.g. a compromised router along the path) could easily trigger a TCP failure by sending a RST. Any time the client falls back, it should include TLS_FALLBACK_SCSV in its cipher-suite list. The question is whether a client that supports fallback (and there's much debate over whether clients should support fallback, particularly in the default configuration) ought to try falling back in response to a TCP failure. On the one hand, it's entirely possible that a broken old server would reject an unrecognized protocol version by simply aborting the connection. On the other, there are a lot of other reasons for a RST, and falling back as the first resort seems rather hasty. Personally, I'd prefer clients not try to fall back at all, except perhaps clients that have a reasonable need to support broken servers, and even then only with some non-default configuration. Browser manufacturers will argue that they have to support fallback in order to support broken web servers, but I think that's dubious. > There are many badly written clients out there. Or poor libraries. Very true. On the other hand, this doesn't really answer Florin's question of why the server sees so many clients falling back. If the load is bursty, it might be listen-queue dumping. I don't know if Nginx lets you configure the listen queue depth, but at some point the stack will start dropping connections (either silently, in the BSD or "correct" manner; or with a RST, in the Winsock or "patently wrong" manner) if the sustained load is too high. But unless this is a pretty busy server or has a really high variance in its load distribution, it's probably an intermediary node that's to blame. Or the clients have a really short timeout, or the connection's being aborted by the user and the client is incorrectly falling back when that happens. Though then I'd assume Florin would see that in the Nginx logs, assuming it logs ECONNRESET. Really there are too many possibilities to make speculation worthwhile. But now that I've written all this I suppose I'll send it. Michael Wojcik Distinguished Engineer, Micro Focus -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
> What I find surprising is the rate of these errors. For every 100 legitimate > HTTP requests that make it to Nginx, I get 2.5 “inappropriate fallback” SSL > errors. That's a lot of noise. > > I guess I'll have to adjust my expectations. That's not out of line with other measurements I've been told. > Related question: assuming the lists of TLS protocol versions and ciphers I've > enabled in Nginx are indeed exactly the same as the default TLS policy in an > AWS ALB, the errors I see now logged by Nginx should be, more or less, the > same population of errors I saw reflected in the ALB metrics before, right? Not necessarily. The network connectivity could be a very large influence. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
On 2017-06-01 11:43, Salz, Rich via openssl-users wrote: Would clients actually attempt to send TLS_FALLBACK_SCSV even if the previous connection attempt failed for reasons other than TLS? If, say, the initial connection attempt failed at the TCP level? That sounds a little strange to me. Yes they do. There are many badly written clients out there. Or poor libraries. What I find surprising is the rate of these errors. For every 100 legitimate HTTP requests that make it to Nginx, I get 2.5 “inappropriate fallback” SSL errors. That's a lot of noise. I guess I'll have to adjust my expectations. Related question: assuming the lists of TLS protocol versions and ciphers I've enabled in Nginx are indeed exactly the same as the default TLS policy in an AWS ALB, the errors I see now logged by Nginx should be, more or less, the same population of errors I saw reflected in the ALB metrics before, right? The whole point of this exercise is to temporarily work around the lack of a TLS error log in an ALB. The error rate does seem quite similar between ALB and Nginx. I'm just wondering if the ALB is doing something that my standard Ubuntu openssl libraries are not. -- Florin Andrei http://florin.myip.org/ -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
> Would clients actually attempt to send TLS_FALLBACK_SCSV even if the > previous connection attempt failed for reasons other than TLS? If, say, the > initial connection attempt failed at the TCP level? That sounds a little > strange > to me. Yes they do. There are many badly written clients out there. Or poor libraries. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
On 2017-06-01 02:13, Matt Caswell wrote: The presence of this error doesn't actually mean that you are under attack. It just means that the client made an earlier connection attempt with a higher version number and it failed. There could be many reasons for the failure. For example, plausibly, if you have a lot of mobile clients then you could imagine that a network glitch could cause an earlier attempt to fail. It's interesting how I see a constant stream of “inappropriate fallback” errors in the logs, but this is pretty much the only error from a TLS perspective. Sure, there's the occasional certificate failure, like once every few minutes or so, and then, rarely, there's some ancient app trying SSLv3 (which is not enabled). But looking at the Nginx error.log the “inappropriate fallback” is basically the only error I get a perpetual flow of. If the TLS_FALLBACK_SCSV attempt is caused by a previously failed connection, that must have been something different from a TLS error, because “inappropriate fallback” is probably over 99% of the lines in error.log - it's the only thing I see as logs are scrolling up in my viewer. Would clients actually attempt to send TLS_FALLBACK_SCSV even if the previous connection attempt failed for reasons other than TLS? If, say, the initial connection attempt failed at the TCP level? That sounds a little strange to me. Again, our clients are a mix of the average mobile devices in general use these days. -- Florin Andrei http://florin.myip.org/ -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
On 01/06/17 02:58, Florin Andrei wrote: > It's a little puzzling because the exchange of crypto messages uses TLS > 1.0 which the server definitely supports, and the client should be very > likely to support too. > > I've seen discussions online saying that the presence of the > TLS_FALLBACK_SCSV cipher suite is an indication that this failed > connection is related to anti-POODLE security measures, and the > communication is likely to be retried again. Is that correct? "Normal" TLS version negotiation works like this (ignoring TLSv1.3 which isn't relevant for this discussion): the client sends the maximum TLS version that it supports in its ClientHello message (this should not be confused with the version number in the record header). The server responds with a ServerHello containing the maximum version that it supports and which is less than or equal to the client's maximum version number. For example, if the client supports TLSv1.2 but the server only supports TLSv1.0, then the ClientHello version will be TLSv1.2 and the server will respond with TLSv1.0. On the other hand if the client only supports TLSv1.0 but the server supports TLSv1.2, then the ClientHello version will be TLSv1.0 and the ServerHello version will also be TLSv1.0. This is all fine but there are some servers out there which are buggy and abort the connection if the ClientHello version is bigger than the maximum version that the server supports. To work around this some clients will do "fallback". So, they start with a ClientHello version of TLSv1.2. If the connection aborts then they might send another one with a version of TLSv1.1. If that aborts then TLSv1.0. The problem with that approach is that it can be exploited by an attacker. An attacker may be able to exploit some weakness in an earlier TLS version which is fixed in a later version. The attacker may then be able to modify the traffic to ensure that a TLSv1.2 ClientHello fails in order to force the client to fallback to the earlier exploitable version. The TLS_FALLBACK_SCSV ciphersuite is used as a counter measure to that type of attack. A client will include it if it is sending a ClientHello with a max version that is lower than the maximum that the client actually supports (because an earlier attempt with the higher version number failed). If a server receives a TLS_FALLBACK_SCSV ciphersuite in a ClientHello and it supports a higher version than the one in the ClientHello then there should have been no reason for the client to fallback and you get the "inappropriate fallback" error. The presence of this error doesn't actually mean that you are under attack. It just means that the client made an earlier connection attempt with a higher version number and it failed. There could be many reasons for the failure. For example, plausibly, if you have a lot of mobile clients then you could imagine that a network glitch could cause an earlier attempt to fail. Matt -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] SSL error “inappropriate fallback” and TLS_FALLBACK_SCSV
A bit of context: I have this endpoint behind an AWS ALB. I do SSL termination at the ALB. To my surprise, when looking at the client_tlsnegotiation_error_count metric for the ALB, I've noticed a substantial amount of failed connection attempts due to TLS negotiation errors - perhaps around 5% of total traffic but this estimate could be wrong by a substantial margin. The clients are mostly an average cross-section of current mobile devices in use these days in the US. The ALB is configured with the default TLS policy that Amazon provides: | ssl-enum-ciphers: | TLSv1.0: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: server | TLSv1.1: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: server | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: server |_ least strength: A Unfortunately, ALBs do not provide error logs, so I could not directly identify why some clients are failing the SSL negotiation. Instead, I've pushed SSL termination deeper into the stack, to the Nginx frontend, and replaced the ALB with a plain TCP-based ELB. Now connections to port 443 are forwarded directly to Nginx for SSL negotiation. I've configured Nginx with the exact same protocol versions and ciphers like the ALB: ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA'; ssl_prefer_server_ciphers on; I've verified with nmap and I get the same ssl-enum-ciphers list from Nginx. Now in the Nginx error log I get lots of lines like this: SSL_do_handshake() failed (SSL: error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback) while SSL handshaking Still not very informative, so I've run tcpdump on port 443 on the Nginx instances. As expected, there's some amount of SSL errors like this: Secure Sockets Layer TLSv1 Record Layer: Alert (Level: Fatal, Description: Inappropriate Fallback) Content Type: Alert (21) Version: TLS 1.0 (0x0301) Length: 2 Alert Message Level: Fatal (2) Description: Inappropriate Fallback (86) In that same TCP stream there's this Client Hello packet: Secure Sockets Layer TLSv1 Record Layer: Handshake Protocol: Client Hello Content Type: Handshake (22) Version: TLS 1.0 (0x0301) Length: 165 Handshake Protocol: Client Hello Handshake Type: Client Hello (1) Length: 161 Version: TLS 1.0 (0x0301) Random GMT Unix Time: Jun 7, 2050 12:50:05.0 PST Random Bytes: da03ff7045a5f76e78edf61c097c75e4e141df6649ef1861... Session ID Length: 0 Cipher Suites Length: 28 Cipher Suites (14 suites) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007) Cipher Suite:
