One more clarification: on point [6] below I stated that Java is able to recover with a retry on a cached connection. Unfortunately that is only valid for higher level classes like HttpUrlConnection which makes 1 retry on IOException (and only on a GET and some other conditions). The lower-level sockets just throw and that’s it...
-----Original Message----- From: George Stanchev [mailto:gstanc...@serena.com] Sent: Friday, October 09, 2015 10:40 AM To: Tomcat Users List Subject: RE: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac Just for the record, https.protocols is a property used by the HttpsUrlConnection class. If your app is using a client that doesn't rely on the internal Oracle HTTP client, it's better to use " jdk.tls.client.protocols" which is read directly by the socket/SSL classes. Apache Http Client is one client that does use sockets directly instead of HttpsUrlConnection. Also, I want to mention that we have finally been able to isolate a really nasty problem with Java SSL implementation and Windows's SSPI based SChannel acting as a server. The issue we ran into is very similar as to what the OP reported, that’s why I asked him a while ago on a separate thread about keystores if the server is IIS-based. The issue that we ran into might sound esoteric but our product ran into it very often. We have a configuration where we lock our intracomponent communication via 2way SSL. We have Java Tomcat-based server components that talk to IIS based native components via web services over HTTP which we secure with 2way SSL. When we upgraded to JRE 8 we started getting those connection resets in Tomcat when we enabled the 2way SSL. Those were on the Java->IIS calls that work over regular SSL. On the java side, if you turn on the ssl debug on, you'd see connection reset IOException thrown, and on the Windows side, you'd see " The following fatal alert was generated: 20. The internal error state is 960." In the System event log. This error is " TLS1_ALERT_BAD_RECORD_MAC" from SSPI. A search in Oracle's bug database found a close resemblance to this issue: https://bugs.openjdk.java.net/browse/JDK-8068812. But the reporter couldn't reproduce it and also was against another Oracle component Java Web Download or something like this. So we started working with Oracle but the intermittent and hard-to-reproduce nature of this issue hampered the investigation. As the OP mentioned below, there are apocryphal reports for this all over the web but people cannot troubleshoot it very well. Anyway here are my findings. In order for the issue to arise all those conditions must be met: [1] The client must start with ClientHello TLSv1.2 [2] The server must respond with ServerHello TLSv1 (so the server should not have TLS1.2 enabled) [3] The cipher key exchange must NOT be ECDHE. I have not tried DHE as I could not make Windows take it. According to this doc [1] it should have several DHE key exchange ciphers available but I could not make it work. RSA key exchange exhibits the issue. I have been using “TLS_RSA_WITH_AES_256_CBC_SHA” to reproduce. [4] The initial connection must not be 2way-SSL [5] The server must upgrade the connection after encrypted payload is read. In case of IIS, it has 2way SSL setting on a Virtual Directory. The initial handshake is not 2way SSL, once the HTTP request is read, it determines that the requested resource (/gsoap) is 2way SSL-protected and upgrades the connection to 2way SSL. [6] The server must kill the connection abruptly when error happens. This is important as if the server terminates the connection gracefully and keeps it around Java is able to recover since it reuses the cached connection. IIS does exactly this. [7] The server must be Windows 7, Win2k8, Win2k8-R2, Win8. Might be other versions affected but bug is not evident on Win2k12 or Win2k12-R2 or Win10. Perhaps it was fixed, perhaps those versions support TLSv1.2 It might sound complicated but it is not, it is very common. All you have to do is to have a Virtual Directory in IIS that requires 2way SSL and is set to "want". RSA-based key exchanges are one of the most commonly used ciphers. The Windows platforms we tested with are still widely used in the enterprises while the adoption rate of win2k12+ is still lagging (from our experience with our customers). And you have equal "chance" of cipher support between Windows servers and Java 8 clients to land on RSA based cipher (which exhibits the issue) or ECDHE key exchange that is ok. Again, it might sound too complicated but it was blocker for us to ship our product. For us, to work around this issue is for the 2way SSL to turn off TLSv1.2 (via "-Djdk.tls.client.protocols=TLSv1" or RSA key exchange algorithm (via the java security properties) on affected systems which fixes it. I have a lot more technical details that I am not sharing here. I have sent them to Oracle so hopefully that gets resolved, but my guess is that is really an SSPI/SChannel issue in which case someone needs to work with Microsoft and I don't know if my company can do that. It boils down to - when all these conditions are met, is Java creating a bad MAC or SChannel fails to verify that MAC correctly on SSL handshake step X. Hopefully this can help someone else, as we have spent quite a bit of time on this issue. George TL;DR: There is a nasty bug in Java or Windows when calling SSPI-based app over 2way SSL that you can work around by turning off TLSv1.2 or RSA key exchanges. -----Original Message----- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, August 31, 2015 11:15 AM To: Tomcat Users List Subject: Re: [OT] Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Diarmuid, On 8/31/15 11:36 AM, dmccrthy wrote: > To cut a long story short openssl helped. Using openssl -connect > showed the error below. When I added the -tls1 flag the error went > away. > > 2104:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or > bad record mac:.\ssl\s3_pkt.c:532: > > Configuring Tomcat with the JVM parameter below resolved the issue. > > -Dhttps.protocols="TLSv1" Yep, you have to use a TLS handshake; the older SSL3 handshake won't wor k. Now... you *can* enable SSLv2Hello and then only support TLSv1+, but let's face it: SSL is dead and clients should get with the program. > This may limit our client Web application but I have very little > influence over the server-side application, so it'll have to do as a > solution for now. Your client should not try to use SSL handshake unless you know it's required. > So the issue seems to be a Java handshake error whereby it has a > problem downgrading from TLSv1.2 to TLSv1 during the handshake. I don't know enough about TLS to know if there is an appreciable difference between the handshake protocol between the two. I know that SSLv2 was the lowest common denominator for a while, and most servers would enable it *just for the handshake* but would refuse to actually negotiate an SSLv2 connection (SSLv2 was basically DOA). These days, nobody should be using SSL at all, but I might understand wanting to use SSL-compatible handshakes, so using "SSLv2Hello" but not supporting the SSLv3 protocol would be the way to do that. Again, we should just let SSL die. The more we push clients and servers to only negotiate TLS from here on out, the better off Internet security will be overall. > Why it works in our non-prod environment with the same versions of > everything is a mystery, but I can speculate at some difference in > underlying O/S settings, or perhaps a Java bug. Or more likely a configuration option you didn't detect when you checked everything out. - -chris -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJV5IudAAoJEBzwKT+lPKRY+xsP/RBgD3VErGPgLZIOu6GDdUn9 4e/yH5kVAdiuOcnn26En+OJKGONK5c9Hf9gW23utPw2ruG1KYwSkr9dJ4yFIiKxh 1wYhq4oTN3j8DwNb8C2Y522eyH9hRYo+W8S7er0pvklcGiImgx+vp/LYitM2dHeG 6MEPJ03+/5CLbrmYfLbcZvnPydqKYVJtIqozHGhrC5+fAbcFFsWBcwrfk0Menwji wE4w7o5yi38/MtXX1b3FtMnvyjfiKHdPoKJ4PJv8KA3hKjWzEVLiehFGEQGb9VS6 uqgMLB5YR3QrI0yYN53soi4apgqktcfwGimK+B5Ar/2uvsarLvZgIF8uYuEHXTYY bn+V7ngmRxuwX71VUJDP47MtwQsgxko4H1qa2jIh7Pp8kRHeHwKytJxOLPUepmKp K1rmii+OtVPMQusXGVJedohXMwOCi3kUf9exmadnN02EgAPmIh/rfy7uBARfYJdC CqjiQxmTCm5yKu1uB9PXFbZMiCFjq+Hw3g6/cVnn1X5nxlfkPBCAODk5UouRrO+w oIErBEAbLSb7twWo8S6B9pupb/8C4RNvdhPk9pCfPf2cC+Ouybg4Sz/S4KSQ1N8P 0zfXVoDL4AJtl3k1ylXbfPVRE5SY+ZMqT4fC3odI1frWGI3oSnnXNQsqxr4ehmpo PUDzJVZbKxLMruMEbnSh =OrTI -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
