Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac
Hi All, We are having a problem with a Tomcat client getting bad_record_mac exceptions when connecting to a server. Other applications are able to connect to that service so this seems specific to our client. I have included a description of the problem, analysis steps taken/not yet taken, environment information, and client side logs below. Any insights or suggestions that you may have will be very welcome indeed. Thanks, Diarmuid Description of the problem == On a Windows 2008 R2 64-bit Enterprise server running Java 7u67 and Tomcat 7.0.55, outbound SSL connections using TLSv1 and cipher suite TLS_RSA_WITH_AES_128_CBC_SHA are failing on the client side with SEND TLSv1 ALERT: fatal, description = bad_record_mac and javax.net.ssl.SSLException: Invalid Padding length errors. This uses one-way SSL and there is no Mutual Authentication involved. The SSL handshake appears to successfully establish a shared secret but fails when it tries to change the new established cipher suite. Analysis Steps == 1) On a non-production server with the same JRE/Tomcat/Windows setup connecting to a non-production version of the same service. This works ok. 2) On the same non-production server, remove the truststore/keystore settings from the Tomcat Java Options and connecting to a non-production version of the same service. Surprisingly, this works ok even though the endpoint uses a self-signed certificate. This connection also uses TLS_RSA_WITH_AES_128_CBC_SHA. 3) On the affected production server, run a standalone Java program that just opens an SSL input stream to the production service. This works ok. 3) Had a working session with the application team on the server side. What they see is: a) On our affected client, when I query their service from Internet Explorer I can successfully retrieve the WSDL and their web server records a GET request b) On our affected client, when I try to invoke a POST request from the Tomcat client application, they don't see any traffic on their web server and our client fails with the bad_record_mac/Invalid Padding Length error 4) Googled for Invalid Padding length and cipher TLS_RSA_WITH_AES_128_CBC_SHA. There are numerous references to this. The most solid is a known Java 7u51 issue recorded here https://bugs.openjdk.java.net/browse/JDK-8054316. However this and the majority of other TLS Padding links refer to javax.net.ssl.SSLHandshakeException or javax.net.ssl.SSLException: Received fatal alert: handshake_failure, whereas our situation the handshake is completing and on the client we get a javax.net.ssl.SSLException: Invalid Padding length. Aside from that, this article recommends using a different cipher suite, but our tests on a different server in step (2) above worked with TLS_RSA_WITH_AES_128_CBC_SHA. 5) Read Oracle documentation on debugging SSL/TLS connections here http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html. What I see from this is: a) the server is changing the cipher suite rather than the client b) in our client trace I don't see Found trusted certificate, but based on removing the truststore on a different server in step (2) above my conclusion is that this shouldn't matter, or if it did matter then the connection would fail with an error like No trusted cert found 6) Queried SEND TLSv1 ALERT: fatal, description = bad_record_mac and handling exception: javax.net.ssl.SSLException: Invalid Padding length on Stackoverflow but didn't learn much. 7) Confirmed that other applications can connect to the service with no issues 8) Check for differences in Windows patches between the affected production server and working non-production server. Using WMIC to compare produced the list of patches below which are on the working server but not on the failing server. I am working on addressing those but analysis of them indicates that they relate to Microsoft Office, .NET, and GDI components that are not being used by Tomcat or our client application. http://support.microsoft.com/?kbid=3046017 http://support.microsoft.com/?kbid=3060716 http://support.microsoft.com/?kbid=3071756 http://support.microsoft.com/?kbid=3072305 http://support.microsoft.com/?kbid=3075220 http://support.microsoft.com/?kbid=3076895 http://support.microsoft.com/?kbid=3077657 http://support.microsoft.com/?kbid=3078071 http://support.microsoft.com/?kbid=3078601 http://support.microsoft.com/?kbid=3079757 http://support.microsoft.com/?kbid=3087985 9) Firewall analysis between the client and server shows a mix of tcp-fin and tcp-rst commands from the server. However, given that no traffic appears on the server-side web server logs it isn't clear if this is normal communication or an O/S level issue. 10) The truststore and keystore are not logged in the Tomcat logs. Our tomcat Java Options are below but even removing the double quotes makes no difference. I don't know if this is an issue given
Re: Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Diarmuid, On 8/27/15 12:08 PM, dmccrthy wrote: We are having a problem with a Tomcat client getting bad_record_mac exceptions when connecting to a server. Other applications are able to connect to that service so this seems specific to our client. I have included a description of the problem, analysis steps taken/not yet taken, environment information, and client side logs below. Any insights or suggestions that you may have will be very welcome indeed. Thanks, Diarmuid Description of the problem == On a Windows 2008 R2 64-bit Enterprise server running Java 7u67 and Tomcat 7.0.55, outbound SSL connections using TLSv1 and cipher suite TLS_RSA_WITH_AES_128_CBC_SHA are failing on the client side with SEND TLSv1 ALERT: fatal, description = bad_record_mac and javax.net.ssl.SSLException: Invalid Padding length errors. This uses one-way SSL and there is no Mutual Authentication involved. The SSL handshake appears to successfully establish a shared secret but fails when it tries to change the new established cipher suite. So, the server tries to make an outgoing HTTPS request using that cipher suite? Can you connect to that (other) server directly without any errors? With that cipher suite? With what tool? Or, you can't reach Tomcat directly using some client. I think I'm confused by the term Tomcat client. Analysis Steps == 1) On a non-production server with the same JRE/Tomcat/Windows setup connecting to a non-production version of the same service. This works ok. 2) On the same non-production server, remove the truststore/keystore settings from the Tomcat Java Options and connecting to a non-production version of the same service. Surprisingly, this works ok even though the endpoint uses a self-signed certificate. This connection also uses TLS_RSA_WITH_AES_128_CBC_SHA. Not surprising to me: if you make an outgoing HTTPS connection from your application, Tomcat's truststore/keystore don't make any difference. If you removed your keystore configuration and a client could still connect to your /Tomcat/ server, then something is definitely broken, because the connector should never have successfully initialized at all. 3) On the affected production server, run a standalone Java program that just opens an SSL input stream to the production service. This works ok. 3) Had a working session with the application team on the server side. What they see is: a) On our affected client, when I query their service from Internet Explorer I can successfully retrieve the WSDL and their web server records a GET request b) On our affected client, when I try to invoke a POST request from the Tomcat client application, they don't see any traffic on their web server and our client fails with the bad_record_mac/Invalid Padding Length error The error happens before the TLS handshake completes, so you wouldn't see anything in the access logs. You might want to check other logs, because you'd want to know if you had clients failing TLS handshakes. 4) Googled for Invalid Padding length and cipher TLS_RSA_WITH_AES_128_CBC_SHA. There are numerous references to this. The most solid is a known Java 7u51 issue recorded here https://bugs.openjdk.java.net/browse/JDK-8054316. However this and the majority of other TLS Padding links refer to javax.net.ssl.SSLHandshakeException or javax.net.ssl.SSLException: Received fatal alert: handshake_failure, whereas our situation the handshake is completing and on the client we get a javax.net.ssl.SSLException: Invalid Padding length. No, your handshake has not completed. Or, possibly, the handshake has completed, but you aren't able to encrypt anything so it doesn't matter that you completed the handshake. Aside from that, this article recommends using a different cipher suite, but our tests on a different server in step (2) above worked with TLS_RSA_WITH_AES_128_CBC_SHA. Really you should be using ECDHE or EDHE ciphers these days. I'm not sure what IIS will actually support (if you are using IIS). You should allow the server to specify the cipher suite order and make clents use the highest-grade cipher suite they can support. 5) Read Oracle documentation on debugging SSL/TLS connections here http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/Re adDebug.html. What I see from this is: a) the server is changing the cipher suite rather than the client It's a negotiation, not a grudge match. Neither side changes anything. Did you mean the the server chooses? You're right: the server always chooses, but sometimes it chooses the first cipher listed by the client, or it can be told to do something else (use server preferred order, for instance). b) in our client trace I don't see Found trusted certificate, but based on removing the truststore on a different server in step (2) above my conclusion is
RE: AW: WebSocket asynchronous reads
Hi, -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Thursday, August 27, 2015 12:30 AM To: Tomcat Users List users@tomcat.apache.org Subject: Re: AW: WebSocket asynchronous reads On 26/08/2015 12:50, Steffen Heil (Mailinglisten) wrote: Hi Is there a way to tell tomcat to stop receiving data for a certain websocket? (Not to close it, but not to read from inputstream for some time.) Sorry, no. The best you can do is a no-op branch in the message handler. Which means I have to either buffer all incoming messages until I am ready to process them (might occupy lots of memory) or I have to drop them. Both is not really ideal. I don’t know, if the websocket specification already defines something, but are there any plans to implement such a feature? Nothing comes to mind. Sounds like what you really want is reactive streams. From my POV it would be rather easy not to unregister a certain connection in the read selector and reregister it later on. However such an API would need to exposed somehow. Yes, but such low level access doesn't seem right to me. I've also thought about that limitation in the Java Websocket API in the past. I could imagine to use some form of sleep/wait (e.g. Semaphore.aquire()) in the onMessage method to block the thread until it should continue reading. However that would of course contradict the aim of asynchronous/non-blocking I/O to not block threads while waiting for data. This is where I like the .Net WebSocket API [1] that is a bit lower-level: It provides ReceiveAsync() and SendAsync() methods that allow you to specify when you actually want to read from the websocket connection without blocking a thread (the ASP.Net API also allows the code in the HTTP pipeline to decide if and when a WebSocket/Upgrade request is accepted). (This could also be useful for forwarding Websocket connections from IIS to Tomcat which currently isn't supported by the ISAPI Redirector and the AJP protocol - I had been working on a C# SPDY redirector module for IIS but haven't yet found time to update it to use HTTP/2 and base it on the new modular ASP.Net 5 runtime [2] which will also be officially supported by MS for Linux and OS X). I don't have knowledge about NIO in Java, but for extending the current API I could imagine some startReceive() method (as a counterpart to RemoteEndpoint.Async#sendText()) using a sort of callback or just raising the onMessage event of the Endpoint as before when data has received (but I suppose there are better approaches). Regards, Konstantin Preißer [1] https://msdn.microsoft.com/de-de/library/system.net.websockets.websocket(v=vs.110).aspx [2] https://github.com/aspnet/home - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7.0.55/Jre 7u67: SEND TLSv1 ALERT: fatal, description = bad_record_mac
Hi Chris, Thanks for responding so quickly. My apologies, I should have been clearer on the topology. We have a Tomcat instance with a 3rd party web app deployed on it (the Tomcat client) running on Windows 2008 . This connects via HTTPS to a 3rd party service running behind IBM Http Server on Solaris 10. Thanks for clearing up a few things, you dispelled the truststore and handshake assumptions that have clouded some of our conclusions. To answer some of your questions: * our non-production server with the same versions of all client software connects with no errors to a non-production instance of the same 3rd party service using the same cipher suite. So the tool we're using is our 3rd party client Web app (the Tomcat client). It shows the same negotiation from TLSv1.2 down to TLSv1. My reading of your comments is that the truststore/keystore has no bearing here because it's an outgoing client connection and there are no inbound connections at play. * I don't know what the highest supported cipher suite is on either client or server hosts. I do know that TLSv1 is the highest supported by the 3rd party server. Unfortunately we have limited control over both client and server hosts because they are provided to us as managed services. However I do know that a standalone Java client successfully connects from our client host to the 3rd party server using TLS_RSA_WITH_AES_128_CBC_SHA, and this shows the same negotiation from TLSv1.2 to TLSv1. * other client applications running lower versions of Tomcat and Java are able to connect to the same 3rd party production service, but at present I don't know what cipher suite they're using * we're not using JCE Unlimited Security because it hasn't been an issue and frankly I didn't think of it when we were doing the installations :) Two additional pieces of analysis that came out of our investigations today are: 1) we are using NextGen Cisco devices. These are application and protocol aware. This means that the firewall might allow allow our standalone Java client to connect using HTTPS, but interfere with the HTTPS SOAP requests from our Tomcat/3rd party app Tomcat client. 2) there are some anecdotes of Java 7 clients failing to downgrade from TLSv1.2 to TLSv1. For example http://stackoverflow.com/questions/31579587/java-tls-1-2-downgrade-to-tls-1-0-sometimes-error. Some takeaway actions for me to explore are below. I'll welcome any other thoughts that you may have. 1) compare the firewall rules on our working non-production client with the rules governing the failing production client 2) use openssl from our client server and see what it shows 3) consider forcing the Tomcat/3rd party Tomcat client to use TLSv1 4) check other logs on the server side to see if other clients are failing TLS handshakes 5) investigate the supported ciphers on client and server 6) consider upgrading Tomcat (as a last resort) Regards, Diarmuid On 27 Aug 2015 20:36, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Diarmuid, On 8/27/15 12:08 PM, dmccrthy wrote: We are having a problem with a Tomcat client getting bad_record_mac exceptions when connecting to a server. Other applications are able to connect to that service so this seems specific to our client. I have included a description of the problem, analysis steps taken/not yet taken, environment information, and client side logs below. Any insights or suggestions that you may have will be very welcome indeed. Thanks, Diarmuid Description of the problem == On a Windows 2008 R2 64-bit Enterprise server running Java 7u67 and Tomcat 7.0.55, outbound SSL connections using TLSv1 and cipher suite TLS_RSA_WITH_AES_128_CBC_SHA are failing on the client side with SEND TLSv1 ALERT: fatal, description = bad_record_mac and javax.net.ssl.SSLException: Invalid Padding length errors. This uses one-way SSL and there is no Mutual Authentication involved. The SSL handshake appears to successfully establish a shared secret but fails when it tries to change the new established cipher suite. So, the server tries to make an outgoing HTTPS request using that cipher suite? Can you connect to that (other) server directly without any errors? With that cipher suite? With what tool? Or, you can't reach Tomcat directly using some client. I think I'm confused by the term Tomcat client. Analysis Steps == 1) On a non-production server with the same JRE/Tomcat/Windows setup connecting to a non-production version of the same service. This works ok. 2) On the same non-production server, remove the truststore/keystore settings from the Tomcat Java Options and connecting to a non-production version of the same service. Surprisingly, this works ok even though the endpoint uses a self-signed certificate. This connection also uses TLS_RSA_WITH_AES_128_CBC_SHA. Not
Re: Problems with tomcat-connectors-1.2.41 on 32bit linux
On Wed, Aug 26, 2015 at 8:13 PM, Rainer Jung rainer.j...@kippdata.de wrote: I added a configure check in http://svn.apache.org/viewvc?view=revisionrevision=1697985 and documented the problem in https://bz.apache.org/bugzilla/show_bug.cgi?id=58285 You might want to cross check. After successfully applying the patches to a fresh copy of 1.2.41 (not trunk) and figuring out how to rebuild the configure script I successfully built mod_jk with disabled gcc atomics [1]. So I guess the patch works as intended. [1] I saw this message in the configure output, so the changes were applied, I guess: on 32bit SLES11 SP3: checking whether the compiler provides atomic builtins... no on 64bit SLES11 SP3: checking whether the compiler provides atomic builtins... yes Again, thank you for your help and the patch. Regards, Falco
AW: Connection resets without timeout
Hi For everyone following this thread, here are some conclusions I came up with. First I need to correct myself: I saw FIN/ACK, not FIN/FIN+ACK (at least I could not reproduce that). So when the client program gets closed, the tcp stack of the client sends FIN and the server running tomcat replies with FIN+ACK. What I did not realize earlier is, that this does NOT close the tcp connection at all. It isn't even supposed to. Because TCP offers no way to reliably close a connection. Instead it simply tells the server that the client will not send any more payload data. So the connection is half-closed. But by definition the server may still send data to the client [2]. And the client has no way to prevent that [1]. So tomcat known that there will be no additional data, but it is absolutely valid to send more. As the connection cannot be closed there is no way to detect that the client application is gone. As soon as tomcat sends something, the client tcp stack recognized incoming packets that do not match an open connection and therefor sends RST. At that point of the connection is completely gone and tomcat does report that to the servlet (using onError, as writing failed). While I did a lot in the area of networking and even implemented my own NIO stack, I must confess I never realized that tcp connections cannot be closed by one side. Regards, Steffen [1] This is not strictly true, as the client might send RST instead of FIN, but that could lead to packet loss for data send before the RST. [2] If you think about HTTP/1.0 you see how that can be useful. In a POST/PUT request the client payload is send to the server. The server needs to detect the end of that payload. That can be done using Content-Length-Header or chunked encodinig (0 terminator) or by closing the connection (inbound only) after the data. However, even if the inbound connection is closed, the response should still be sent to the client, so in that case the client even expects some answer after he sent FIN... -Ursprüngliche Nachricht- Von: Steffen Heil (Mailinglisten) [mailto:li...@steffen-heil.de] Gesendet: Dienstag, 25. August 2015 18:13 An: Tomcat Users List users@tomcat.apache.org Betreff: Connection resets without timeout Hi When using async servlets with timeout set to 0, tomcat seems not to detect, if the client closes the connection. At least the servlet listener is not notified. I expected AsyncListener.onComplete or AsyncListener.onError or AsyncListener.onTimeout, but none of these are called. The only thing called is ReadListener.onAllDataRead but that’s expected as the request is a POST request with few inbound data but long running outbound data. Now we have no way to detect when the connection to the client is broken. On TCP level the connection was closed (FIN/FIN+ACK), so tomcat should know that the connection is gone. Is there any way to get notified about that or at least to check manually? Regards, Steffen BTW: I am posting new questions that are not direct follow-ups as top messages. If there is another policy for this list, please let me know. smime.p7s Description: S/MIME cryptographic signature
