-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Diarmuid,
On 8/28/15 3:25 PM, dmccrthy wrote: > Thanks again for replying. I agree it doesn't look like a Tomcat > issue and would be happy to close the issue. I've included > responses to your points below prefixed with [DMC] for my initials, > but think I need to focus on the Java side of things and perhaps > open an Oracle ticket. Feel free to continue the conversation here. I just marked it as "off topic" because it's not directly-related to Tomcat. This could have happened in any other application server, for instance. There's nothing wrong with having an off-topic discussion, here. I just wanted everyone to know that this wasn't directly-related to Tomcat . See below. > On 8/27/15 4:31 PM, dmccrthy wrote: >>>> Thanks for responding so quickly. My apologies, I should have >>>> been clearer on the topology. We have a Tomcat instance with >>>> a 3rd party web app deployed on it (the "Tomcat client") >>>> running on Windows 2008 . This connects via HTTPS to a 3rd >>>> party service running behind IBM Http Server on Solaris 10. > > Okay, so, just to confirm: > > Tomcat (app) -> HTTPS -> 3rd party service > > Configurations and results: > > 0. prod Tomcat -> prod 3rd-party service FAILS with > TLS_RSA_WITH_AES_128_CBC_SHA > >> [DMC] Correct Ok. > (do other ciphers work? or are you completely dead in the water?) > >> [DMC] No, this is the next step. Our production environment is >> completely locked down and we have to exhaust all other avenues >> before making speculative changes. I suppose what I meant was: do 100% of your connections to this 3rd-party service fail because this one cipher suite seems to be broken, or do you have some connections that work and others that do not? If some work, do they have anything in common with each other? > 2. prod Tomcat -> prod 3rd-party service with lower version of > Tomcat and/or Java WORKS with TLS_RSA_WITH_AES_128_CBC_SHA > >> [DMC] Correct So, downgrading does in fact improve the situation. Can you try upgrading Java in one of your working production environments but leave Tomcat alone? That would definitely rule-out Tomcat as a problem. > 1. test Tocmat -> test 3rd-party service WORKS with > TLS_RSA_WITH_AES_128_CBC_SHA > >> [DMC] Correct Good. What version was this running again? Same Java/Tomcat versions as the production server which seems to fail? > 2. test client (command-line?) on production host -> prod 3rd > parts service WORKS with TLS_RSA_WITH_AES_128_CBC_SHA > >> [DMC] Correct. The command line test client is a simple Java >> class that just opens an InputStream to the 3rd party service So: same Java, same host, same client library: works. But within Tomcat on that host with that Java with that same client library: fails? :( > No changes are known/expected, other than Tomcat and Java version. > >> [DMC] It's a greenfield app and no changes are planned bar >> functional patches. The same version of operating system, java, >> tomcat, and the application is in our non-prod environment and >> works ok There is obviously a difference. We just have to figure out what it is. What are your JVM parameters to launch Tomcat? You can find those out using "jinfo" from the command-line if you have the process id (at least I *think* that works on Windows). Check the running production application as well as the CLI client running on the same machine. Quadruple-check the Java versions of those processes. >>>> Two additional pieces of analysis that came out of our >>>> investigations today are: >>>> >>>> 1) we are using NextGen Cisco devices. These are application >>>> and protocol aware. This means that the firewall might allow >>>> allow our standalone Java client to connect using HTTPS, but >>>> interfere with the HTTPS SOAP requests from our Tomcat/3rd >>>> party app "Tomcat client". > > How can the device be "application aware"? How does the > applicaiton identify itself to the device? How is that > authenticated? What is stopping EvilApp.exe from claiming to be > InnocentApp.exe? > >> [DMC] the firewall devices check the tcp connections. I don't >> know how. I do know we had endless trouble from overzealous >> firewall guys putting restrictive rules in place. The impact of >> that was that, for example, we could telnet through the firewall >> but not establish a Sybase connection. In any case, we compared >> all the non-prod and prod firewall rules and they are configured >> the same way. We also had several debug sessions with the >> firewall guys and no traffic is being blocked. So at least we've >> ruled that out. Good. >>>> 2) there are some anecdotes of Java 7 clients failing to >>>> downgrade from TLSv1.2 to TLSv1. For example >>>> http://stackoverflow.com/questions/31579587/java-tls-1-2-downgrade- to- > >>>> tls-1-0-sometimes-error. > > This >>>> > is not a "piece of analysis". This is something you found online > that might be related. But you haven't collected any information > to support the idea that it's related. I note that the experience > is different in that SO article: the user is getting a different > error at a different part of the handshake. > >> [DMC] I agree. But in the spirit of trying different cipher >> suites, this is something that may (or may not) be relevant Okay. Conceptually, it's not hard to black-list a cipher suite. Your client library might not have that capability, though. Having done some of that work recently, I know that it takes miles of plumbing code so that the user can just call client.setAllowedCipherSuites(String[]). The client may simply not support it. >>>> Some takeaway actions for me to explore are below. I'll >>>> welcome any other thoughts that you may have. >>>> >>>> 1) compare the firewall rules on our working non-production >>>> client with the rules governing the failing production >>>> client > > Firewalls are usually pretty dumb. I'd be surprised if the firewall > is breaking anything, here. > >>>> 2) use openssl from our client server and see what it shows > > Yes, please do that. Why wasn't that the first thing you tried? > >> [DMC] capacity issues, to be honest. Analysing this type of issue >> in my organisation involves numerous teams across multiple >> timezones and the communication overheads are enormous. Plus our >> production servers are strictly controlled so I can't install it >> myself. It's taken me a week to get this far and it's been on my >> list all that time. Well, you can use "openssl s_client" from your own workstation, right? That's a step: find out if the server just can't handle that cipher, TLS1.2 handshakes, etc. No need to touch your own production servers. Also, you can download openssl and install it as a user: it's an all-in-one .exe on Windows: no administrative privileges are required. (Also, I'm pretty sure that Tomcat ships with openssl.exe. Check that out, too.) >>>> 3) consider forcing the Tomcat/3rd party Tomcat client to >>>> use TLSv1 > > This is a possibility, but you shouldn't have to hamper your own > application because there are some servers out there that are so > old they don't support TLSv1.2: you should always offer the best > encryption you can, and only fall-back when necessary. > >> [DMC] The 3rd party service only supports TLSv1. So as a >> diagnostic step we have nothing to lose. If the 3rd-party service only supports TLSv1, then why is their client library trying to connect using TLSv1.2? >>>> 4) check other logs on the server side to see if other >>>> clients are failing TLS handshakes > > You didn't do this already? > >> [DMC] our server-side partners are under the same production >> limitations that we are and insisted on discounting all possible >> infrastructure and client possibilities before increasing >> diagnostics. They did that today and found that the connections >> are being received but are being terminated by a generic SSL >> handshake error. Is that all connections from your servers, or only from those running the upgraded Tomcat/Java versions? >>>> 6) consider upgrading Tomcat (as a last resort) > > I thought the initial problem was that an upgrade to Tomcat "broke > something". I think it's more likely to be the Java upgrade that > did something, and not Tomcat itself. > >> [DMC] No, sorry for not being clear. This production instance is >> a greenfield installation and has never successfully connected >> (browser and standalone tests excepted) > > Are you sure that the standalone test client is using the exact > same version of Java as the Tomcat server is? > >> [DMC] 100%. There's only one version of java installed Okay. I'm interested to find out how this all ends. - -chris -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJV4LpoAAoJEBzwKT+lPKRYkdwP/jC08lLoa5P25ACD64wbv1ZA VBnVjfcg00gGE8UmRYNxdk6Aheqoj2o8UlPXM7HenrwUAPFg8XUEv1Mj63mt6d0A 1fLFn5fb5/KmozCNiOb/VwWKG5TnGvqHmfxbYgUHmzA5r1OV1rLzkYYYk6ed2ynw XJi+za4LWRcr2RDrEQkgcqFx1aF1EWy7PPxCmiuNAKDv53zCnXfUyCe19bDmGJgo IcT+bRA9mRYC5I7lzjLJwSThG3qyEHXR/YAfZHUt0Sd3Iv/7EacOoUyaUdmOsrAY 5n9Hy98Pckeb9XvdmrhxRhPr7kQINC4F0AuJF+mKN0m84jPJgrQjYogvXhiIJTMM WDHtjqbcM4xmd2OUKfbvFgYxHhBbOeSBDgKoV0Qpa8ydk1nPkCvyyjSHY3djyQod mo5wQociKmSLZhMC/R/A2beGnC+lYl2mI9Zz0qUPIspq7ZoNOgr2iDfjPgKaeIZW FhJ2jks0PoLt4AWwrs2A2Ia4/hSVOmlmW6d1eTt0uQIT9C4Zw6S33w4qWRNo8hrn quQPZUsLY+snA+Gc1Et/ksH3GBGDZOgMlxd6qDmcSMFmSS/qEQepDYKryuvLtyqq YPul5dwNgWCR6/j7cDBxgyqaTcNdScbxh1r6QIzu5BNLOfQtDl/BD3+x5u2WuP+G aFlTGsFRQKpXPrtbZiDH =92uM -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
