Mark, Thanks for the feedback!
If you do want do dig into this, have a look at the httpd virtual host configuration. Depending on configuration, httpd will use the host header or the IP address on which the connection was received to select the virtual host. It will then send the cert relevant to that virtual host. This makes sense, and this is something to look info, although my application code did not change, it just stopped working. So it must have been a subtle change in the support libraries, as the application code just stopped working on both of my macOS 10.14 servers after working flawlessly for several years, and the same code continued to work flawlessly on my macOS 10.12 servers. >> java.lang.ClassCastException: >> com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl >> cannot be cast to javax cannot be cast to javax.net.ssl.HttpsURLConnection This I think is most telling as I never saw this problem before (until Friday when this all started), and since the application code did not change, the underlying libraries must have (I think the hint is 'Oldmpl' in the class name suffix, indicating a deprecation). Your explanation makes sense about matching the certificate (self-signed vs. GoDaddy), and I did not knowingly touch the JRE in any way, but this wouldn't be the first time that an Apple macOS update "fixed" something and broke something else. I just never thought that Apple would touch anything Java related in their updates (unless it is related to some JNI code that is calling some OS specific stuff somewhere?). Nonetheless, thanks for your feedback, and I'm closing down this thread now. Regards, Dave. > On Aug 10, 2020, at 4:28 AM, Mark Thomas <ma...@apache.org> wrote: > > On 09/08/2020 21:04, David Filip wrote: >> Mark, >> >>> If the code works in a standalone Java app, it will work in a Servlet >>> assuming the code is thread safe (I don't see why it wouldn't be but worth >>> double checking any library being used) and configuration information is >>> accessible. >> >> >> All things being equal, that makes sense, and is probably true. I think the >> problem was that at least in this case, not all things were equal. >> >> Felix actually pointed me in right direction (thanks Felix!), as when I ran: >> >> </dev/null openssl s_client -connect <server>:<port> | openssl x509 -text >> >> I was shocked to see a self-signed certificate with the physical name of the >> server, and not my GoDaddy 10-SAN SSL certificate that I paid $$$ for. This >> despite the fact that if I pointed a web browser at the web service w/SSL >> and clicked on the lock icon (tried both Safari and Chrome) it displayed the >> proper GoDaddy 10-SAN SSL certificate. >> >> The SSL in front of my web service is provided by Apache 2.4. Digging into >> the configuration, I found that "something" (I believe this is happens when >> you do a 'yum install mod_proxy' on CentOS) put a default self-signed >> certificate reference into .../conf.d/ssl.conf. I referenced only the >> GoDaddy 10-SAN certificate in the virtual host for the web service, but >> Apache must have added the additional certificate (globally across virtual >> hosts?) in the SSL handshake? >> >> What appeared to be happening is that on the macOS 10.14 (Mojave) servers -- >> I have two of them -- they getting the self-signed certificate, even though >> at the same time the macOS 10.12 (High Sierra) servers -- I have several of >> them -- were finding the GoDaddy 10-SAN certificate. As well as the brewers >> on BOTH 10.14 and 10.12. >> >> So my working theory? Apache was sending out both the self-signed and >> GoDaddy certificates, and browsers, curl, etc., were accepting the one that >> they liked (GoDaddy) and ignoring the one they didn't (self-signed). >> >> What is left unresolved is why the macOS 10.14 servers worked until >> somewhere between a few days and a week ago without me knowing changing >> anything ... as the application code did not change, and it is an >> application that I probably hit about once or twice a week or so. The only >> thing that I knowingly did is apply some updates from Apple, but not >> explicitly to the JVM as far as I know. >> >> So why would it work stand-alone and not in Tomcat? Must be different >> libraries in my CLASSPATH ... which I have yet to completely figure out ... >> as another thing that I noticed was another bit of code that used to work >> flawlessly for several years up until about a week ago was now throwing the >> exception (only on the macOS 10.14 servers): >> >> java.lang.ClassCastException: >> com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl >> cannot be cast to javax cannot be cast to javax.net.ssl.HttpsURLConnection >> So, that same code continued to work on my macOS 10.12 servers, but not not >> the macOS 10.14 servers. I wrote that code back in 2015 and forget why I >> was using that particular Sun library, but it no longer worked for HTTPS >> connections on macOS 10.14, so I changed it to java.net <http://java.net/> >> <http://java.net/ <http://java.net/>>.HttpURLConnection and it started >> working again. >> >> So something somewhere changed in how SSL is being used between macOS 10.12 >> and 10.14 ... the former worked with the extraneous self-signed certificate, >> and with com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnection, and >> the latter did not work with either. >> >> What I do not yet understand is why after various restarts of the >> application and/or a reboot of the server and/or restart of Tomcat -- and >> moving between Tomcat versions -- had it sporadically work, but most times >> not. Something cached somewhere? Does the O/S cache SSL connections >> somewhere in the kernel? What and where? Sadly, I still don't know, and >> can't explain some of what I was seeing. >> >> But it is working reliably now (even on macOS 10.14), after removing the >> extraneous self-signed Apache certificate, and after re-writing a bit of >> code (changing >> com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnection to java.net >> <http://java.net/> <http://java.net/ <http://java.net/>>.HttpURLConnection). >> >> And it still bothers me that on the older Macs (macOS 10.12) it never >> stopped working, and up until about a week ago, it worked correctly on the >> newer Macs (macOS 10.14), until it didn't, without knowing what changed. >> >> Clear as mud? > > Absolutely. > > If you do want do dig into this, have a look at the httpd virtual host > configuration. Depending on configuration, httpd will use the host > header or the IP address on which the connection was received to select > the virtual host. It will then send the cert relevant to that virtual host. > > Given you had client issues as well, I suspect what you were seeing was > the result of the client sending a different host header and/or the > client connection via localhost vs the public IP or some combination of > the that. And if IPv6 is enabled then that adds another potential dimension. > > Generally, I'd recommend Wireshark or similar in this type of scenario. > Especially if you configure things so it can decrypt the TLS, it can be > very helpful to spot differences between different setups. > > Mark > > >> >>> On Aug 9, 2020, at 3:19 PM, Mark Thomas <ma...@apache.org> wrote: >>> >>> On August 8, 2020 6:59:23 PM UTC, David Filip <dfi...@colornet.com> wrote: >>>> Hello Everyone! >>>> >>>> I spent a large part of yesterday and this morning trying to debug an >>>> SSL problem on Tomcat 8.5.57 to no avail. I've seen some discussion on >>>> either this problem or something related back in 2016, but wanted to >>>> confirm what the "correct" solution might be, because I got lost in the >>>> threads. >>>> >>>> I never had this problem with Tomcat 7.0.x, but it started once I >>>> upgraded to 8.5.57 (same application code), and it is related to making >>>> outgoing SSL connections to web services. And this is NOT related to a >>>> self-signed, but to a commercial (GoDaddy) SSL certificate, albeit on a >>>> server that I also run in the cloud. >>>> >>>> The exception is being thrown when trying to connect to an SSL >>>> protected web service is: >>>> >>>> sun.security.validator.ValidatorException: PKIX path building failed: >>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>>> find valid certification path to requested target >>>> >>>> although the exact same code worked (and still works on other servers) >>>> reliably under Tomcat 7.0.x for several years. >>>> >>>> Now, here is the weird part: after Google'ing around, I thought the >>>> problem might be that Tomcat 8.5.5 and later -- at least this is the >>>> gist that I got -- no longer finds the 'default' Java certificate store >>>> (cacerts), so I added the following to /bin/catalina.sh (running on a >>>> Mac 10.14 / Mojave): >>>> >>>> export >>>> -Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts >>>> >>>> The weird part is that this appeared to fix the problem, so I thought I >>>> was done. Then, I rebooted, and the problem re-appeared! >>>> >>>> I stopped and started Tomcat, and the problem was resolved again. I >>>> rebooted again, and the problem re-appeared. >>>> >>>> Previously, when it worked, I refreshed the page several times, and it >>>> kept working. When it doesn't work, if I keep refreshing the page, it >>>> continues to throw the exception. >>>> >>>> Does this mean that some "worker threads" can find the certificate >>>> store, and others can't? Or am I going down the wrong rabbit hole? >>>> >>>> So, any idea? The intermittent nature is driving me crazy! >>>> >>>> And I have can reproduce the problem on two separate servers (both Mac >>>> 10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and >>>> one (slightly older server) running 8.5.35. But again, I have several >>>> 7.0.x instances where I've never seen this problem before. >>>> >>>> Also, the generic 'SSLPoke' always connects to the service, and it >>>> appears that if I run (mostly) the same code from the command line >>>> outside of Tomcat (javac / java) it always works. And if I paste the >>>> web service URL into Safari or Chrome, it always works. And if I use >>>> the web service URL with curl (just for good measure), it always works. >>>> So it only seems to fall under Tomcat 8.5.x. >>>> >>>> Thanks in advance for any guidance, as I'm running out of things to >>>> Google and try. >>>> >>>> Regards, >>>> >>>> Dave. >>> >>> Tomcat has zero involvement in outgoing TLS connections. >>> >>> If the code works in a standalone Java app, it will work in a Servlet >>> assuming the code is thread safe (I don't see why it wouldn't be but worth >>> double checking any library being used) and configuration information is >>> accessible. >>> >>> Mark >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > <mailto:users-unsubscr...@tomcat.apache.org> > For additional commands, e-mail: users-h...@tomcat.apache.org > <mailto:users-h...@tomcat.apache.org>
