RE: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[John.E.Gregg]

> -Original Message-
> From: Mark Thomas 
> Sent: Sunday, August 09, 2020 2:19 PM
> To: Tomcat Users List 
> Subject: Re: Tomcat 8.5.(x > 5) & SSL Connections
> (sun.security.provider.certpath.SunCertPathBuilderException)
> 
> On August 8, 2020 6:59:23 PM UTC, David Filip  wrote:
> >Hello Everyone!
> >
> >I spent a large part of yesterday and this morning trying to debug an
> >SSL problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on
> >either this problem or something related back in 2016, but wanted to
> >confirm what the "correct" solution might be, because I got lost in the
> >threads.
> >
> >I never had this problem with Tomcat 7.0.x, but it started once I
> >upgraded to 8.5.57 (same application code), and it is related to making
> >outgoing SSL connections to web services.  And this is NOT related to a
> >self-signed, but to a commercial (GoDaddy) SSL certificate, albeit on a
> >server that I also run in the cloud.
> >
> >The exception is being thrown when trying to connect to an SSL
> >protected web service is:
> >
> >sun.security.validator.ValidatorException: PKIX path building failed:
> >sun.security.provider.certpath.SunCertPathBuilderException: unable to
> >find valid certification path to requested target
> >
> >although the exact same code worked (and still works on other servers)
> >reliably under Tomcat 7.0.x for several years.
> >
> >Now, here is the weird part: after Google'ing around, I thought the
> >problem might be that Tomcat 8.5.5 and later -- at least this is the
> >gist that I got -- no longer finds the 'default' Java certificate store
> >(cacerts), so I added the following to /bin/catalina.sh (running on a
> >Mac 10.14 / Mojave):
> >
> >export
> >-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_1
> >21.jdk/Contents/Home/jre/lib/security/cacerts
> >
> >The weird part is that this appeared to fix the problem, so I thought I
> >was done.  Then, I rebooted, and the problem re-appeared!
> >
> >I stopped and started Tomcat, and the problem was resolved again.  I
> >rebooted again, and the problem re-appeared.
> >
> >Previously, when it worked, I refreshed the page several times, and it
> >kept working.  When it doesn't work, if I keep refreshing the page, it
> >continues to throw the exception.
> >
> >Does this mean that some "worker threads" can find the certificate
> >store, and others can't?  Or am I going down the wrong rabbit hole?
> >
> >So, any idea?  The intermittent nature is driving me crazy!
> >
> >And I have can reproduce the problem on two separate servers (both Mac
> >10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and
> >one (slightly older server) running 8.5.35.  But again, I have several
> >7.0.x instances where I've never seen this problem before.
> >
> >Also, the generic 'SSLPoke' always connects to the service, and it
> >appears that if I run (mostly) the same code from the command line
> >outside of Tomcat (javac / java) it always works.  And if I paste the
> >web service URL into Safari or Chrome, it always works.  And if I use
> >the web service URL with curl (just for good measure), it always works.
> > So it only seems to fall under Tomcat 8.5.x.
> >
> >Thanks in advance for any guidance, as I'm running out of things to
> >Google and try.
> >
> >Regards,
> >
> >Dave.
> 
> Tomcat has zero involvement in outgoing TLS connections.
> 
> If the code works in a standalone Java app, it will work in a Servlet assuming
> the code is thread safe (I don't see why it wouldn't be but worth double
> checking any library being used) and configuration information is accessible.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

I agree with Mark.

This is probably an issue with the inability to find a CA in your trust store 
that matches the cert the server is presenting.  I would guess the different 
versions of Tomcat are looking at different trust stores, which are not 
included with Tomcat.

Probably the best thing to do would be to run with -Djavax.net.debug=ssl.  This 
will tell you what trust store is being used and what's in it.  It says right 
near the beginning of the output "trustStore is: /path/to/trust/store."  It 
will also show you the cert that the server is presenting.  Look below 
"ServerHello."  It produces a ton of output and is almost unusable if more than 
one request is going on at one time however, so it's best if you can run it on 
a "quiet" system.  Output will go to catalina.out.

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[David Filip]

Mark,

Thanks for the feedback!

If you do want do dig into this, have a look at the httpd virtual host
configuration. Depending on configuration, httpd will use the host
header or the IP address on which the connection was received to select
the virtual host. It will then send the cert relevant to that virtual host.

This makes sense, and this is something to look info, although my application 
code did not change, it just stopped working.  So it must have been a subtle 
change in the support libraries, as the application code just stopped working 
on both of my macOS 10.14 servers after working flawlessly for several years, 
and the same code continued to work flawlessly on my macOS 10.12 servers.

>> java.lang.ClassCastException: 
>> com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl 

>> cannot be cast to javax cannot be cast to javax.net.ssl.HttpsURLConnection


This I think is most telling as I never saw this problem before (until Friday 
when this all started), and since the application code did not change, the 
underlying libraries must have (I think the hint is 'Oldmpl' in the class name 
suffix, indicating a deprecation).

Your explanation makes sense about matching the certificate (self-signed vs. 
GoDaddy), and I did not knowingly touch the JRE in any way, but this wouldn't 
be the first time that an Apple macOS update "fixed" something and broke 
something else.  I just never thought that Apple would touch anything Java 
related in their updates (unless it is related to some JNI code that is calling 
some OS specific stuff somewhere?).

Nonetheless, thanks for your feedback, and I'm closing down this thread now.

Regards,

Dave.

> On Aug 10, 2020, at 4:28 AM, Mark Thomas  wrote:
> 
> On 09/08/2020 21:04, David Filip wrote:
>> Mark,
>> 
>>> If the code works in a standalone Java app, it will work in a Servlet 
>>> assuming the code is thread safe (I don't see why it wouldn't be but worth 
>>> double checking any library being used) and configuration information is 
>>> accessible.
>> 
>> 
>> All things being equal, that makes sense, and is probably true.  I think the 
>> problem was that at least in this case, not all things were equal.
>> 
>> Felix actually pointed me in right direction (thanks Felix!), as when I ran:
>> 
>> : | openssl x509 -text
>> 
>> I was shocked to see a self-signed certificate with the physical name of the 
>> server, and not my GoDaddy 10-SAN SSL certificate that I paid $$$ for.  This 
>> despite the fact that if I pointed a web browser at the web service w/SSL 
>> and clicked on the lock icon (tried both Safari and Chrome) it displayed the 
>> proper GoDaddy 10-SAN SSL certificate.
>> 
>> The SSL in front of my web service is provided by Apache 2.4.  Digging into 
>> the configuration, I found that "something" (I believe this is happens when 
>> you do a 'yum install mod_proxy' on CentOS) put a default self-signed 
>> certificate reference into .../conf.d/ssl.conf.  I referenced only the 
>> GoDaddy 10-SAN certificate in the virtual host for the web service, but 
>> Apache must have added the additional certificate (globally across virtual 
>> hosts?) in the SSL handshake?
>> 
>> What appeared to be happening is that on the macOS 10.14 (Mojave) servers -- 
>> I have two of them -- they getting the self-signed certificate, even though 
>> at the same time the macOS 10.12 (High Sierra) servers -- I have several of 
>> them -- were finding the GoDaddy 10-SAN certificate.  As well as the brewers 
>> on BOTH 10.14 and 10.12.
>> 
>> So my working theory?  Apache was sending out both the self-signed and 
>> GoDaddy certificates, and browsers, curl, etc., were accepting the one that 
>> they liked (GoDaddy) and ignoring the one they didn't (self-signed).
>> 
>> What is left unresolved is why the macOS 10.14 servers worked until 
>> somewhere between a few days and a week ago without me knowing changing 
>> anything ... as the application code did not change, and it is an 
>> application that I probably hit about once or twice a week or so.  The only 
>> thing that I knowingly did is apply some updates from Apple, but not 
>> explicitly to the JVM as far as I know.
>> 
>> So why would it work stand-alone and not in Tomcat?  Must be different 
>> libraries in my CLASSPATH ... which I have yet to completely figure out ... 
>> as another thing that I noticed was another bit of code that used to work 
>> flawlessly for several years up until about a week ago was now throwing the 
>> exception (only on the macOS 10.14 servers):
>> 
>> java.lang.ClassCastException: 
>> com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl 
>> cannot be cast to javax cannot be cast to javax.net.ssl.HttpsURLConnection
>> So, that same code continued to work on my macOS 10.12 servers, but not not 
>> the macOS 10.14 servers.  I wrote that code back in 2015 and forget why I 
>> was using that particular Sun library, but it no longer worked for HTTPS 
>> connections 

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[Mark Thomas]

On 09/08/2020 21:04, David Filip wrote:
> Mark,
> 
>> If the code works in a standalone Java app, it will work in a Servlet 
>> assuming the code is thread safe (I don't see why it wouldn't be but worth 
>> double checking any library being used) and configuration information is 
>> accessible.
> 
> 
> All things being equal, that makes sense, and is probably true.  I think the 
> problem was that at least in this case, not all things were equal.
> 
> Felix actually pointed me in right direction (thanks Felix!), as when I ran:
> 
> : | openssl x509 -text
> 
> I was shocked to see a self-signed certificate with the physical name of the 
> server, and not my GoDaddy 10-SAN SSL certificate that I paid $$$ for.  This 
> despite the fact that if I pointed a web browser at the web service w/SSL and 
> clicked on the lock icon (tried both Safari and Chrome) it displayed the 
> proper GoDaddy 10-SAN SSL certificate.
> 
> The SSL in front of my web service is provided by Apache 2.4.  Digging into 
> the configuration, I found that "something" (I believe this is happens when 
> you do a 'yum install mod_proxy' on CentOS) put a default self-signed 
> certificate reference into .../conf.d/ssl.conf.  I referenced only the 
> GoDaddy 10-SAN certificate in the virtual host for the web service, but 
> Apache must have added the additional certificate (globally across virtual 
> hosts?) in the SSL handshake?
> 
> What appeared to be happening is that on the macOS 10.14 (Mojave) servers -- 
> I have two of them -- they getting the self-signed certificate, even though 
> at the same time the macOS 10.12 (High Sierra) servers -- I have several of 
> them -- were finding the GoDaddy 10-SAN certificate.  As well as the brewers 
> on BOTH 10.14 and 10.12.
> 
> So my working theory?  Apache was sending out both the self-signed and 
> GoDaddy certificates, and browsers, curl, etc., were accepting the one that 
> they liked (GoDaddy) and ignoring the one they didn't (self-signed).
> 
> What is left unresolved is why the macOS 10.14 servers worked until somewhere 
> between a few days and a week ago without me knowing changing anything ... as 
> the application code did not change, and it is an application that I probably 
> hit about once or twice a week or so.  The only thing that I knowingly did is 
> apply some updates from Apple, but not explicitly to the JVM as far as I know.
> 
> So why would it work stand-alone and not in Tomcat?  Must be different 
> libraries in my CLASSPATH ... which I have yet to completely figure out ... 
> as another thing that I noticed was another bit of code that used to work 
> flawlessly for several years up until about a week ago was now throwing the 
> exception (only on the macOS 10.14 servers):
> 
> java.lang.ClassCastException: 
> com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl 
> cannot be cast to javax cannot be cast to javax.net.ssl.HttpsURLConnection
> So, that same code continued to work on my macOS 10.12 servers, but not not 
> the macOS 10.14 servers.  I wrote that code back in 2015 and forget why I was 
> using that particular Sun library, but it no longer worked for HTTPS 
> connections on macOS 10.14, so I changed it to java.net 
> .HttpURLConnection and it started working again.
> 
> So something somewhere changed in how SSL is being used between macOS 10.12 
> and 10.14 ... the former worked with the extraneous self-signed certificate, 
> and with com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnection, and 
> the latter did not work with either.
> 
> What I do not yet understand is why after various restarts of the application 
> and/or a reboot of the server and/or restart of Tomcat -- and moving between 
> Tomcat versions -- had it sporadically work, but most times not.  Something 
> cached somewhere?  Does the O/S cache SSL connections somewhere in the 
> kernel?  What and where?  Sadly, I still don't know, and can't explain some 
> of what I was seeing.
> 
> But it is working reliably now (even on macOS 10.14), after removing the 
> extraneous self-signed Apache certificate, and after re-writing a bit of code 
> (changing com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnection to 
> java.net .HttpURLConnection).
> 
> And it still bothers me that on the older Macs (macOS 10.12) it never stopped 
> working, and up until about a week ago, it worked correctly on the newer Macs 
> (macOS 10.14), until it didn't, without knowing what changed.
> 
> Clear as mud?

Absolutely.

If you do want do dig into this, have a look at the httpd virtual host
configuration. Depending on configuration, httpd will use the host
header or the IP address on which the connection was received to select
the virtual host. It will then send the cert relevant to that virtual host.

Given you had client issues as well, I suspect what you were seeing was
the result of the client sending a different host header and/or the
client 

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[David Filip]

Mark,

> If the code works in a standalone Java app, it will work in a Servlet 
> assuming the code is thread safe (I don't see why it wouldn't be but worth 
> double checking any library being used) and configuration information is 
> accessible.


All things being equal, that makes sense, and is probably true.  I think the 
problem was that at least in this case, not all things were equal.

Felix actually pointed me in right direction (thanks Felix!), as when I ran:

: | openssl x509 -text

I was shocked to see a self-signed certificate with the physical name of the 
server, and not my GoDaddy 10-SAN SSL certificate that I paid $$$ for.  This 
despite the fact that if I pointed a web browser at the web service w/SSL and 
clicked on the lock icon (tried both Safari and Chrome) it displayed the proper 
GoDaddy 10-SAN SSL certificate.

The SSL in front of my web service is provided by Apache 2.4.  Digging into the 
configuration, I found that "something" (I believe this is happens when you do 
a 'yum install mod_proxy' on CentOS) put a default self-signed certificate 
reference into .../conf.d/ssl.conf.  I referenced only the GoDaddy 10-SAN 
certificate in the virtual host for the web service, but Apache must have added 
the additional certificate (globally across virtual hosts?) in the SSL 
handshake?

What appeared to be happening is that on the macOS 10.14 (Mojave) servers -- I 
have two of them -- they getting the self-signed certificate, even though at 
the same time the macOS 10.12 (High Sierra) servers -- I have several of them 
-- were finding the GoDaddy 10-SAN certificate.  As well as the brewers on BOTH 
10.14 and 10.12.

So my working theory?  Apache was sending out both the self-signed and GoDaddy 
certificates, and browsers, curl, etc., were accepting the one that they liked 
(GoDaddy) and ignoring the one they didn't (self-signed).

What is left unresolved is why the macOS 10.14 servers worked until somewhere 
between a few days and a week ago without me knowing changing anything ... as 
the application code did not change, and it is an application that I probably 
hit about once or twice a week or so.  The only thing that I knowingly did is 
apply some updates from Apple, but not explicitly to the JVM as far as I know.

So why would it work stand-alone and not in Tomcat?  Must be different 
libraries in my CLASSPATH ... which I have yet to completely figure out ... as 
another thing that I noticed was another bit of code that used to work 
flawlessly for several years up until about a week ago was now throwing the 
exception (only on the macOS 10.14 servers):

java.lang.ClassCastException: 
com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl 
cannot be cast to javax cannot be cast to javax.net.ssl.HttpsURLConnection
So, that same code continued to work on my macOS 10.12 servers, but not not the 
macOS 10.14 servers.  I wrote that code back in 2015 and forget why I was using 
that particular Sun library, but it no longer worked for HTTPS connections on 
macOS 10.14, so I changed it to java.net .HttpURLConnection 
and it started working again.

So something somewhere changed in how SSL is being used between macOS 10.12 and 
10.14 ... the former worked with the extraneous self-signed certificate, and 
with com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnection, and the 
latter did not work with either.

What I do not yet understand is why after various restarts of the application 
and/or a reboot of the server and/or restart of Tomcat -- and moving between 
Tomcat versions -- had it sporadically work, but most times not.  Something 
cached somewhere?  Does the O/S cache SSL connections somewhere in the kernel?  
What and where?  Sadly, I still don't know, and can't explain some of what I 
was seeing.

But it is working reliably now (even on macOS 10.14), after removing the 
extraneous self-signed Apache certificate, and after re-writing a bit of code 
(changing com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnection to 
java.net .HttpURLConnection).

And it still bothers me that on the older Macs (macOS 10.12) it never stopped 
working, and up until about a week ago, it worked correctly on the newer Macs 
(macOS 10.14), until it didn't, without knowing what changed.

Clear as mud?

> On Aug 9, 2020, at 3:19 PM, Mark Thomas  wrote:
> 
> On August 8, 2020 6:59:23 PM UTC, David Filip  wrote:
>> Hello Everyone!
>> 
>> I spent a large part of yesterday and this morning trying to debug an
>> SSL problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on
>> either this problem or something related back in 2016, but wanted to
>> confirm what the "correct" solution might be, because I got lost in the
>> threads.
>> 
>> I never had this problem with Tomcat 7.0.x, but it started once I
>> upgraded to 8.5.57 (same application code), and it is related to making
>> outgoing SSL connections to web services.  And this is NOT related 

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[Mark Thomas]

On August 8, 2020 6:59:23 PM UTC, David Filip  wrote:
>Hello Everyone!
>
>I spent a large part of yesterday and this morning trying to debug an
>SSL problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on
>either this problem or something related back in 2016, but wanted to
>confirm what the "correct" solution might be, because I got lost in the
>threads.
>
>I never had this problem with Tomcat 7.0.x, but it started once I
>upgraded to 8.5.57 (same application code), and it is related to making
>outgoing SSL connections to web services.  And this is NOT related to a
>self-signed, but to a commercial (GoDaddy) SSL certificate, albeit on a
>server that I also run in the cloud.
>
>The exception is being thrown when trying to connect to an SSL
>protected web service is:
>
>sun.security.validator.ValidatorException: PKIX path building failed:
>sun.security.provider.certpath.SunCertPathBuilderException: unable to
>find valid certification path to requested target
>
>although the exact same code worked (and still works on other servers)
>reliably under Tomcat 7.0.x for several years.
>
>Now, here is the weird part: after Google'ing around, I thought the
>problem might be that Tomcat 8.5.5 and later -- at least this is the
>gist that I got -- no longer finds the 'default' Java certificate store
>(cacerts), so I added the following to /bin/catalina.sh (running on a
>Mac 10.14 / Mojave):
>
>export
>-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
>
>The weird part is that this appeared to fix the problem, so I thought I
>was done.  Then, I rebooted, and the problem re-appeared!
>
>I stopped and started Tomcat, and the problem was resolved again.  I
>rebooted again, and the problem re-appeared.
>
>Previously, when it worked, I refreshed the page several times, and it
>kept working.  When it doesn't work, if I keep refreshing the page, it
>continues to throw the exception.
>
>Does this mean that some "worker threads" can find the certificate
>store, and others can't?  Or am I going down the wrong rabbit hole?
>
>So, any idea?  The intermittent nature is driving me crazy!
>
>And I have can reproduce the problem on two separate servers (both Mac
>10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and
>one (slightly older server) running 8.5.35.  But again, I have several
>7.0.x instances where I've never seen this problem before.
>
>Also, the generic 'SSLPoke' always connects to the service, and it
>appears that if I run (mostly) the same code from the command line
>outside of Tomcat (javac / java) it always works.  And if I paste the
>web service URL into Safari or Chrome, it always works.  And if I use
>the web service URL with curl (just for good measure), it always works.
> So it only seems to fall under Tomcat 8.5.x.
>
>Thanks in advance for any guidance, as I'm running out of things to
>Google and try.
>
>Regards,
>
>Dave.

Tomcat has zero involvement in outgoing TLS connections.

If the code works in a standalone Java app, it will work in a Servlet assuming 
the code is thread safe (I don't see why it wouldn't be but worth double 
checking any library being used) and configuration information is accessible.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[Felix Schumacher]

Hi David,

when debugging SSL problems, it is important to know, which certs are
the troublemakers. Are those errors reported when tomcat is used as a
client as a server?

Try to get more information about the cert that is making problems. I
always try to get the cert with an openssl command like

: | openssl x509 -text

It will print out a lot of information about the cert of the target you
want to access (I assume, that you use tomcat as a client, that is your
app code is the client).

Problems with a cert could arise from an old hashing algorithm, a
signing CA, that your trust store doesn't trust, a signature algorithm, ...

If you want to debug yourself a bit further, you could switch on the
debugging of certs/ssl stuff by setting the JVM system property
javax.net.debug=all (or a bit less).

As a tip, I would add properties via $CATALINA_BASE/bin/setenv.sh
instead of editing catalina.sh.

Cert problems depend on the trust store you are using and each JVM
brings there own set of trusted ca certs, so be sure to have a look at
the used JVMs. Which worked, which doesn't?

Hope this will help you debugging the problem further.

Felix

Am 09.08.20 um 00:16 schrieb David Filip:
> Well, it is not consistent ... sometimes when I stop and start it from the 
> command line it works, and other times it doesn't, but every time it is 
> getting the -Djavax.net .ssl.trustStore parameter ... 
> which I check by doing a:
>
> $ ps -aef | grep java | tr ' ' '\n'
>
> which lists each parameter on a separate line.  So I wish it were that 
> simple, but it does not appear to be so.  But specifically to answer your 
> question, I use this script to toggle tomcat on and off:
>
> # tomcat
> #
> # Start / Stop Tomcat Application Server
> #
> # - If tomcat is running, stop it
> # - if tomcat is not running, start it
> #
> # 24-Apr-2010 - DEF, original coding
> #
>
> found=`ps -aef | grep /Library/Tomcat/bin/bootstrap.jar | grep -v grep | wc 
> -l`
>
> if [ $found -eq 0 ]
> then
>   echo Starting Tomcat Application Server ...
>   sudo launchctl load /Library/LaunchDaemons/org.apache.tomcat.plist
> else
>   echo Stopping Tomcat Application Server ...
>   sudo launchctl unload /Library/LaunchDaemons/org.apache.tomcat.plist
> fi
>
> And the org.apache.tomcat.plist contains:
>
> 
>  "http://www.apple.com/DTDs/PropertyList-1.0.dtd;>
> 
>   
>   Label
>   org.apache.tomcat
>   RunAtLoad
>   
>   ProgramArguments
>   
>   /Library/Tomcat/bin/catalina.sh
>   run
>   
>   StandardOutPath
>   /tmp/Tomcat.log
>   UserName
>   tomcat
>   
> 
>
> so it is using catalina.sh.  But right now, I just ran this command:
>
> $ ps -aef | grep java | tr ' ' '\n' | grep -- '-D'
> -Djava.util.logging.config.file=/Library/Tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Dbis.home=/home/infodesk
> -Dbis.download=/tmp
> -Dinfodoc.home=/home/infodesk
> -Dinfodoc.download=/tmp
> -Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Djava.endorsed.dirs=/Library/Tomcat/endorsed
> -Dcatalina.base=/Library/Tomcat
> -Dcatalina.home=/Library/Tomcat
> -Djava.io.tmpdir=/Library/Tomcat/temp
>
> and that file exists:
>
> $ ls -l 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
> -rw-rw-r--  1 root  wheel  115588 Dec  1  2019 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
>
> but I am getting the error:
>
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
>
> However, I may have previously misspoken, as I *thought* it was related to 
> Apache 8.5.x, because I saw it on the new server with 8.5.57 but not an older 
> server running 7.0.x, and when I upgraded the old 7.0.x server to 8.5.57 it 
> immediately began to exhibit the problem.  So cause and effect, right?  But I 
> just downgraded the old server back to 7.0.x, and I am still seeing the 
> problem!  Ugh!
>
> So I tried running the same app on an 8.5.37 server running on AWS Linux 2 
> (similar to CentOS 7), and it works fine there, even after stopping and 
> starting the Tomcat server 6 (!) times -- just because I don't trust anything 
> right now.
>
> My original thought -- which I did not previously share because I convinced 
> myself it was crazy -- was that I first noticed the problem after I applied 
> the latest macOS security patch.  However, once I saw that the Tomcat 7.0.x 
> server 

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[David Filip]

Well, it is not consistent ... sometimes when I stop and start it from the 
command line it works, and other times it doesn't, but every time it is getting 
the -Djavax.net .ssl.trustStore parameter ... which I check 
by doing a:

$ ps -aef | grep java | tr ' ' '\n'

which lists each parameter on a separate line.  So I wish it were that simple, 
but it does not appear to be so.  But specifically to answer your question, I 
use this script to toggle tomcat on and off:

# tomcat
#
# Start / Stop Tomcat Application Server
#
# - If tomcat is running, stop it
# - if tomcat is not running, start it
#
# 24-Apr-2010 - DEF, original coding
#

found=`ps -aef | grep /Library/Tomcat/bin/bootstrap.jar | grep -v grep | wc -l`

if [ $found -eq 0 ]
then
echo Starting Tomcat Application Server ...
sudo launchctl load /Library/LaunchDaemons/org.apache.tomcat.plist
else
echo Stopping Tomcat Application Server ...
sudo launchctl unload /Library/LaunchDaemons/org.apache.tomcat.plist
fi

And the org.apache.tomcat.plist contains:


http://www.apple.com/DTDs/PropertyList-1.0.dtd;>


Label
org.apache.tomcat
RunAtLoad

ProgramArguments

/Library/Tomcat/bin/catalina.sh
run

StandardOutPath
/tmp/Tomcat.log
UserName
tomcat



so it is using catalina.sh.  But right now, I just ran this command:

$ ps -aef | grep java | tr ' ' '\n' | grep -- '-D'
-Djava.util.logging.config.file=/Library/Tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Dbis.home=/home/infodesk
-Dbis.download=/tmp
-Dinfodoc.home=/home/infodesk
-Dinfodoc.download=/tmp
-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Djava.endorsed.dirs=/Library/Tomcat/endorsed
-Dcatalina.base=/Library/Tomcat
-Dcatalina.home=/Library/Tomcat
-Djava.io.tmpdir=/Library/Tomcat/temp

and that file exists:

$ ls -l 
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
-rw-rw-r--  1 root  wheel  115588 Dec  1  2019 
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts

but I am getting the error:

sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

However, I may have previously misspoken, as I *thought* it was related to 
Apache 8.5.x, because I saw it on the new server with 8.5.57 but not an older 
server running 7.0.x, and when I upgraded the old 7.0.x server to 8.5.57 it 
immediately began to exhibit the problem.  So cause and effect, right?  But I 
just downgraded the old server back to 7.0.x, and I am still seeing the 
problem!  Ugh!

So I tried running the same app on an 8.5.37 server running on AWS Linux 2 
(similar to CentOS 7), and it works fine there, even after stopping and 
starting the Tomcat server 6 (!) times -- just because I don't trust anything 
right now.

My original thought -- which I did not previously share because I convinced 
myself it was crazy -- was that I first noticed the problem after I applied the 
latest macOS security patch.  However, once I saw that the Tomcat 7.0.x server 
with the same macOS security patch did not exhibit the problem, I ASSUMED it 
was related to the Tomcat version ... but as I said, now that I have downgraded 
that server back to 7.0.x, I am not still -- sometimes -- seeing the problem.  
Ugh.

So I am now taking the "new" server and restoring from a backup taken a week 
ago -- before I applied the macOS security patch -- to see if that makes a 
difference.

Given all of that, I can assure you that I am not drinking -- at least not 
while in front of a computer -- and I am not taking any drugs, and as far as I 
know, I am not clinically insane.  But I still can't explain all of the 
inconsistencies I am seeing, and the one thing that I always hate most when 
debugging a problem is lack of a consistent reproducibility.

> On Aug 8, 2020, at 5:17 PM, calder  wrote:
> 
> On Sat, Aug 8, 2020, 13:59 David Filip  > wrote:
> 
>> Hello Everyone!
>> 
>> I spent a large part of yesterday and this morning trying to debug an SSL
>> problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on either
>> this problem or something related back in 2016, but wanted to confirm what
>> the "correct" solution might be, because I got lost in the threads.
>> 
>> I never had this problem with Tomcat 7.0.x, but it started once I upgraded
>> to 8.5.57 (same 

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[calder]

On Sat, Aug 8, 2020, 13:59 David Filip  wrote:

> Hello Everyone!
>
> I spent a large part of yesterday and this morning trying to debug an SSL
> problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on either
> this problem or something related back in 2016, but wanted to confirm what
> the "correct" solution might be, because I got lost in the threads.
>
> I never had this problem with Tomcat 7.0.x, but it started once I upgraded
> to 8.5.57 (same application code), and it is related to making outgoing SSL
> connections to web services.  And this is NOT related to a self-signed, but
> to a commercial (GoDaddy) SSL certificate, albeit on a server that I also
> run in the cloud.
>
> The exception is being thrown when trying to connect to an SSL protected
> web service is:
>
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
>
> although the exact same code worked (and still works on other servers)
> reliably under Tomcat 7.0.x for several years.
>
> Now, here is the weird part: after Google'ing around, I thought the
> problem might be that Tomcat 8.5.5 and later -- at least this is the gist
> that I got -- no longer finds the 'default' Java certificate store
> (cacerts), so I added the following to /bin/catalina.sh (running on a Mac
> 10.14 / Mojave):
>
> export
> -Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
>
> The weird part is that this appeared to fix the problem, so I thought I
> was done.  Then, I rebooted, and the problem re-appeared!
>
> I stopped and started Tomcat, and the problem was resolved again.  I
> rebooted again, and the problem re-appeared.
>



When you "stopped and started Tomcat", how did restart it?  At the command
line using one of the Tomcat shell scripts?

My thought is, "whatever" fires up Tomcat after an iOS system reboot - that
startup process does not call catalina.sh.

But when you start Tomcat manually, using catalina.sh or startup.sh (which
calls catalina.sh), it works because the Java option is being set.



Previously, when it worked, I refreshed the page several times, and it kept
> working.  When it doesn't work, if I keep refreshing the page, it continues
> to throw the exception.
>
> Does this mean that some "worker threads" can find the certificate store,
> and others can't?  Or am I going down the wrong rabbit hole?
>
> So, any idea?  The intermittent nature is driving me crazy!
>
> And I have can reproduce the problem on two separate servers (both Mac
> 10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and one
> (slightly older server) running 8.5.35.  But again, I have several 7.0.x
> instances where I've never seen this problem before.
>
> Also, the generic 'SSLPoke' always connects to the service, and it appears
> that if I run (mostly) the same code from the command line outside of
> Tomcat (javac / java) it always works.  And if I paste the web service URL
> into Safari or Chrome, it always works.  And if I use the web service URL
> with curl (just for good measure), it always works.  So it only seems to
> fall under Tomcat 8.5.x.
>
> Thanks in advance for any guidance, as I'm running out of things to Google
> and try.
>

Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[David Filip]

Hello Everyone!

I spent a large part of yesterday and this morning trying to debug an SSL 
problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on either this 
problem or something related back in 2016, but wanted to confirm what the 
"correct" solution might be, because I got lost in the threads.

I never had this problem with Tomcat 7.0.x, but it started once I upgraded to 
8.5.57 (same application code), and it is related to making outgoing SSL 
connections to web services.  And this is NOT related to a self-signed, but to 
a commercial (GoDaddy) SSL certificate, albeit on a server that I also run in 
the cloud.

The exception is being thrown when trying to connect to an SSL protected web 
service is:

sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

although the exact same code worked (and still works on other servers) reliably 
under Tomcat 7.0.x for several years.

Now, here is the weird part: after Google'ing around, I thought the problem 
might be that Tomcat 8.5.5 and later -- at least this is the gist that I got -- 
no longer finds the 'default' Java certificate store (cacerts), so I added the 
following to /bin/catalina.sh (running on a Mac 10.14 / Mojave):

export 
-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts

The weird part is that this appeared to fix the problem, so I thought I was 
done.  Then, I rebooted, and the problem re-appeared!

I stopped and started Tomcat, and the problem was resolved again.  I rebooted 
again, and the problem re-appeared.

Previously, when it worked, I refreshed the page several times, and it kept 
working.  When it doesn't work, if I keep refreshing the page, it continues to 
throw the exception.

Does this mean that some "worker threads" can find the certificate store, and 
others can't?  Or am I going down the wrong rabbit hole?

So, any idea?  The intermittent nature is driving me crazy!

And I have can reproduce the problem on two separate servers (both Mac 10.14 / 
Mojave, both Java 1.8.0), one (new server) running 8.5.57 and one (slightly 
older server) running 8.5.35.  But again, I have several 7.0.x instances where 
I've never seen this problem before.

Also, the generic 'SSLPoke' always connects to the service, and it appears that 
if I run (mostly) the same code from the command line outside of Tomcat (javac 
/ java) it always works.  And if I paste the web service URL into Safari or 
Chrome, it always works.  And if I use the web service URL with curl (just for 
good measure), it always works.  So it only seems to fall under Tomcat 8.5.x.

Thanks in advance for any guidance, as I'm running out of things to Google and 
try.

Regards,

Dave.