[OT] Re: Critical Random "Can't read cryptographic policy directory: unlimited"

[Mark Eggers]

Jerry:

On 12/21/2021 4:17 PM, Jerry Malcolm wrote:

Mark,







Moving on the next fire that needs to be put out...

Thx

Jerry


On 12/20/2021 5:22 AM, Mark Thomas wrote:

Thanks for the comments.  I have somewhat of a solution.  I just want to document what I learned in case anyone else has this problem.  From what I have been able to determine, you are correct about it being an Amazon problem.  I don't really understand what is happening.  But here's what I found... After a day of testing all types of scenarios, I found that it only happens the first time after installing a new ami (image) on a brand new EC2 and bringing up the EC2 and auto-starting TC for  the first time.  But it fails EVERY time with this scenario. Bouncing TC fixed it every time. Bouncing the EC2 never causes it to come back. Only booting a brand new EC2 instance causes it. I tried removing the tomcat autostart, ran the ami EC2 creation, waited a bit, and went to the console and manually started tomcat.  No problem. So all I can deduce from this is that there's a timing issue. I'm suspecting that AWS is starting up the EC2 before it is fully created (likely something not complete in the file system setup). I added a 2-minute sleep in rc.local to delay the startup of tomcat long enough for whatever wasn't finished yet to complete. With a 2-min sleep, it now works every time.  Pretty ugly fix IMHO.  But working ugly beats not working pretty.  I'll probably play around with the 2-min sleep to see if I can reduce that.  I may try to talk to Amazon about it.  I may also try banging my head against the wall, probably with the same results as talking to amazon about this. The takeaway from this is that AWS appears to be trying to shave a little performance delay by jumping the gun on the EC2 boot. The result in this case is TC saying the crypto file doesn't exist, but it does exist a minute or two later.  Likely nothing to do specifically with the crypto file.  That's just likely the first file needed that wasn't ready yet.

On 20/12/2021 06:59, Jerry Malcolm wrote:

I'm adding a slight variation to the error I get at times (see bottom of stack trace below)

This is the code that throws the root exception:

if (!Files.isDirectory(cryptoPolicyPath)
        || !Files.isReadable(cryptoPolicyPath)) {
    throw new SecurityException(
        "Can't read cryptographic policy directory: " +
        cryptoPolicyProperty);
}




Mark

That points very strongly to a file system issue. My recommendation is to take this up with AWS support.

On 12/19/2021 11:55 PM, Jerry Malcolm wrote:

I have a production environment of 10+ AWS EC2 servers all built from a single EC2 snapshot image. The master EC2 works fine.  But when we propagated the image to all of the production servers, we started getting  "Can't read cryptographic policy directory: unlimited" errors when I try to get a jdbc connection object from the pool.  Full stack trace is below.  The three critical lines in the trace are:

java.lang.ExceptionInInitializerError












Stack trace of error:

java.lang.ExceptionInInitializerError
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:185)

at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:472)
at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
at java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:67)

at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)

at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)

at com.mysql.cj.NativeSession.connect(NativeSession.java:163)

at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:447)

at jwm.db.DBData.getConnection(DBData.java:443)
...

at java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120)
... 81 more

at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
.. 81 more

Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited On some machines rebooting the tomcat service fixes the problem and it continues to work fine after that.  On other machines, the problem is still there after rebooting.  But even on the machines that are fixed after TC bounce, if I bounce the full EC2, the problem is back. We've had this environment working for almost two years.  I did a minor version upgrade to TC 8.5.73 a month ago.  And I upgraded to java 11 probably a year ago.  Otherwise, no changes that I'm aware of.  Definitely nothing in the past few days before the error hit yesterday. Everything I can find on google about the error messages says to make sure the 'unlimited' folder is present and accessible.  The folder is there and has been there untouched.  And I know it's not disappearing and reappearing to become accessible after the TC reboot that sometimes fixes the problem. The mySQL RDS all of these instances talk to hasn't changed. And other servers that aren't part of this image distribution have no problems accessing it.  It just makes no sense that the same EC2 image that works on one machine started failing on a bunch of identical configuration EC2s yesterday. I know this call goes through layers of tomcat, layers of the mysql driver, and then layers of the jvm before it occurs. But can anyone help me understand what TC/mySQL might be trying to do with this call to JVM crypto code and why it has started failing on all of my servers?  Any Java crypto gurus out there? at java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.java:483) at java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuites(SSLContextImpl.java:348) at java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:580) at java.base/java.security.Provider$Service.getImplClass(Provider.java:1918) at java.base/java.security.Provider$Service.newInstance(Provider.java:1894) at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at com.mysql.cj.protocol.ExportControlled.getSSLContext(ExportControlled.java:565) at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:302) at com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:188) at com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:99) at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:331) at com.mysql.cj.protocol.a.NativeAuthenticationProvider.negotiateSSLConnection(NativeAuthenticationProvider.java:777) at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider. at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:202) at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1348) at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:947) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:817) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:237) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199) at org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) at org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:365) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:116) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:547) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:747) Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited at java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)

================= This is a variation of the error I've also found in the logs.  NoClassDefFoundError that is random just adds to the strangeness of this problem...

at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)

at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)

java.lang.NoClassDefFoundError: Could not initialize class sun.security.ssl.SSLContextImpl$TLSContext at java.base/java.security.Provider$Service.getImplClass(Provider.java:1918) at java.base/java.security.Provider$Service.newInstance(Provider.java:1894) at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at com.mysql.cj.protocol.ExportControlled.getSSLContext(ExportControlled.java:565) at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:302) at com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.j

Did you post the systemd files provided by AWS? If you did, I missed it.


/var/run/tomcat/[servicename].pid



Something like (tomcat-pre.service):

[Unit]
Description=Create Tomcat PID directory

[Service]
Type=oneshot
RuntimeDirectory=tomcat
User=tcadmin
Group=tcadmin
ExecStart=/bin/true
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

And then, my Tomcat systemd file looks like (in part):

[Unit]
Description=Apache Tomcat loki
After=network.target tomcat-pre.service



Hopefully that helps.

. . . just my two cents
/mde/

In my setup (local CentOS 7 and Ubuntu 20.04), I rely on a PID file written to Unfortunately, the /var/run directory is a tmpfs file system which is created at boot time. What I had to do was to create a systemd process that created the subdirectory, and have that as a requirement for Tomcat to start. You might check if the endorsed directory is created by a systemd service (normally a oneshot), and then add that service to the After line of the tomcat service systemd file.

OpenPGP_signature
Description: OpenPGP digital signature