Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Simon, On 6/21/23 03:19, Simon Matter wrote: Jerry, On 6/15/23 00:41, Jerry Malcolm wrote: On 6/13/2023 3:46 PM, Jerry Malcolm wrote: On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. One observation even though the error doesn't show up in the log until my first db call, I think the real error has to be occurring during tomcat boot. If the failure occurs, I continue to get the same "Can't read cryptographic policy directory: unlimited" even hours later after boot every time I send another request to TC. If I then reboot TC and this time I don't get the failure, that unlimited folder magically becomes available. There is something happening in TC boot with some sort of crypto initialization that either succeeds or fails. And the exception when trying to get connections long after TC boot is just hitting whatever problem occurred during boot. In other words, delaying my calls, even for hours, has no effect on accessing that folder if the problem is present. Rebooting TC gives it another chance to succeed or fail. Any ideas what TC could be doing on boot that could lock up that folder? I never really found a definitive fix for this. But after all of the research and logging and everything, I pretty much concluded that there's some kind of timing/race condition between Tomcat boot and Java crypto initialization. Given that, I figured it was highly unlikely that two different Java JVM implementations would have the same precise implementation code to trigger that race condition. So as a last ditch effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java JVM. It's circumstantial evidence at best, and running a production environment on fixes that just went away goes against my grain. But if the problem goes away, maybe it won't come back. At this time, when using Corretto JVM, I have not encountered the Crypto directory error. It's been running on all server instances now for almost 24 hours with no problems. So I guess my suggestion to anyone else that might encounter something like this, trying using a different JVM from a different provider. That appears at this time to have worked for me. Yeah, that's certainly a very unsatisfying result, but ... you can't argue with results. I'd love to hear if you end up with any further information. -chris I was asking this before and ask again, how do we know it's not a missing RNG entropy while the crypto stuff of the Java VM is initialised? Because the symptoms of *that* problem (very long lags when requesting random numbers) are not being experienced. And also because the error message is crystal clear: [...] Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited at java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326) [...] The cryptographic provider itself cannot even initialize. It's *way* before any randomness is needed. The "unlimited" directory is presumably the directory which contains the policy files which do not impose the archaic US-imposed "export restrictions" on cryptographic primitives. The Provider cannot find the configuration directory, and fails. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
> Jerry, > > On 6/15/23 00:41, Jerry Malcolm wrote: >> >> On 6/13/2023 3:46 PM, Jerry Malcolm wrote: >>> >>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: > In /etc/rc.local I have: >> >> -- >> sleep 120s >> systemctl start tomcat9 >> >> - >> >> I put the sleep in back a couple of years ago that for some reason >> 'fixed' this same random, intermittent crypto file exception. >> >> > >>> One observation even though the error doesn't show up in the log >>> until my first db call, I think the real error has to be occurring >>> during tomcat boot. If the failure occurs, I continue to get the same >>> "Can't read cryptographic policy directory: unlimited" even hours >>> later after boot every time I send another request to TC. If I then >>> reboot TC and this time I don't get the failure, that unlimited folder >>> magically becomes available. There is something happening in TC boot >>> with some sort of crypto initialization that either succeeds or >>> fails. And the exception when trying to get connections long after TC >>> boot is just hitting whatever problem occurred during boot. In other >>> words, delaying my calls, even for hours, has no effect on accessing >>> that folder if the problem is present. Rebooting TC gives it another >>> chance to succeed or fail. Any ideas what TC could be doing on boot >>> that could lock up that folder? >>> >>> >> I never really found a definitive fix for this. But after all of the >> research and logging and everything, I pretty much concluded that >> there's some kind of timing/race condition between Tomcat boot and Java >> crypto initialization. Given that, I figured it was highly unlikely >> that two different Java JVM implementations would have the same precise >> implementation code to trigger that race condition. So as a last ditch >> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java >> JVM. It's circumstantial evidence at best, and running a production >> environment on fixes that just went away goes against my grain. But if >> the problem goes away, maybe it won't come back. At this time, when >> using Corretto JVM, I have not encountered the Crypto directory error. >> It's been running on all server instances now for almost 24 hours with >> no problems. So I guess my suggestion to anyone else that might >> encounter something like this, trying using a different JVM from a >> different provider. That appears at this time to have worked for me. > > Yeah, that's certainly a very unsatisfying result, but ... you can't > argue with results. I'd love to hear if you end up with any further > information. > > -chris I was asking this before and ask again, how do we know it's not a missing RNG entropy while the crypto stuff of the Java VM is initialised? Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Jerry, On 6/15/23 00:41, Jerry Malcolm wrote: On 6/13/2023 3:46 PM, Jerry Malcolm wrote: On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. One observation even though the error doesn't show up in the log until my first db call, I think the real error has to be occurring during tomcat boot. If the failure occurs, I continue to get the same "Can't read cryptographic policy directory: unlimited" even hours later after boot every time I send another request to TC. If I then reboot TC and this time I don't get the failure, that unlimited folder magically becomes available. There is something happening in TC boot with some sort of crypto initialization that either succeeds or fails. And the exception when trying to get connections long after TC boot is just hitting whatever problem occurred during boot. In other words, delaying my calls, even for hours, has no effect on accessing that folder if the problem is present. Rebooting TC gives it another chance to succeed or fail. Any ideas what TC could be doing on boot that could lock up that folder? I never really found a definitive fix for this. But after all of the research and logging and everything, I pretty much concluded that there's some kind of timing/race condition between Tomcat boot and Java crypto initialization. Given that, I figured it was highly unlikely that two different Java JVM implementations would have the same precise implementation code to trigger that race condition. So as a last ditch effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java JVM. It's circumstantial evidence at best, and running a production environment on fixes that just went away goes against my grain. But if the problem goes away, maybe it won't come back. At this time, when using Corretto JVM, I have not encountered the Crypto directory error. It's been running on all server instances now for almost 24 hours with no problems. So I guess my suggestion to anyone else that might encounter something like this, trying using a different JVM from a different provider. That appears at this time to have worked for me. Yeah, that's certainly a very unsatisfying result, but ... you can't argue with results. I'd love to hear if you end up with any further information. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
> > On 6/13/2023 3:46 PM, Jerry Malcolm wrote: >> >> On 6/13/2023 12:39 PM, Jerry Malcolm wrote: >>> Rob, >>> >>> On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: > > -- > sleep 120s > systemctl start tomcat9 > > - > > I put the sleep in back a couple of years ago that for some reason > 'fixed' this same random, intermittent crypto file exception. > > >> One observation even though the error doesn't show up in the log >> until my first db call, I think the real error has to be occurring >> during tomcat boot. If the failure occurs, I continue to get the same >> "Can't read cryptographic policy directory: unlimited" even hours >> later after boot every time I send another request to TC. If I then >> reboot TC and this time I don't get the failure, that unlimited folder >> magically becomes available. There is something happening in TC boot >> with some sort of crypto initialization that either succeeds or >> fails. And the exception when trying to get connections long after TC >> boot is just hitting whatever problem occurred during boot. In other >> words, delaying my calls, even for hours, has no effect on accessing >> that folder if the problem is present. Rebooting TC gives it another >> chance to succeed or fail. Any ideas what TC could be doing on boot >> that could lock up that folder? >> >> > I never really found a definitive fix for this. But after all of the > research and logging and everything, I pretty much concluded that > there's some kind of timing/race condition between Tomcat boot and Java > crypto initialization. Given that, I figured it was highly unlikely > that two different Java JVM implementations would have the same precise > implementation code to trigger that race condition. So as a last ditch > effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java > JVM. It's circumstantial evidence at best, and running a production > environment on fixes that just went away goes against my grain. But if > the problem goes away, maybe it won't come back. At this time, when > using Corretto JVM, I have not encountered the Crypto directory error. > It's been running on all server instances now for almost 24 hours with > no problems. So I guess my suggestion to anyone else that might > encounter something like this, trying using a different JVM from a > different provider. That appears at this time to have worked for me. > Maybe you could report this as a bug in the affected JVM? If it's really coming from the JVM then it would be nice to have it fixed. Regards, Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Conclusion - Re: Crypto Randomly Not Getting Initialized
On 6/13/2023 3:46 PM, Jerry Malcolm wrote: On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. One observation even though the error doesn't show up in the log until my first db call, I think the real error has to be occurring during tomcat boot. If the failure occurs, I continue to get the same "Can't read cryptographic policy directory: unlimited" even hours later after boot every time I send another request to TC. If I then reboot TC and this time I don't get the failure, that unlimited folder magically becomes available. There is something happening in TC boot with some sort of crypto initialization that either succeeds or fails. And the exception when trying to get connections long after TC boot is just hitting whatever problem occurred during boot. In other words, delaying my calls, even for hours, has no effect on accessing that folder if the problem is present. Rebooting TC gives it another chance to succeed or fail. Any ideas what TC could be doing on boot that could lock up that folder? I never really found a definitive fix for this. But after all of the research and logging and everything, I pretty much concluded that there's some kind of timing/race condition between Tomcat boot and Java crypto initialization. Given that, I figured it was highly unlikely that two different Java JVM implementations would have the same precise implementation code to trigger that race condition. So as a last ditch effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java JVM. It's circumstantial evidence at best, and running a production environment on fixes that just went away goes against my grain. But if the problem goes away, maybe it won't come back. At this time, when using Corretto JVM, I have not encountered the Crypto directory error. It's been running on all server instances now for almost 24 hours with no problems. So I guess my suggestion to anyone else that might encounter something like this, trying using a different JVM from a different provider. That appears at this time to have worked for me. Thanks for all the support. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Jerry, On 6/13/23 12:25, Jerry Malcolm wrote: Chris, On 6/13/2023 11:03 AM, Christopher Schultz wrote: Jerry, On 6/13/23 11:51, Jerry Malcolm wrote: Reg, On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote: Hey Jerry, Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited Does this help? https://stackoverflow.com/a/58183460/ Yours, Reg The stackoverflow link provides a lot of useful info about the error. But unless something in the boot process deletes those files and recreates them later, they are all there and correct. One boot works fine. The next boot throws this error. The next boot is fine. Is there a possibility that permissions get altered on this 'unlimited' folder during boot? I can't think of many reasons it would say a folder doesn't exist that clearly does exist. How are you launching Tomcat on boot? Are you able to add some instrumentation/logging to that? -chris In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 That's an odd way to do things (in rc.local). If you have systemctl, you are using systemd. If you are using systemd, you isn't it auto-starting that service? $ sudo systemctl enable tomcat9 That ought to do it, and the tomcat9 service should start on startup, resolving any dependencies it needs. Are you able to do an "ls [path to policy files] > /tmp/some_log_file" in there to see the situation when the service is trying to start? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. One observation even though the error doesn't show up in the log until my first db call, I think the real error has to be occurring during tomcat boot. If the failure occurs, I continue to get the same "Can't read cryptographic policy directory: unlimited" even hours later after boot every time I send another request to TC. If I then reboot TC and this time I don't get the failure, that unlimited folder magically becomes available. There is something happening in TC boot with some sort of crypto initialization that either succeeds or fails. And the exception when trying to get connections long after TC boot is just hitting whatever problem occurred during boot. In other words, delaying my calls, even for hours, has no effect on accessing that folder if the problem is present. Rebooting TC gives it another chance to succeed or fail. Any ideas what TC could be doing on boot that could lock up that folder? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. You mentioned earlier that you had recently tried a 3 minute delay, to no effect. Did you try any delay longer than 3 minutes? Maybe something ridiculous like 15m? It could be that whatever was taking two minutes before is now taking much longer. No change in file permissions at or above the "unlimited" dir? I put the time back to 2 min after the 3 min had no effect. But after the failure occurred on an instance, and I got to all of the logs, I just did a tomcat reboot from the command line and it worked. So in that case 2-3 min failed. But several hours later it worked. Still not what could cause a "missing" folder to finally become available after several minutes (or hours). Definitely not doing anything with permissions on that file. Didn't even know it existed until this problem surfaced. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Hi, > Jerry, > > On 6/13/23 11:42, Jerry Malcolm wrote: >> Simon, >> >> On 6/13/2023 2:20 AM, Simon Matter wrote: >>> Hi, >>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon Linux2 in a production environment. A couple of years ago, we started getting weird errors that the "Crypto Mechanism" failed to initialize. Through a lot of trial and error, and reasons I don't quite remember, we put a 2-min delay in rc.local before starting Tomcat, and the problem went away. I'm not a Linux nor a crypto guru. But we traced it to some crypto file that we assumed was not available until later in the Linux boot sequence. Anyway, the 2 minute delay made it go away, for over two years. Then all of a sudden in the last day or so, it's back with a vengeance. It fails with the same crypto error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and no change. I need help. Our whole production environment is unstable now since every time an ASG brings a new instance online, I've got a 50-50 chance that tomcat is going to die (and the health check doesn't catch it, but that's a different issue). There are no errors in the Tomcat boot sequence logs. But the first time and every subsequent time I try to get a connection from the DataSource pool, I get the stack dump shown below. I figure it has to be a timing/race condition. But I have no clue what to do to fix it. I'm baffled that it worked for two years, and now fails every other time I start an instance. And every instance is running copies of the exact same Amazon Machine Image. The same EC2 will come up clean 50% of the time the next time it boots. >>> Could it be that your hosts are running out of entropy on boot? >>> >>> Maybe there were RNG related changes in the kernel or rng-tools? >>> >>> Maybe monitor available entropy in >>> /proc/sys/kernel/random/entropy_avail, >>> it should not go below 100 or so. >>> >>> Regards, >>> Simon >>> >> I haven't done any Linux or other system updates in several weeks. I'll >> look into the entropy possibility. Would running out of entropy cause >> an exception stating that a crypto directory doesn't exist? I don't >> know much about Java entropy. Any ideas what would cause entropy to be >> good on one boot and bad on the next boot? Thx. > > This isn't about entropy. Focus on the actual error message that the > system can't find the unlimited policy file for some reason. > > -chris Sorry, I didn't read the errors too carefully :) I don't know Amazon Linux2 but one question, does it have the directory /etc/crypto-policies? Apart from this, does systemd somehow fiddling with crypto stuff while booting? Since systemd parallelizes the whole boot process many things can (and sometimes do) break and it's difficult to find it out. Regards, Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. You mentioned earlier that you had recently tried a 3 minute delay, to no effect. Did you try any delay longer than 3 minutes? Maybe something ridiculous like 15m? It could be that whatever was taking two minutes before is now taking much longer. No change in file permissions at or above the "unlimited" dir? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
chris On 6/13/2023 11:03 AM, Christopher Schultz wrote: Jerry, On 6/13/23 11:51, Jerry Malcolm wrote: Reg, On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote: Hey Jerry, Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited Does this help? https://stackoverflow.com/a/58183460/ Yours, Reg The stackoverflow link provides a lot of useful info about the error. But unless something in the boot process deletes those files and recreates them later, they are all there and correct. One boot works fine. The next boot throws this error. The next boot is fine. Is there a possibility that permissions get altered on this 'unlimited' folder during boot? I can't think of many reasons it would say a folder doesn't exist that clearly does exist. How are you launching Tomcat on boot? Are you able to add some instrumentation/logging to that? -chris The Tomcat boot log in catalina.out runs fine. The exception first occurs when a request comes in and I have to go to the database. It happens on getting a connection from the datasource. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Chris, On 6/13/2023 11:03 AM, Christopher Schultz wrote: Jerry, On 6/13/23 11:51, Jerry Malcolm wrote: Reg, On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote: Hey Jerry, Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited Does this help? https://stackoverflow.com/a/58183460/ Yours, Reg The stackoverflow link provides a lot of useful info about the error. But unless something in the boot process deletes those files and recreates them later, they are all there and correct. One boot works fine. The next boot throws this error. The next boot is fine. Is there a possibility that permissions get altered on this 'unlimited' folder during boot? I can't think of many reasons it would say a folder doesn't exist that clearly does exist. How are you launching Tomcat on boot? Are you able to add some instrumentation/logging to that? -chris In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Jerry, On 6/13/23 11:51, Jerry Malcolm wrote: Reg, On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote: Hey Jerry, Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited Does this help? https://stackoverflow.com/a/58183460/ Yours, Reg The stackoverflow link provides a lot of useful info about the error. But unless something in the boot process deletes those files and recreates them later, they are all there and correct. One boot works fine. The next boot throws this error. The next boot is fine. Is there a possibility that permissions get altered on this 'unlimited' folder during boot? I can't think of many reasons it would say a folder doesn't exist that clearly does exist. How are you launching Tomcat on boot? Are you able to add some instrumentation/logging to that? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Jerry, On 6/13/23 11:42, Jerry Malcolm wrote: Simon, On 6/13/2023 2:20 AM, Simon Matter wrote: Hi, I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon Linux2 in a production environment. A couple of years ago, we started getting weird errors that the "Crypto Mechanism" failed to initialize. Through a lot of trial and error, and reasons I don't quite remember, we put a 2-min delay in rc.local before starting Tomcat, and the problem went away. I'm not a Linux nor a crypto guru. But we traced it to some crypto file that we assumed was not available until later in the Linux boot sequence. Anyway, the 2 minute delay made it go away, for over two years. Then all of a sudden in the last day or so, it's back with a vengeance. It fails with the same crypto error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and no change. I need help. Our whole production environment is unstable now since every time an ASG brings a new instance online, I've got a 50-50 chance that tomcat is going to die (and the health check doesn't catch it, but that's a different issue). There are no errors in the Tomcat boot sequence logs. But the first time and every subsequent time I try to get a connection from the DataSource pool, I get the stack dump shown below. I figure it has to be a timing/race condition. But I have no clue what to do to fix it. I'm baffled that it worked for two years, and now fails every other time I start an instance. And every instance is running copies of the exact same Amazon Machine Image. The same EC2 will come up clean 50% of the time the next time it boots. Could it be that your hosts are running out of entropy on boot? Maybe there were RNG related changes in the kernel or rng-tools? Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail, it should not go below 100 or so. Regards, Simon I haven't done any Linux or other system updates in several weeks. I'll look into the entropy possibility. Would running out of entropy cause an exception stating that a crypto directory doesn't exist? I don't know much about Java entropy. Any ideas what would cause entropy to be good on one boot and bad on the next boot? Thx. This isn't about entropy. Focus on the actual error message that the system can't find the unlimited policy file for some reason. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: Crypto Randomly Not Getting Initialized
Thomas, On 6/13/2023 10:45 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Jerry, -Ursprüngliche Nachricht- Von: Jerry Malcolm Gesendet: Dienstag, 13. Juni 2023 17:35 An: users@tomcat.apache.org Betreff: Re: AW: Crypto Randomly Not Getting Initialized Hi Thomas, On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Jerry, -Ursprüngliche Nachricht- Von: Jerry Malcolm Gesendet: Dienstag, 13. Juni 2023 08:50 An: users@tomcat.apache.org Betreff: Crypto Randomly Not Getting Initialized I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon Linux2 in a production environment. A couple of years ago, we started getting weird errors that the "Crypto Mechanism" failed to initialize. Through a lot of trial and error, and reasons I don't quite remember, we put a 2-min delay in rc.local before starting Tomcat, and the problem went away. I'm not a Linux nor a crypto guru. But we traced it to some crypto file that we assumed was not available until later in the Linux boot sequence. Anyway, the 2 minute delay made it go away, for over two years. Then all of a sudden in the last day or so, it's back with a vengeance. It fails with the same crypto error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and no change. I need help. Our whole production environment is unstable now since every time an ASG brings a new instance online, I've got a 50-50 chance that tomcat is going to die (and the health check doesn't catch it, but that's a different issue). There are no errors in the Tomcat boot sequence logs. But the first time and every subsequent time I try to get a connection from the DataSource pool, I get the stack dump shown below. I figure it has to be a timing/race condition. But I have no clue what to do to fix it. I'm baffled that it worked for two years, and now fails every other time I start an instance. And every instance is running copies of the exact same Amazon Machine Image. The same EC2 will come up clean 50% of the time the next time it boots. Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going on here? Thx java.lang.ExceptionInInitializerError at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540) at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190) at java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCip her.jav a:509) at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498) at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81) at java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65) at java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCiphe rSuit es(SSLContextImpl.java:348) at java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext. (SSLC ontextImpl.java:580) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) ... at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java: 948) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java: 198) at org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection ( DriverConnectionFactory.java:52) at org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po olableConnectionFactory.java:415) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactor y (BasicDataSource.java:111) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection Factory(BasicDataSource.java:649) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa taSource.java:532) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS ource.java:731) at jwm.db.DBData.getConnection(DBData.java:506) my call to get a db connection from connection pool ... Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited at java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java: 326) at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111) at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.crypto.Jc
Re: Crypto Randomly Not Getting Initialized
Reg, On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote: Hey Jerry, Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited Does this help? https://stackoverflow.com/a/58183460/ Yours, Reg The stackoverflow link provides a lot of useful info about the error. But unless something in the boot process deletes those files and recreates them later, they are all there and correct. One boot works fine. The next boot throws this error. The next boot is fine. Is there a possibility that permissions get altered on this 'unlimited' folder during boot? I can't think of many reasons it would say a folder doesn't exist that clearly does exist. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: Crypto Randomly Not Getting Initialized
Hello Jerry, > -Ursprüngliche Nachricht- > Von: Jerry Malcolm > Gesendet: Dienstag, 13. Juni 2023 17:35 > An: users@tomcat.apache.org > Betreff: Re: AW: Crypto Randomly Not Getting Initialized > > Hi Thomas, > > On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > Hello Jerry, > > > >> -Ursprüngliche Nachricht- > >> Von: Jerry Malcolm > >> Gesendet: Dienstag, 13. Juni 2023 08:50 > >> An: users@tomcat.apache.org > >> Betreff: Crypto Randomly Not Getting Initialized > >> > >> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon > >> Linux2 in a production environment. A couple of years ago, we > >> started getting weird errors that the "Crypto Mechanism" failed to > >> initialize. Through a lot of trial and error, and reasons I don't > >> quite remember, we put a 2-min delay in rc.local before starting > >> Tomcat, and the problem went away. I'm not a Linux nor a crypto > >> guru. But we traced it to some crypto file that we assumed was not > >> available until later in the Linux boot sequence. Anyway, the 2 > >> minute delay made it go away, for over two years. Then all of a > >> sudden in the last day or so, it's back with a vengeance. It fails > >> with the same crypto error from 2 years ago in about 50% of the EC2 boot > ups. I tried bumping the wait to 3 min, and no change. > >> > >> I need help. Our whole production environment is unstable now since > >> every time an ASG brings a new instance online, I've got a 50-50 > >> chance that tomcat is going to die (and the health check doesn't > >> catch it, but that's a different issue). > >> > >> There are no errors in the Tomcat boot sequence logs. But the first > >> time and every subsequent time I try to get a connection from the > >> DataSource pool, I get the stack dump shown below. > >> > >> I figure it has to be a timing/race condition. But I have no clue > >> what to do to fix it. I'm baffled that it worked for two years, and > >> now fails every other time I start an instance. And every instance > >> is running copies of the exact same Amazon Machine Image. The same > >> EC2 will come up clean 50% of the time the next time it boots. > >> > >> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's > going > >> on here? Thx > >> > >> java.lang.ExceptionInInitializerError > >> at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540) > >> at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190) > >> at > >> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCip > >> her.jav > >> a:509) > >> at > >> java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498) > >> at > >> java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81) > >> at > >> java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65) > >> at > >> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCiphe > >> rSuit > >> es(SSLContextImpl.java:348) > >> at > >> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext. > >> (SSLC > >> ontextImpl.java:580) > >> at java.base/java.lang.Class.forName0(Native Method) > >> at java.base/java.lang.Class.forName(Class.java:315) > >> ... > >> > >> at > >> > com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java: > >> 948) > >> at > >> > com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818) > >> at > com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448) > >> at > >> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) > >> at > >> > com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java: > >> 198) > >> at > >> > org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection > >> ( > >> DriverConnectionFactory.java:52) > >> at > >> > org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po > >> olableConnectionFactory.java:415) > >> at > >> > org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactor > >> y > &g
Re: Crypto Randomly Not Getting Initialized
Simon, On 6/13/2023 2:20 AM, Simon Matter wrote: Hi, I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon Linux2 in a production environment. A couple of years ago, we started getting weird errors that the "Crypto Mechanism" failed to initialize. Through a lot of trial and error, and reasons I don't quite remember, we put a 2-min delay in rc.local before starting Tomcat, and the problem went away. I'm not a Linux nor a crypto guru. But we traced it to some crypto file that we assumed was not available until later in the Linux boot sequence. Anyway, the 2 minute delay made it go away, for over two years. Then all of a sudden in the last day or so, it's back with a vengeance. It fails with the same crypto error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and no change. I need help. Our whole production environment is unstable now since every time an ASG brings a new instance online, I've got a 50-50 chance that tomcat is going to die (and the health check doesn't catch it, but that's a different issue). There are no errors in the Tomcat boot sequence logs. But the first time and every subsequent time I try to get a connection from the DataSource pool, I get the stack dump shown below. I figure it has to be a timing/race condition. But I have no clue what to do to fix it. I'm baffled that it worked for two years, and now fails every other time I start an instance. And every instance is running copies of the exact same Amazon Machine Image. The same EC2 will come up clean 50% of the time the next time it boots. Could it be that your hosts are running out of entropy on boot? Maybe there were RNG related changes in the kernel or rng-tools? Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail, it should not go below 100 or so. Regards, Simon I haven't done any Linux or other system updates in several weeks. I'll look into the entropy possibility. Would running out of entropy cause an exception stating that a crypto directory doesn't exist? I don't know much about Java entropy. Any ideas what would cause entropy to be good on one boot and bad on the next boot? Thx. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: Crypto Randomly Not Getting Initialized
Hi Thomas, On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Jerry, -Ursprüngliche Nachricht- Von: Jerry Malcolm Gesendet: Dienstag, 13. Juni 2023 08:50 An: users@tomcat.apache.org Betreff: Crypto Randomly Not Getting Initialized I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon Linux2 in a production environment. A couple of years ago, we started getting weird errors that the "Crypto Mechanism" failed to initialize. Through a lot of trial and error, and reasons I don't quite remember, we put a 2-min delay in rc.local before starting Tomcat, and the problem went away. I'm not a Linux nor a crypto guru. But we traced it to some crypto file that we assumed was not available until later in the Linux boot sequence. Anyway, the 2 minute delay made it go away, for over two years. Then all of a sudden in the last day or so, it's back with a vengeance. It fails with the same crypto error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and no change. I need help. Our whole production environment is unstable now since every time an ASG brings a new instance online, I've got a 50-50 chance that tomcat is going to die (and the health check doesn't catch it, but that's a different issue). There are no errors in the Tomcat boot sequence logs. But the first time and every subsequent time I try to get a connection from the DataSource pool, I get the stack dump shown below. I figure it has to be a timing/race condition. But I have no clue what to do to fix it. I'm baffled that it worked for two years, and now fails every other time I start an instance. And every instance is running copies of the exact same Amazon Machine Image. The same EC2 will come up clean 50% of the time the next time it boots. Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going on here? Thx java.lang.ExceptionInInitializerError at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540) at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190) at java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.jav a:509) at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498) at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81) at java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65) at java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuit es(SSLContextImpl.java:348) at java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.(SSLC ontextImpl.java:580) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) ... at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java: 948) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java: 198) at org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection( DriverConnectionFactory.java:52) at org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po olableConnectionFactory.java:415) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory (BasicDataSource.java:111) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection Factory(BasicDataSource.java:649) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa taSource.java:532) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS ource.java:731) at jwm.db.DBData.getConnection(DBData.java:506) my call to get a db connection from connection pool ... Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited at java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java: 326) at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111) at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.crypto.JceSecurity.(JceSecurity.java:107) ... 86 more - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Could it be this issue? https://github.com/docker-library/openjdk/issues/101 Maybe you can add i
Re: Crypto Randomly Not Getting Initialized
Hey Jerry, > Caused by: java.lang.SecurityException: Can not initialize cryptographic > mechanism > at > java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo > Caused by: java.lang.SecurityException: Can't read cryptographic policy > directory: unlimited Does this help? https://stackoverflow.com/a/58183460/ Yours, Reg - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Hi, > I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon > Linux2 in a production environment. A couple of years ago, we started > getting weird errors that the "Crypto Mechanism" failed to initialize. > Through a lot of trial and error, and reasons I don't quite remember, we > put a 2-min delay in rc.local before starting Tomcat, and the problem > went away. I'm not a Linux nor a crypto guru. But we traced it to some > crypto file that we assumed was not available until later in the Linux > boot sequence. Anyway, the 2 minute delay made it go away, for over two > years. Then all of a sudden in the last day or so, it's back with a > vengeance. It fails with the same crypto error from 2 years ago in > about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and > no change. > > I need help. Our whole production environment is unstable now since > every time an ASG brings a new instance online, I've got a 50-50 chance > that tomcat is going to die (and the health check doesn't catch it, but > that's a different issue). > > There are no errors in the Tomcat boot sequence logs. But the first > time and every subsequent time I try to get a connection from the > DataSource pool, I get the stack dump shown below. > > I figure it has to be a timing/race condition. But I have no clue what > to do to fix it. I'm baffled that it worked for two years, and now > fails every other time I start an instance. And every instance is > running copies of the exact same Amazon Machine Image. The same EC2 > will come up clean 50% of the time the next time it boots. Could it be that your hosts are running out of entropy on boot? Maybe there were RNG related changes in the kernel or rng-tools? Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail, it should not go below 100 or so. Regards, Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: Crypto Randomly Not Getting Initialized
Hello Jerry, > -Ursprüngliche Nachricht- > Von: Jerry Malcolm > Gesendet: Dienstag, 13. Juni 2023 08:50 > An: users@tomcat.apache.org > Betreff: Crypto Randomly Not Getting Initialized > > I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon > Linux2 in a production environment. A couple of years ago, we started > getting weird errors that the "Crypto Mechanism" failed to initialize. Through > a lot of trial and error, and reasons I don't quite remember, we put a 2-min > delay in rc.local before starting Tomcat, and the problem went away. I'm > not a Linux nor a crypto guru. But we traced it to some crypto file that we > assumed was not available until later in the Linux boot sequence. Anyway, > the 2 minute delay made it go away, for over two years. Then all of a sudden > in the last day or so, it's back with a vengeance. It fails with the same > crypto > error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the > wait to 3 min, and no change. > > I need help. Our whole production environment is unstable now since every > time an ASG brings a new instance online, I've got a 50-50 chance that > tomcat is going to die (and the health check doesn't catch it, but that's a > different issue). > > There are no errors in the Tomcat boot sequence logs. But the first time and > every subsequent time I try to get a connection from the DataSource pool, I > get the stack dump shown below. > > I figure it has to be a timing/race condition. But I have no clue what to do > to > fix it. I'm baffled that it worked for two years, and now fails every other > time I start an instance. And every instance is running copies of the exact > same Amazon Machine Image. The same EC2 will come up clean 50% of the > time the next time it boots. > > Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going > on here? Thx > > java.lang.ExceptionInInitializerError > at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540) > at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190) > at > java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.jav > a:509) > at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498) > at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81) > at > java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65) > at > java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuit > es(SSLContextImpl.java:348) > at > java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.(SSLC > ontextImpl.java:580) > at java.base/java.lang.Class.forName0(Native Method) > at java.base/java.lang.Class.forName(Class.java:315) > ... > > at > com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java: > 948) > at > com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818) > at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448) > at > com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) > at > com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java: > 198) > at > org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection( > DriverConnectionFactory.java:52) > at > org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po > olableConnectionFactory.java:415) > at > org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory > (BasicDataSource.java:111) > at > org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection > Factory(BasicDataSource.java:649) > at > org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa > taSource.java:532) > at > org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS > ource.java:731) > at jwm.db.DBData.getConnection(DBData.java:506) my call to get > a db connection from connection pool > > ... > > Caused by: java.lang.SecurityException: Can not initialize cryptographic > mechanism > at > java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo > Caused by: java.lang.SecurityException: Can't read cryptographic policy > directory: unlimited > at > java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java: > 326) > at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111) > at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108) > at java.base/java.security.AccessController.doPrivi
Crypto Randomly Not Getting Initialized
I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon Linux2 in a production environment. A couple of years ago, we started getting weird errors that the "Crypto Mechanism" failed to initialize. Through a lot of trial and error, and reasons I don't quite remember, we put a 2-min delay in rc.local before starting Tomcat, and the problem went away. I'm not a Linux nor a crypto guru. But we traced it to some crypto file that we assumed was not available until later in the Linux boot sequence. Anyway, the 2 minute delay made it go away, for over two years. Then all of a sudden in the last day or so, it's back with a vengeance. It fails with the same crypto error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and no change. I need help. Our whole production environment is unstable now since every time an ASG brings a new instance online, I've got a 50-50 chance that tomcat is going to die (and the health check doesn't catch it, but that's a different issue). There are no errors in the Tomcat boot sequence logs. But the first time and every subsequent time I try to get a connection from the DataSource pool, I get the stack dump shown below. I figure it has to be a timing/race condition. But I have no clue what to do to fix it. I'm baffled that it worked for two years, and now fails every other time I start an instance. And every instance is running copies of the exact same Amazon Machine Image. The same EC2 will come up clean 50% of the time the next time it boots. Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going on here? Thx java.lang.ExceptionInInitializerError at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540) at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190) at java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.java:509) at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498) at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81) at java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65) at java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuites(SSLContextImpl.java:348) at java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.(SSLContextImpl.java:580) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) ... at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:948) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) at org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:415) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:111) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:532) at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731) at jwm.db.DBData.getConnection(DBData.java:506) my call to get a db connection from connection pool ... Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited at java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326) at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111) at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.crypto.JceSecurity.(JceSecurity.java:107) ... 86 more - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
