Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Simon, On 6/21/23 03:19, Simon Matter wrote: Jerry, On 6/15/23 00:41, Jerry Malcolm wrote: On 6/13/2023 3:46 PM, Jerry Malcolm wrote: On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. One observation even though the error doesn't show up in the log until my first db call, I think the real error has to be occurring during tomcat boot. If the failure occurs, I continue to get the same "Can't read cryptographic policy directory: unlimited" even hours later after boot every time I send another request to TC. If I then reboot TC and this time I don't get the failure, that unlimited folder magically becomes available. There is something happening in TC boot with some sort of crypto initialization that either succeeds or fails. And the exception when trying to get connections long after TC boot is just hitting whatever problem occurred during boot. In other words, delaying my calls, even for hours, has no effect on accessing that folder if the problem is present. Rebooting TC gives it another chance to succeed or fail. Any ideas what TC could be doing on boot that could lock up that folder? I never really found a definitive fix for this. But after all of the research and logging and everything, I pretty much concluded that there's some kind of timing/race condition between Tomcat boot and Java crypto initialization. Given that, I figured it was highly unlikely that two different Java JVM implementations would have the same precise implementation code to trigger that race condition. So as a last ditch effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java JVM. It's circumstantial evidence at best, and running a production environment on fixes that just went away goes against my grain. But if the problem goes away, maybe it won't come back. At this time, when using Corretto JVM, I have not encountered the Crypto directory error. It's been running on all server instances now for almost 24 hours with no problems. So I guess my suggestion to anyone else that might encounter something like this, trying using a different JVM from a different provider. That appears at this time to have worked for me. Yeah, that's certainly a very unsatisfying result, but ... you can't argue with results. I'd love to hear if you end up with any further information. -chris I was asking this before and ask again, how do we know it's not a missing RNG entropy while the crypto stuff of the Java VM is initialised? Because the symptoms of *that* problem (very long lags when requesting random numbers) are not being experienced. And also because the error message is crystal clear: [...] Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo Caused by: java.lang.SecurityException: Can't read cryptographic policy directory: unlimited at java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326) [...] The cryptographic provider itself cannot even initialize. It's *way* before any randomness is needed. The "unlimited" directory is presumably the directory which contains the policy files which do not impose the archaic US-imposed "export restrictions" on cryptographic primitives. The Provider cannot find the configuration directory, and fails. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
> Jerry, > > On 6/15/23 00:41, Jerry Malcolm wrote: >> >> On 6/13/2023 3:46 PM, Jerry Malcolm wrote: >>> >>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: > In /etc/rc.local I have: >> >> -- >> sleep 120s >> systemctl start tomcat9 >> >> - >> >> I put the sleep in back a couple of years ago that for some reason >> 'fixed' this same random, intermittent crypto file exception. >> >> > >>> One observation even though the error doesn't show up in the log >>> until my first db call, I think the real error has to be occurring >>> during tomcat boot. If the failure occurs, I continue to get the same >>> "Can't read cryptographic policy directory: unlimited" even hours >>> later after boot every time I send another request to TC. If I then >>> reboot TC and this time I don't get the failure, that unlimited folder >>> magically becomes available. There is something happening in TC boot >>> with some sort of crypto initialization that either succeeds or >>> fails. And the exception when trying to get connections long after TC >>> boot is just hitting whatever problem occurred during boot. In other >>> words, delaying my calls, even for hours, has no effect on accessing >>> that folder if the problem is present. Rebooting TC gives it another >>> chance to succeed or fail. Any ideas what TC could be doing on boot >>> that could lock up that folder? >>> >>> >> I never really found a definitive fix for this. But after all of the >> research and logging and everything, I pretty much concluded that >> there's some kind of timing/race condition between Tomcat boot and Java >> crypto initialization. Given that, I figured it was highly unlikely >> that two different Java JVM implementations would have the same precise >> implementation code to trigger that race condition. So as a last ditch >> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java >> JVM. It's circumstantial evidence at best, and running a production >> environment on fixes that just went away goes against my grain. But if >> the problem goes away, maybe it won't come back. At this time, when >> using Corretto JVM, I have not encountered the Crypto directory error. >> It's been running on all server instances now for almost 24 hours with >> no problems. So I guess my suggestion to anyone else that might >> encounter something like this, trying using a different JVM from a >> different provider. That appears at this time to have worked for me. > > Yeah, that's certainly a very unsatisfying result, but ... you can't > argue with results. I'd love to hear if you end up with any further > information. > > -chris I was asking this before and ask again, how do we know it's not a missing RNG entropy while the crypto stuff of the Java VM is initialised? Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Jerry, On 6/15/23 00:41, Jerry Malcolm wrote: On 6/13/2023 3:46 PM, Jerry Malcolm wrote: On 6/13/2023 12:39 PM, Jerry Malcolm wrote: Rob, On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: -- sleep 120s systemctl start tomcat9 - I put the sleep in back a couple of years ago that for some reason 'fixed' this same random, intermittent crypto file exception. One observation even though the error doesn't show up in the log until my first db call, I think the real error has to be occurring during tomcat boot. If the failure occurs, I continue to get the same "Can't read cryptographic policy directory: unlimited" even hours later after boot every time I send another request to TC. If I then reboot TC and this time I don't get the failure, that unlimited folder magically becomes available. There is something happening in TC boot with some sort of crypto initialization that either succeeds or fails. And the exception when trying to get connections long after TC boot is just hitting whatever problem occurred during boot. In other words, delaying my calls, even for hours, has no effect on accessing that folder if the problem is present. Rebooting TC gives it another chance to succeed or fail. Any ideas what TC could be doing on boot that could lock up that folder? I never really found a definitive fix for this. But after all of the research and logging and everything, I pretty much concluded that there's some kind of timing/race condition between Tomcat boot and Java crypto initialization. Given that, I figured it was highly unlikely that two different Java JVM implementations would have the same precise implementation code to trigger that race condition. So as a last ditch effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java JVM. It's circumstantial evidence at best, and running a production environment on fixes that just went away goes against my grain. But if the problem goes away, maybe it won't come back. At this time, when using Corretto JVM, I have not encountered the Crypto directory error. It's been running on all server instances now for almost 24 hours with no problems. So I guess my suggestion to anyone else that might encounter something like this, trying using a different JVM from a different provider. That appears at this time to have worked for me. Yeah, that's certainly a very unsatisfying result, but ... you can't argue with results. I'd love to hear if you end up with any further information. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
> > On 6/13/2023 3:46 PM, Jerry Malcolm wrote: >> >> On 6/13/2023 12:39 PM, Jerry Malcolm wrote: >>> Rob, >>> >>> On 6/13/2023 11:34 AM, Rob Sargent wrote: In /etc/rc.local I have: > > -- > sleep 120s > systemctl start tomcat9 > > - > > I put the sleep in back a couple of years ago that for some reason > 'fixed' this same random, intermittent crypto file exception. > > >> One observation even though the error doesn't show up in the log >> until my first db call, I think the real error has to be occurring >> during tomcat boot. If the failure occurs, I continue to get the same >> "Can't read cryptographic policy directory: unlimited" even hours >> later after boot every time I send another request to TC. If I then >> reboot TC and this time I don't get the failure, that unlimited folder >> magically becomes available. There is something happening in TC boot >> with some sort of crypto initialization that either succeeds or >> fails. And the exception when trying to get connections long after TC >> boot is just hitting whatever problem occurred during boot. In other >> words, delaying my calls, even for hours, has no effect on accessing >> that folder if the problem is present. Rebooting TC gives it another >> chance to succeed or fail. Any ideas what TC could be doing on boot >> that could lock up that folder? >> >> > I never really found a definitive fix for this. But after all of the > research and logging and everything, I pretty much concluded that > there's some kind of timing/race condition between Tomcat boot and Java > crypto initialization. Given that, I figured it was highly unlikely > that two different Java JVM implementations would have the same precise > implementation code to trigger that race condition. So as a last ditch > effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java > JVM. It's circumstantial evidence at best, and running a production > environment on fixes that just went away goes against my grain. But if > the problem goes away, maybe it won't come back. At this time, when > using Corretto JVM, I have not encountered the Crypto directory error. > It's been running on all server instances now for almost 24 hours with > no problems. So I guess my suggestion to anyone else that might > encounter something like this, trying using a different JVM from a > different provider. That appears at this time to have worked for me. > Maybe you could report this as a bug in the affected JVM? If it's really coming from the JVM then it would be nice to have it fixed. Regards, Simon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
