Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-21 Thread Christopher Schultz

Simon,

On 6/21/23 03:19, Simon Matter wrote:

Jerry,

On 6/15/23 00:41, Jerry Malcolm wrote:


On 6/13/2023 3:46 PM, Jerry Malcolm wrote:


On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log
until my first db call, I think the real error has to be occurring
during tomcat boot.  If the failure occurs, I continue to get the same
"Can't read cryptographic policy directory: unlimited" even hours
later after boot every time I send another request to TC.  If I then
reboot TC and this time I don't get the failure, that unlimited folder
magically becomes available.  There is something happening in TC boot
with some sort of crypto initialization that either succeeds or
fails.  And the exception when trying to get connections long after TC
boot is just hitting whatever problem occurred during boot.  In other
words, delaying my calls, even for hours, has no effect on accessing
that folder if the problem is present.  Rebooting TC gives it another
chance to succeed or fail. Any ideas what TC could be doing on boot
that could lock up that folder?



I never really found a definitive fix for this.  But after all of the
research and logging and everything, I pretty much concluded that
there's some kind of timing/race condition between Tomcat boot and Java
crypto initialization.  Given that, I figured it was highly unlikely
that two different Java JVM implementations would have the same precise
implementation code to trigger that race condition.  So as a last ditch
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
JVM.  It's circumstantial evidence at best, and running a production
environment on fixes that just went away goes against my grain. But if
the problem goes away, maybe it won't come back. At this time, when
using Corretto JVM, I have not encountered the Crypto directory error.
It's been running on all server instances now for almost 24 hours with
no problems.  So I guess my suggestion to anyone else that might
encounter something like this, trying using a different JVM from a
different provider.  That appears at this time to have worked for me.


Yeah, that's certainly a very unsatisfying result, but ... you can't
argue with results. I'd love to hear if you end up with any further
information.

-chris


I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?


Because the symptoms of *that* problem (very long lags when requesting 
random numbers) are not being experienced. And also because the error 
message is crystal clear:


[...]
Caused by: java.lang.SecurityException: Can not initialize cryptographic 
mechanism
at 
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy 
directory: unlimited
at 
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)

[...]

The cryptographic provider itself cannot even initialize. It's *way* 
before any randomness is needed. The "unlimited" directory is presumably 
the directory which contains the policy files which do not impose the 
archaic US-imposed "export restrictions" on cryptographic primitives. 
The Provider cannot find the configuration directory, and fails.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-21 Thread Simon Matter
> Jerry,
>
> On 6/15/23 00:41, Jerry Malcolm wrote:
>>
>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
 Rob,

 On 6/13/2023 11:34 AM, Rob Sargent wrote:
> In /etc/rc.local I have:
>>
>> --
>> sleep 120s
>> systemctl start tomcat9
>>
>> -
>>
>> I put the sleep in back a couple of years ago that for some reason
>> 'fixed' this same random, intermittent crypto file exception.
>>
>>
>
>>> One observation even though the error doesn't show up in the log
>>> until my first db call, I think the real error has to be occurring
>>> during tomcat boot.  If the failure occurs, I continue to get the same
>>> "Can't read cryptographic policy directory: unlimited" even hours
>>> later after boot every time I send another request to TC.  If I then
>>> reboot TC and this time I don't get the failure, that unlimited folder
>>> magically becomes available.  There is something happening in TC boot
>>> with some sort of crypto initialization that either succeeds or
>>> fails.  And the exception when trying to get connections long after TC
>>> boot is just hitting whatever problem occurred during boot.  In other
>>> words, delaying my calls, even for hours, has no effect on accessing
>>> that folder if the problem is present.  Rebooting TC gives it another
>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>> that could lock up that folder?
>>>
>>>
>> I never really found a definitive fix for this.  But after all of the
>> research and logging and everything, I pretty much concluded that
>> there's some kind of timing/race condition between Tomcat boot and Java
>> crypto initialization.  Given that, I figured it was highly unlikely
>> that two different Java JVM implementations would have the same precise
>> implementation code to trigger that race condition.  So as a last ditch
>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>> JVM.  It's circumstantial evidence at best, and running a production
>> environment on fixes that just went away goes against my grain. But if
>> the problem goes away, maybe it won't come back. At this time, when
>> using Corretto JVM, I have not encountered the Crypto directory error.
>> It's been running on all server instances now for almost 24 hours with
>> no problems.  So I guess my suggestion to anyone else that might
>> encounter something like this, trying using a different JVM from a
>> different provider.  That appears at this time to have worked for me.
>
> Yeah, that's certainly a very unsatisfying result, but ... you can't
> argue with results. I'd love to hear if you end up with any further
> information.
>
> -chris

I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?

Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-20 Thread Christopher Schultz

Jerry,

On 6/15/23 00:41, Jerry Malcolm wrote:


On 6/13/2023 3:46 PM, Jerry Malcolm wrote:


On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log 
until my first db call, I think the real error has to be occurring 
during tomcat boot.  If the failure occurs, I continue to get the same 
"Can't read cryptographic policy directory: unlimited" even hours 
later after boot every time I send another request to TC.  If I then 
reboot TC and this time I don't get the failure, that unlimited folder 
magically becomes available.  There is something happening in TC boot 
with some sort of crypto initialization that either succeeds or 
fails.  And the exception when trying to get connections long after TC 
boot is just hitting whatever problem occurred during boot.  In other 
words, delaying my calls, even for hours, has no effect on accessing 
that folder if the problem is present.  Rebooting TC gives it another 
chance to succeed or fail. Any ideas what TC could be doing on boot 
that could lock up that folder?



I never really found a definitive fix for this.  But after all of the 
research and logging and everything, I pretty much concluded that 
there's some kind of timing/race condition between Tomcat boot and Java 
crypto initialization.  Given that, I figured it was highly unlikely 
that two different Java JVM implementations would have the same precise 
implementation code to trigger that race condition.  So as a last ditch 
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java 
JVM.  It's circumstantial evidence at best, and running a production 
environment on fixes that just went away goes against my grain. But if 
the problem goes away, maybe it won't come back. At this time, when 
using Corretto JVM, I have not encountered the Crypto directory error. 
It's been running on all server instances now for almost 24 hours with 
no problems.  So I guess my suggestion to anyone else that might 
encounter something like this, trying using a different JVM from a 
different provider.  That appears at this time to have worked for me.


Yeah, that's certainly a very unsatisfying result, but ... you can't 
argue with results. I'd love to hear if you end up with any further 
information.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-15 Thread Simon Matter
>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
 In /etc/rc.local I have:
>
> --
> sleep 120s
> systemctl start tomcat9
>
> -
>
> I put the sleep in back a couple of years ago that for some reason
> 'fixed' this same random, intermittent crypto file exception.
>
>

>> One observation even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot.  If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC.  If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available.  There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails.  And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot.  In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present.  Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this.  But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization.  Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition.  So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM.  It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error. 
> It's been running on all server instances now for almost 24 hours with
> no problems.  So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider.  That appears at this time to have worked for me.
>

Maybe you could report this as a bug in the affected JVM? If it's really
coming from the JVM then it would be nice to have it fixed.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-14 Thread Jerry Malcolm



On 6/13/2023 3:46 PM, Jerry Malcolm wrote:


On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log 
until my first db call, I think the real error has to be occurring 
during tomcat boot.  If the failure occurs, I continue to get the same 
"Can't read cryptographic policy directory: unlimited" even hours 
later after boot every time I send another request to TC.  If I then 
reboot TC and this time I don't get the failure, that unlimited folder 
magically becomes available.  There is something happening in TC boot 
with some sort of crypto initialization that either succeeds or 
fails.  And the exception when trying to get connections long after TC 
boot is just hitting whatever problem occurred during boot.  In other 
words, delaying my calls, even for hours, has no effect on accessing 
that folder if the problem is present.  Rebooting TC gives it another 
chance to succeed or fail. Any ideas what TC could be doing on boot 
that could lock up that folder?



I never really found a definitive fix for this.  But after all of the 
research and logging and everything, I pretty much concluded that 
there's some kind of timing/race condition between Tomcat boot and Java 
crypto initialization.  Given that, I figured it was highly unlikely 
that two different Java JVM implementations would have the same precise 
implementation code to trigger that race condition.  So as a last ditch 
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java 
JVM.  It's circumstantial evidence at best, and running a production 
environment on fixes that just went away goes against my grain. But if 
the problem goes away, maybe it won't come back. At this time, when 
using Corretto JVM, I have not encountered the Crypto directory error.  
It's been running on all server instances now for almost 24 hours with 
no problems.  So I guess my suggestion to anyone else that might 
encounter something like this, trying using a different JVM from a 
different provider.  That appears at this time to have worked for me.


Thanks for all the support.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org