Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-21 Thread Christopher Schultz

Simon,

On 6/21/23 03:19, Simon Matter wrote:

Jerry,

On 6/15/23 00:41, Jerry Malcolm wrote:


On 6/13/2023 3:46 PM, Jerry Malcolm wrote:


On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log
until my first db call, I think the real error has to be occurring
during tomcat boot.  If the failure occurs, I continue to get the same
"Can't read cryptographic policy directory: unlimited" even hours
later after boot every time I send another request to TC.  If I then
reboot TC and this time I don't get the failure, that unlimited folder
magically becomes available.  There is something happening in TC boot
with some sort of crypto initialization that either succeeds or
fails.  And the exception when trying to get connections long after TC
boot is just hitting whatever problem occurred during boot.  In other
words, delaying my calls, even for hours, has no effect on accessing
that folder if the problem is present.  Rebooting TC gives it another
chance to succeed or fail. Any ideas what TC could be doing on boot
that could lock up that folder?



I never really found a definitive fix for this.  But after all of the
research and logging and everything, I pretty much concluded that
there's some kind of timing/race condition between Tomcat boot and Java
crypto initialization.  Given that, I figured it was highly unlikely
that two different Java JVM implementations would have the same precise
implementation code to trigger that race condition.  So as a last ditch
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
JVM.  It's circumstantial evidence at best, and running a production
environment on fixes that just went away goes against my grain. But if
the problem goes away, maybe it won't come back. At this time, when
using Corretto JVM, I have not encountered the Crypto directory error.
It's been running on all server instances now for almost 24 hours with
no problems.  So I guess my suggestion to anyone else that might
encounter something like this, trying using a different JVM from a
different provider.  That appears at this time to have worked for me.


Yeah, that's certainly a very unsatisfying result, but ... you can't
argue with results. I'd love to hear if you end up with any further
information.

-chris


I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?


Because the symptoms of *that* problem (very long lags when requesting 
random numbers) are not being experienced. And also because the error 
message is crystal clear:


[...]
Caused by: java.lang.SecurityException: Can not initialize cryptographic 
mechanism
at 
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy 
directory: unlimited
at 
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)

[...]

The cryptographic provider itself cannot even initialize. It's *way* 
before any randomness is needed. The "unlimited" directory is presumably 
the directory which contains the policy files which do not impose the 
archaic US-imposed "export restrictions" on cryptographic primitives. 
The Provider cannot find the configuration directory, and fails.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-21 Thread Simon Matter
> Jerry,
>
> On 6/15/23 00:41, Jerry Malcolm wrote:
>>
>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
 Rob,

 On 6/13/2023 11:34 AM, Rob Sargent wrote:
> In /etc/rc.local I have:
>>
>> --
>> sleep 120s
>> systemctl start tomcat9
>>
>> -
>>
>> I put the sleep in back a couple of years ago that for some reason
>> 'fixed' this same random, intermittent crypto file exception.
>>
>>
>
>>> One observation even though the error doesn't show up in the log
>>> until my first db call, I think the real error has to be occurring
>>> during tomcat boot.  If the failure occurs, I continue to get the same
>>> "Can't read cryptographic policy directory: unlimited" even hours
>>> later after boot every time I send another request to TC.  If I then
>>> reboot TC and this time I don't get the failure, that unlimited folder
>>> magically becomes available.  There is something happening in TC boot
>>> with some sort of crypto initialization that either succeeds or
>>> fails.  And the exception when trying to get connections long after TC
>>> boot is just hitting whatever problem occurred during boot.  In other
>>> words, delaying my calls, even for hours, has no effect on accessing
>>> that folder if the problem is present.  Rebooting TC gives it another
>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>> that could lock up that folder?
>>>
>>>
>> I never really found a definitive fix for this.  But after all of the
>> research and logging and everything, I pretty much concluded that
>> there's some kind of timing/race condition between Tomcat boot and Java
>> crypto initialization.  Given that, I figured it was highly unlikely
>> that two different Java JVM implementations would have the same precise
>> implementation code to trigger that race condition.  So as a last ditch
>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>> JVM.  It's circumstantial evidence at best, and running a production
>> environment on fixes that just went away goes against my grain. But if
>> the problem goes away, maybe it won't come back. At this time, when
>> using Corretto JVM, I have not encountered the Crypto directory error.
>> It's been running on all server instances now for almost 24 hours with
>> no problems.  So I guess my suggestion to anyone else that might
>> encounter something like this, trying using a different JVM from a
>> different provider.  That appears at this time to have worked for me.
>
> Yeah, that's certainly a very unsatisfying result, but ... you can't
> argue with results. I'd love to hear if you end up with any further
> information.
>
> -chris

I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?

Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-20 Thread Christopher Schultz

Jerry,

On 6/15/23 00:41, Jerry Malcolm wrote:


On 6/13/2023 3:46 PM, Jerry Malcolm wrote:


On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log 
until my first db call, I think the real error has to be occurring 
during tomcat boot.  If the failure occurs, I continue to get the same 
"Can't read cryptographic policy directory: unlimited" even hours 
later after boot every time I send another request to TC.  If I then 
reboot TC and this time I don't get the failure, that unlimited folder 
magically becomes available.  There is something happening in TC boot 
with some sort of crypto initialization that either succeeds or 
fails.  And the exception when trying to get connections long after TC 
boot is just hitting whatever problem occurred during boot.  In other 
words, delaying my calls, even for hours, has no effect on accessing 
that folder if the problem is present.  Rebooting TC gives it another 
chance to succeed or fail. Any ideas what TC could be doing on boot 
that could lock up that folder?



I never really found a definitive fix for this.  But after all of the 
research and logging and everything, I pretty much concluded that 
there's some kind of timing/race condition between Tomcat boot and Java 
crypto initialization.  Given that, I figured it was highly unlikely 
that two different Java JVM implementations would have the same precise 
implementation code to trigger that race condition.  So as a last ditch 
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java 
JVM.  It's circumstantial evidence at best, and running a production 
environment on fixes that just went away goes against my grain. But if 
the problem goes away, maybe it won't come back. At this time, when 
using Corretto JVM, I have not encountered the Crypto directory error. 
It's been running on all server instances now for almost 24 hours with 
no problems.  So I guess my suggestion to anyone else that might 
encounter something like this, trying using a different JVM from a 
different provider.  That appears at this time to have worked for me.


Yeah, that's certainly a very unsatisfying result, but ... you can't 
argue with results. I'd love to hear if you end up with any further 
information.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-15 Thread Simon Matter
>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
 In /etc/rc.local I have:
>
> --
> sleep 120s
> systemctl start tomcat9
>
> -
>
> I put the sleep in back a couple of years ago that for some reason
> 'fixed' this same random, intermittent crypto file exception.
>
>

>> One observation even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot.  If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC.  If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available.  There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails.  And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot.  In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present.  Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this.  But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization.  Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition.  So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM.  It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error. 
> It's been running on all server instances now for almost 24 hours with
> no problems.  So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider.  That appears at this time to have worked for me.
>

Maybe you could report this as a bug in the affected JVM? If it's really
coming from the JVM then it would be nice to have it fixed.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Conclusion - Re: Crypto Randomly Not Getting Initialized

2023-06-14 Thread Jerry Malcolm



On 6/13/2023 3:46 PM, Jerry Malcolm wrote:


On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log 
until my first db call, I think the real error has to be occurring 
during tomcat boot.  If the failure occurs, I continue to get the same 
"Can't read cryptographic policy directory: unlimited" even hours 
later after boot every time I send another request to TC.  If I then 
reboot TC and this time I don't get the failure, that unlimited folder 
magically becomes available.  There is something happening in TC boot 
with some sort of crypto initialization that either succeeds or 
fails.  And the exception when trying to get connections long after TC 
boot is just hitting whatever problem occurred during boot.  In other 
words, delaying my calls, even for hours, has no effect on accessing 
that folder if the problem is present.  Rebooting TC gives it another 
chance to succeed or fail. Any ideas what TC could be doing on boot 
that could lock up that folder?



I never really found a definitive fix for this.  But after all of the 
research and logging and everything, I pretty much concluded that 
there's some kind of timing/race condition between Tomcat boot and Java 
crypto initialization.  Given that, I figured it was highly unlikely 
that two different Java JVM implementations would have the same precise 
implementation code to trigger that race condition.  So as a last ditch 
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java 
JVM.  It's circumstantial evidence at best, and running a production 
environment on fixes that just went away goes against my grain. But if 
the problem goes away, maybe it won't come back. At this time, when 
using Corretto JVM, I have not encountered the Crypto directory error.  
It's been running on all server instances now for almost 24 hours with 
no problems.  So I guess my suggestion to anyone else that might 
encounter something like this, trying using a different JVM from a 
different provider.  That appears at this time to have worked for me.


Thanks for all the support.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-14 Thread Christopher Schultz

Jerry,

On 6/13/23 12:25, Jerry Malcolm wrote:

Chris,

On 6/13/2023 11:03 AM, Christopher Schultz wrote:

Jerry,

On 6/13/23 11:51, Jerry Malcolm wrote:

Reg,

On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote:

Hey Jerry,

Caused by: java.lang.SecurityException: Can not initialize 
cryptographic

mechanism
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) 
... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic 
policy

directory: unlimited

Does this help?
https://stackoverflow.com/a/58183460/

Yours,
Reg
The stackoverflow link provides a lot of useful info about the error. 
But unless something in the boot process deletes those files and 
recreates them later, they are all there and correct. One boot works 
fine.  The next boot throws this error.  The next boot is fine.  Is 
there a possibility that permissions get altered on this 'unlimited' 
folder during boot?  I can't think of many reasons it would say a 
folder doesn't exist that clearly does exist.


How are you launching Tomcat on boot?

Are you able to add some instrumentation/logging to that?

-chris


In /etc/rc.local I have:

--
sleep 120s
systemctl start tomcat9


That's an odd way to do things (in rc.local).

If you have systemctl, you are using systemd. If you are using systemd, 
you isn't it auto-starting that service?


$ sudo systemctl enable tomcat9

That ought to do it, and the tomcat9 service should start on startup, 
resolving any dependencies it needs.


Are you able to do an "ls [path to policy files] > /tmp/some_log_file" 
in there to see the situation when the service is trying to start?


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm



On 6/13/2023 12:39 PM, Jerry Malcolm wrote:

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





One observation even though the error doesn't show up in the log 
until my first db call, I think the real error has to be occurring 
during tomcat boot.  If the failure occurs, I continue to get the same 
"Can't read cryptographic policy directory: unlimited" even hours later 
after boot every time I send another request to TC.  If I then reboot TC 
and this time I don't get the failure, that unlimited folder magically 
becomes available.  There is something happening in TC boot with some 
sort of crypto initialization that either succeeds or fails.  And the 
exception when trying to get connections long after TC boot is just 
hitting whatever problem occurred during boot.  In other words, delaying 
my calls, even for hours, has no effect on accessing that folder if the 
problem is present.  Rebooting TC gives it another chance to succeed or 
fail. Any ideas what TC could be doing on boot that could lock up that 
folder?


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

Rob,

On 6/13/2023 11:34 AM, Rob Sargent wrote:

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





You mentioned earlier that you had recently tried a 3 minute delay, to 
no effect.  Did you try any delay longer than 3 minutes? Maybe 
something ridiculous like 15m?  It could be that whatever was taking 
two minutes before is now taking much longer.


No change in file permissions at or above the "unlimited" dir?


I put the time back to 2 min after the 3 min had no effect.  But after 
the failure occurred on an instance, and I got to all of the logs, I 
just did a tomcat reboot from the command line and it worked.  So in 
that case 2-3 min failed.  But several hours later it worked.  Still not 
what could cause a "missing" folder to finally become available after 
several minutes (or hours).  Definitely not doing anything with 
permissions on that file.  Didn't even know it existed until this 
problem surfaced.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Simon Matter
Hi,

> Jerry,
>
> On 6/13/23 11:42, Jerry Malcolm wrote:
>> Simon,
>>
>> On 6/13/2023 2:20 AM, Simon Matter wrote:
>>> Hi,
>>>
 I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
 Linux2 in a production environment.  A couple of years ago, we started
 getting weird errors that the "Crypto Mechanism" failed to initialize.
 Through a lot of trial and error, and reasons I don't quite remember,
 we
 put a 2-min delay in rc.local before starting Tomcat, and the problem
 went away.  I'm not a Linux nor a crypto guru.  But we traced it to
 some
 crypto file that we assumed was not available until later in the Linux
 boot sequence.  Anyway, the 2 minute delay made it go away, for over
 two
 years.  Then all of a sudden in the last day or so, it's back with a
 vengeance.  It fails with the same crypto error from 2 years ago in
 about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and
 no change.

 I need help.  Our whole production environment is unstable now since
 every time an ASG brings a new instance online, I've got a 50-50
 chance
 that tomcat is going to die (and the health check doesn't catch it,
 but
 that's a different issue).

 There are no errors in the Tomcat boot sequence logs.  But the first
 time and every subsequent time I try to get a connection from the
 DataSource pool, I get the stack dump shown below.

 I figure it has to be a timing/race condition.  But I have no clue
 what
 to do to fix it.  I'm baffled that it worked for two years, and now
 fails every other time I start an instance.  And every instance is
 running copies of the exact same Amazon Machine Image.  The same EC2
 will come up clean 50% of the time the next time it boots.
>>> Could it be that your hosts are running out of entropy on boot?
>>>
>>> Maybe there were RNG related changes in the kernel or rng-tools?
>>>
>>> Maybe monitor available entropy in
>>> /proc/sys/kernel/random/entropy_avail,
>>> it should not go below 100 or so.
>>>
>>> Regards,
>>> Simon
>>>
>> I haven't done any Linux or other system updates in several weeks. I'll
>> look into the entropy possibility.  Would running out of entropy cause
>> an exception stating that a crypto directory doesn't exist?  I don't
>> know much about Java entropy.  Any ideas what would cause entropy to be
>> good on one boot and bad on the next boot?  Thx.
>
> This isn't about entropy. Focus on the actual error message that the
> system can't find the unlimited policy file for some reason.
>
> -chris

Sorry, I didn't read the errors too carefully :)

I don't know Amazon Linux2 but one question, does it have the directory
/etc/crypto-policies? Apart from this, does systemd somehow fiddling with
crypto stuff while booting? Since systemd parallelizes the whole boot
process many things can (and sometimes do) break and it's difficult to
find it out.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Rob Sargent

In /etc/rc.local I have:


--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.





You mentioned earlier that you had recently tried a 3 minute delay, to 
no effect.  Did you try any delay longer than 3 minutes? Maybe something 
ridiculous like 15m?  It could be that whatever was taking two minutes 
before is now taking much longer.


No change in file permissions at or above the "unlimited" dir?


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

chris

On 6/13/2023 11:03 AM, Christopher Schultz wrote:

Jerry,

On 6/13/23 11:51, Jerry Malcolm wrote:

Reg,

On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote:

Hey Jerry,

Caused by: java.lang.SecurityException: Can not initialize 
cryptographic

mechanism
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) 
... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic 
policy

directory: unlimited

Does this help?
https://stackoverflow.com/a/58183460/

Yours,
Reg
The stackoverflow link provides a lot of useful info about the error. 
But unless something in the boot process deletes those files and 
recreates them later, they are all there and correct. One boot works 
fine.  The next boot throws this error.  The next boot is fine.  Is 
there a possibility that permissions get altered on this 'unlimited' 
folder during boot?  I can't think of many reasons it would say a 
folder doesn't exist that clearly does exist.


How are you launching Tomcat on boot?

Are you able to add some instrumentation/logging to that?

-chris
The Tomcat boot log in catalina.out runs fine.  The exception first 
occurs when a request comes in and I have to go to the database.  It 
happens on getting a connection from the datasource.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

Chris,

On 6/13/2023 11:03 AM, Christopher Schultz wrote:

Jerry,

On 6/13/23 11:51, Jerry Malcolm wrote:

Reg,

On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote:

Hey Jerry,

Caused by: java.lang.SecurityException: Can not initialize 
cryptographic

mechanism
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) 
... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic 
policy

directory: unlimited

Does this help?
https://stackoverflow.com/a/58183460/

Yours,
Reg
The stackoverflow link provides a lot of useful info about the error. 
But unless something in the boot process deletes those files and 
recreates them later, they are all there and correct. One boot works 
fine.  The next boot throws this error.  The next boot is fine.  Is 
there a possibility that permissions get altered on this 'unlimited' 
folder during boot?  I can't think of many reasons it would say a 
folder doesn't exist that clearly does exist.


How are you launching Tomcat on boot?

Are you able to add some instrumentation/logging to that?

-chris


In /etc/rc.local I have:

--
sleep 120s
systemctl start tomcat9

-

I put the sleep in back a couple of years ago that for some reason 
'fixed' this same random, intermittent crypto file exception.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Christopher Schultz

Jerry,

On 6/13/23 11:51, Jerry Malcolm wrote:

Reg,

On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote:

Hey Jerry,


Caused by: java.lang.SecurityException: Can not initialize cryptographic
mechanism
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 
86 mo

Caused by: java.lang.SecurityException: Can't read cryptographic policy
directory: unlimited

Does this help?
https://stackoverflow.com/a/58183460/

Yours,
Reg
The stackoverflow link provides a lot of useful info about the error. 
But unless something in the boot process deletes those files and 
recreates them later, they are all there and correct.  One boot works 
fine.  The next boot throws this error.  The next boot is fine.  Is 
there a possibility that permissions get altered on this 'unlimited' 
folder during boot?  I can't think of many reasons it would say a folder 
doesn't exist that clearly does exist.


How are you launching Tomcat on boot?

Are you able to add some instrumentation/logging to that?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Christopher Schultz

Jerry,

On 6/13/23 11:42, Jerry Malcolm wrote:

Simon,

On 6/13/2023 2:20 AM, Simon Matter wrote:

Hi,


I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
Linux2 in a production environment.  A couple of years ago, we started
getting weird errors that the "Crypto Mechanism" failed to initialize.
Through a lot of trial and error, and reasons I don't quite remember, we
put a 2-min delay in rc.local before starting Tomcat, and the problem
went away.  I'm not a Linux nor a crypto guru.  But we traced it to some
crypto file that we assumed was not available until later in the Linux
boot sequence.  Anyway, the 2 minute delay made it go away, for over two
years.  Then all of a sudden in the last day or so, it's back with a
vengeance.  It fails with the same crypto error from 2 years ago in
about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and
no change.

I need help.  Our whole production environment is unstable now since
every time an ASG brings a new instance online, I've got a 50-50 chance
that tomcat is going to die (and the health check doesn't catch it, but
that's a different issue).

There are no errors in the Tomcat boot sequence logs.  But the first
time and every subsequent time I try to get a connection from the
DataSource pool, I get the stack dump shown below.

I figure it has to be a timing/race condition.  But I have no clue what
to do to fix it.  I'm baffled that it worked for two years, and now
fails every other time I start an instance.  And every instance is
running copies of the exact same Amazon Machine Image.  The same EC2
will come up clean 50% of the time the next time it boots.

Could it be that your hosts are running out of entropy on boot?

Maybe there were RNG related changes in the kernel or rng-tools?

Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
it should not go below 100 or so.

Regards,
Simon

I haven't done any Linux or other system updates in several weeks. I'll 
look into the entropy possibility.  Would running out of entropy cause 
an exception stating that a crypto directory doesn't exist?  I don't 
know much about Java entropy.  Any ideas what would cause entropy to be 
good on one boot and bad on the next boot?  Thx.


This isn't about entropy. Focus on the actual error message that the 
system can't find the unlimited policy file for some reason.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: AW: AW: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

Thomas,

On 6/13/2023 10:45 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Jerry,



-Ursprüngliche Nachricht-
Von: Jerry Malcolm 
Gesendet: Dienstag, 13. Juni 2023 17:35
An: users@tomcat.apache.org
Betreff: Re: AW: Crypto Randomly Not Getting Initialized

Hi Thomas,

On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Jerry,


-Ursprüngliche Nachricht-
Von: Jerry Malcolm 
Gesendet: Dienstag, 13. Juni 2023 08:50
An: users@tomcat.apache.org
Betreff: Crypto Randomly Not Getting Initialized

I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
Linux2 in a production environment.  A couple of years ago, we
started getting weird errors that the "Crypto Mechanism" failed to
initialize. Through a lot of trial and error, and reasons I don't
quite remember, we put a 2-min delay in rc.local before starting
Tomcat, and the problem went away.  I'm not a Linux nor a crypto
guru.  But we traced it to some crypto file that we assumed was not
available until later in the Linux boot sequence.  Anyway, the 2
minute delay made it go away, for over two years.  Then all of a
sudden in the last day or so, it's back with a vengeance.  It fails
with the same crypto error from 2 years ago in about 50% of the EC2 boot

ups.  I tried bumping the wait to 3 min, and no change.

I need help.  Our whole production environment is unstable now since
every time an ASG brings a new instance online, I've got a 50-50
chance that tomcat is going to die (and the health check doesn't
catch it, but that's a different issue).

There are no errors in the Tomcat boot sequence logs.  But the first
time and every subsequent time I try to get a connection from the
DataSource pool, I get the stack dump shown below.

I figure it has to be a timing/race condition.  But I have no clue
what to do to fix it.  I'm baffled that it worked for two years, and
now fails every other time I start an instance.  And every instance
is running copies of the exact same Amazon Machine Image.  The same
EC2 will come up clean 50% of the time the next time it boots.

Can somebody with Tomcat/Crypto/Linux knowledge unravel what's

going

on here?  Thx

java.lang.ExceptionInInitializerError
       at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
       at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
       at
java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCip
her.jav
a:509)
       at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498)
       at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81)
       at
java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65)
       at
java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCiphe
rSuit
es(SSLContextImpl.java:348)
       at
java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.
(SSLC
ontextImpl.java:580)
       at java.base/java.lang.Class.forName0(Native Method)
       at java.base/java.lang.Class.forName(Class.java:315)
...

       at


com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:

948)
       at


com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)

       at

com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)

       at
com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
       at


com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:

198)
       at


org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection

(
DriverConnectionFactory.java:52)
       at


org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po

olableConnectionFactory.java:415)
       at


org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactor

y
(BasicDataSource.java:111)
       at


org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection

Factory(BasicDataSource.java:649)
       at


org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa

taSource.java:532)
       at


org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS

ource.java:731)
       at jwm.db.DBData.getConnection(DBData.java:506)    my
call to get a db connection from connection pool 

...

Caused by: java.lang.SecurityException: Can not initialize
cryptographic mechanism
       at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ...
86 mo Caused by: java.lang.SecurityException: Can't read
cryptographic policy
directory: unlimited
       at


java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:

326)
       at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
       at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
       at
java.base/java.security.AccessController.doPrivileged(Native
Method)
       at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:107)
       .

Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

Reg,

On 6/13/2023 2:57 AM, r.barc...@habmalnefrage.de wrote:

Hey Jerry,


Caused by: java.lang.SecurityException: Can not initialize cryptographic
mechanism
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy
directory: unlimited

Does this help?
https://stackoverflow.com/a/58183460/

Yours,
Reg
The stackoverflow link provides a lot of useful info about the error.  
But unless something in the boot process deletes those files and 
recreates them later, they are all there and correct.  One boot works 
fine.  The next boot throws this error.  The next boot is fine.  Is 
there a possibility that permissions get altered on this 'unlimited' 
folder during boot?  I can't think of many reasons it would say a folder 
doesn't exist that clearly does exist.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



AW: AW: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Thomas Hoffmann (Speed4Trade GmbH)
Hello Jerry,


> -Ursprüngliche Nachricht-
> Von: Jerry Malcolm 
> Gesendet: Dienstag, 13. Juni 2023 17:35
> An: users@tomcat.apache.org
> Betreff: Re: AW: Crypto Randomly Not Getting Initialized
> 
> Hi Thomas,
> 
> On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > Hello Jerry,
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Jerry Malcolm 
> >> Gesendet: Dienstag, 13. Juni 2023 08:50
> >> An: users@tomcat.apache.org
> >> Betreff: Crypto Randomly Not Getting Initialized
> >>
> >> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> >> Linux2 in a production environment.  A couple of years ago, we
> >> started getting weird errors that the "Crypto Mechanism" failed to
> >> initialize. Through a lot of trial and error, and reasons I don't
> >> quite remember, we put a 2-min delay in rc.local before starting
> >> Tomcat, and the problem went away.  I'm not a Linux nor a crypto
> >> guru.  But we traced it to some crypto file that we assumed was not
> >> available until later in the Linux boot sequence.  Anyway, the 2
> >> minute delay made it go away, for over two years.  Then all of a
> >> sudden in the last day or so, it's back with a vengeance.  It fails
> >> with the same crypto error from 2 years ago in about 50% of the EC2 boot
> ups.  I tried bumping the wait to 3 min, and no change.
> >>
> >> I need help.  Our whole production environment is unstable now since
> >> every time an ASG brings a new instance online, I've got a 50-50
> >> chance that tomcat is going to die (and the health check doesn't
> >> catch it, but that's a different issue).
> >>
> >> There are no errors in the Tomcat boot sequence logs.  But the first
> >> time and every subsequent time I try to get a connection from the
> >> DataSource pool, I get the stack dump shown below.
> >>
> >> I figure it has to be a timing/race condition.  But I have no clue
> >> what to do to fix it.  I'm baffled that it worked for two years, and
> >> now fails every other time I start an instance.  And every instance
> >> is running copies of the exact same Amazon Machine Image.  The same
> >> EC2 will come up clean 50% of the time the next time it boots.
> >>
> >> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's
> going
> >> on here?  Thx
> >>
> >> java.lang.ExceptionInInitializerError
> >>       at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
> >>       at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
> >>       at
> >> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCip
> >> her.jav
> >> a:509)
> >>       at 
> >> java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498)
> >>       at 
> >> java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81)
> >>       at
> >> java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65)
> >>       at
> >> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCiphe
> >> rSuit
> >> es(SSLContextImpl.java:348)
> >>       at
> >> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.
> >> (SSLC
> >> ontextImpl.java:580)
> >>       at java.base/java.lang.Class.forName0(Native Method)
> >>       at java.base/java.lang.Class.forName(Class.java:315)
> >> ...
> >>
> >>       at
> >>
> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
> >> 948)
> >>       at
> >>
> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
> >>       at
> com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)
> >>       at
> >> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
> >>       at
> >>
> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
> >> 198)
> >>       at
> >>
> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection
> >> (
> >> DriverConnectionFactory.java:52)
> >>       at
> >>
> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
> >> olableConnectionFactory.java:415)
> >>       at
> >>
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactor
> >> y
> >> (BasicDataSourc

Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

Simon,

On 6/13/2023 2:20 AM, Simon Matter wrote:

Hi,


I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
Linux2 in a production environment.  A couple of years ago, we started
getting weird errors that the "Crypto Mechanism" failed to initialize.
Through a lot of trial and error, and reasons I don't quite remember, we
put a 2-min delay in rc.local before starting Tomcat, and the problem
went away.  I'm not a Linux nor a crypto guru.  But we traced it to some
crypto file that we assumed was not available until later in the Linux
boot sequence.  Anyway, the 2 minute delay made it go away, for over two
years.  Then all of a sudden in the last day or so, it's back with a
vengeance.  It fails with the same crypto error from 2 years ago in
about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and
no change.

I need help.  Our whole production environment is unstable now since
every time an ASG brings a new instance online, I've got a 50-50 chance
that tomcat is going to die (and the health check doesn't catch it, but
that's a different issue).

There are no errors in the Tomcat boot sequence logs.  But the first
time and every subsequent time I try to get a connection from the
DataSource pool, I get the stack dump shown below.

I figure it has to be a timing/race condition.  But I have no clue what
to do to fix it.  I'm baffled that it worked for two years, and now
fails every other time I start an instance.  And every instance is
running copies of the exact same Amazon Machine Image.  The same EC2
will come up clean 50% of the time the next time it boots.

Could it be that your hosts are running out of entropy on boot?

Maybe there were RNG related changes in the kernel or rng-tools?

Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
it should not go below 100 or so.

Regards,
Simon

I haven't done any Linux or other system updates in several weeks. I'll 
look into the entropy possibility.  Would running out of entropy cause 
an exception stating that a crypto directory doesn't exist?  I don't 
know much about Java entropy.  Any ideas what would cause entropy to be 
good on one boot and bad on the next boot?  Thx.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: AW: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm

Hi Thomas,

On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Jerry,


-Ursprüngliche Nachricht-
Von: Jerry Malcolm 
Gesendet: Dienstag, 13. Juni 2023 08:50
An: users@tomcat.apache.org
Betreff: Crypto Randomly Not Getting Initialized

I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
Linux2 in a production environment.  A couple of years ago, we started
getting weird errors that the "Crypto Mechanism" failed to initialize. Through
a lot of trial and error, and reasons I don't quite remember, we put a 2-min
delay in rc.local before starting Tomcat, and the problem went away.  I'm
not a Linux nor a crypto guru.  But we traced it to some crypto file that we
assumed was not available until later in the Linux boot sequence.  Anyway,
the 2 minute delay made it go away, for over two years.  Then all of a sudden
in the last day or so, it's back with a vengeance.  It fails with the same 
crypto
error from 2 years ago in about 50% of the EC2 boot ups.  I tried bumping the
wait to 3 min, and no change.

I need help.  Our whole production environment is unstable now since every
time an ASG brings a new instance online, I've got a 50-50 chance that
tomcat is going to die (and the health check doesn't catch it, but that's a
different issue).

There are no errors in the Tomcat boot sequence logs.  But the first time and
every subsequent time I try to get a connection from the DataSource pool, I
get the stack dump shown below.

I figure it has to be a timing/race condition.  But I have no clue what to do to
fix it.  I'm baffled that it worked for two years, and now fails every other
time I start an instance.  And every instance is running copies of the exact
same Amazon Machine Image.  The same EC2 will come up clean 50% of the
time the next time it boots.

Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going
on here?  Thx

java.lang.ExceptionInInitializerError
      at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
      at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
      at
java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.jav
a:509)
      at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498)
      at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81)
      at
java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65)
      at
java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuit
es(SSLContextImpl.java:348)
      at
java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.(SSLC
ontextImpl.java:580)
      at java.base/java.lang.Class.forName0(Native Method)
      at java.base/java.lang.Class.forName(Class.java:315)
...

      at
com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
948)
      at
com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
      at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)
      at
com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
      at
com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
198)
      at
org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
DriverConnectionFactory.java:52)
      at
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
olableConnectionFactory.java:415)
      at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory
(BasicDataSource.java:111)
      at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection
Factory(BasicDataSource.java:649)
      at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa
taSource.java:532)
      at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS
ource.java:731)
      at jwm.db.DBData.getConnection(DBData.java:506)    my call to get
a db connection from connection pool 

...

Caused by: java.lang.SecurityException: Can not initialize cryptographic
mechanism
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy
directory: unlimited
      at
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:
326)
      at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
      at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
      at java.base/java.security.AccessController.doPrivileged(Native
Method)
      at
java.base/javax.crypto.JceSecurity.(JceSecurity.java:107)
      ... 86 more


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Could it be this issue?
https://github.com/docker-library/openjdk/issues/101

Maybe you can add information about the used jdk and whether you

Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread r . barclay
Hey Jerry,

> Caused by: java.lang.SecurityException: Can not initialize cryptographic 
> mechanism
>      at 
> java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
> Caused by: java.lang.SecurityException: Can't read cryptographic policy 
> directory: unlimited

Does this help?
https://stackoverflow.com/a/58183460/

Yours,
Reg


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Simon Matter
Hi,

> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> Linux2 in a production environment.  A couple of years ago, we started
> getting weird errors that the "Crypto Mechanism" failed to initialize. 
> Through a lot of trial and error, and reasons I don't quite remember, we
> put a 2-min delay in rc.local before starting Tomcat, and the problem
> went away.  I'm not a Linux nor a crypto guru.  But we traced it to some
> crypto file that we assumed was not available until later in the Linux
> boot sequence.  Anyway, the 2 minute delay made it go away, for over two
> years.  Then all of a sudden in the last day or so, it's back with a
> vengeance.  It fails with the same crypto error from 2 years ago in
> about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and
> no change.
>
> I need help.  Our whole production environment is unstable now since
> every time an ASG brings a new instance online, I've got a 50-50 chance
> that tomcat is going to die (and the health check doesn't catch it, but
> that's a different issue).
>
> There are no errors in the Tomcat boot sequence logs.  But the first
> time and every subsequent time I try to get a connection from the
> DataSource pool, I get the stack dump shown below.
>
> I figure it has to be a timing/race condition.  But I have no clue what
> to do to fix it.  I'm baffled that it worked for two years, and now
> fails every other time I start an instance.  And every instance is
> running copies of the exact same Amazon Machine Image.  The same EC2
> will come up clean 50% of the time the next time it boots.

Could it be that your hosts are running out of entropy on boot?

Maybe there were RNG related changes in the kernel or rng-tools?

Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
it should not go below 100 or so.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



AW: Crypto Randomly Not Getting Initialized

2023-06-13 Thread Thomas Hoffmann (Speed4Trade GmbH)
Hello Jerry,

> -Ursprüngliche Nachricht-
> Von: Jerry Malcolm 
> Gesendet: Dienstag, 13. Juni 2023 08:50
> An: users@tomcat.apache.org
> Betreff: Crypto Randomly Not Getting Initialized
> 
> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> Linux2 in a production environment.  A couple of years ago, we started
> getting weird errors that the "Crypto Mechanism" failed to initialize. Through
> a lot of trial and error, and reasons I don't quite remember, we put a 2-min
> delay in rc.local before starting Tomcat, and the problem went away.  I'm
> not a Linux nor a crypto guru.  But we traced it to some crypto file that we
> assumed was not available until later in the Linux boot sequence.  Anyway,
> the 2 minute delay made it go away, for over two years.  Then all of a sudden
> in the last day or so, it's back with a vengeance.  It fails with the same 
> crypto
> error from 2 years ago in about 50% of the EC2 boot ups.  I tried bumping the
> wait to 3 min, and no change.
> 
> I need help.  Our whole production environment is unstable now since every
> time an ASG brings a new instance online, I've got a 50-50 chance that
> tomcat is going to die (and the health check doesn't catch it, but that's a
> different issue).
> 
> There are no errors in the Tomcat boot sequence logs.  But the first time and
> every subsequent time I try to get a connection from the DataSource pool, I
> get the stack dump shown below.
> 
> I figure it has to be a timing/race condition.  But I have no clue what to do 
> to
> fix it.  I'm baffled that it worked for two years, and now fails every other
> time I start an instance.  And every instance is running copies of the exact
> same Amazon Machine Image.  The same EC2 will come up clean 50% of the
> time the next time it boots.
> 
> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going
> on here?  Thx
> 
> java.lang.ExceptionInInitializerError
>      at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
>      at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
>      at
> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.jav
> a:509)
>      at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498)
>      at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81)
>      at
> java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65)
>      at
> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuit
> es(SSLContextImpl.java:348)
>      at
> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.(SSLC
> ontextImpl.java:580)
>      at java.base/java.lang.Class.forName0(Native Method)
>      at java.base/java.lang.Class.forName(Class.java:315)
> ...
> 
>      at
> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
> 948)
>      at
> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
>      at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)
>      at
> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
>      at
> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
> 198)
>      at
> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
> DriverConnectionFactory.java:52)
>      at
> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
> olableConnectionFactory.java:415)
>      at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory
> (BasicDataSource.java:111)
>      at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection
> Factory(BasicDataSource.java:649)
>      at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa
> taSource.java:532)
>      at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS
> ource.java:731)
>      at jwm.db.DBData.getConnection(DBData.java:506)    my call to get
> a db connection from connection pool 
> 
> ...
> 
> Caused by: java.lang.SecurityException: Can not initialize cryptographic
> mechanism
>      at
> java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
> Caused by: java.lang.SecurityException: Can't read cryptographic policy
> directory: unlimited
>      at
> java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:
> 326)
>      at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
>      at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
>      at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>      at
&g

Crypto Randomly Not Getting Initialized

2023-06-13 Thread Jerry Malcolm
I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon 
Linux2 in a production environment.  A couple of years ago, we started 
getting weird errors that the "Crypto Mechanism" failed to initialize.  
Through a lot of trial and error, and reasons I don't quite remember, we 
put a 2-min delay in rc.local before starting Tomcat, and the problem 
went away.  I'm not a Linux nor a crypto guru.  But we traced it to some 
crypto file that we assumed was not available until later in the Linux 
boot sequence.  Anyway, the 2 minute delay made it go away, for over two 
years.  Then all of a sudden in the last day or so, it's back with a 
vengeance.  It fails with the same crypto error from 2 years ago in 
about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and 
no change.


I need help.  Our whole production environment is unstable now since 
every time an ASG brings a new instance online, I've got a 50-50 chance 
that tomcat is going to die (and the health check doesn't catch it, but 
that's a different issue).


There are no errors in the Tomcat boot sequence logs.  But the first 
time and every subsequent time I try to get a connection from the 
DataSource pool, I get the stack dump shown below.


I figure it has to be a timing/race condition.  But I have no clue what 
to do to fix it.  I'm baffled that it worked for two years, and now 
fails every other time I start an instance.  And every instance is 
running copies of the exact same Amazon Machine Image.  The same EC2 
will come up clean 50% of the time the next time it boots.


Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going on 
here?  Thx


java.lang.ExceptionInInitializerError
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
    at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
    at 
java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.java:509)

    at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:498)
    at java.base/sun.security.ssl.SSLCipher.(SSLCipher.java:81)
    at 
java.base/sun.security.ssl.CipherSuite.(CipherSuite.java:65)
    at 
java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuites(SSLContextImpl.java:348)
    at 
java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.(SSLContextImpl.java:580)

    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:315)
...

    at 
com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:948)
    at 
com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)

    at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448)
    at 
com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
    at 
com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
    at 
org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
    at 
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:415)
    at 
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:111)
    at 
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649)
    at 
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:532)
    at 
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731)
    at jwm.db.DBData.getConnection(DBData.java:506)    my call 
to get a db connection from connection pool 


...

Caused by: java.lang.SecurityException: Can not initialize cryptographic 
mechanism
    at 
java.base/javax.crypto.JceSecurity.(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy 
directory: unlimited
    at 
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)

    at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
    at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
    at java.base/java.security.AccessController.doPrivileged(Native 
Method)
    at 
java.base/javax.crypto.JceSecurity.(JceSecurity.java:107)

    ... 86 more


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org