AW: AW: Slow startup first time after reboot

[Thomas Hoffmann (Speed4Trade GmbH)]

Hello,

> -Ursprüngliche Nachricht-
> Von: Jerry Malcolm 
> Gesendet: Freitag, 30. September 2022 21:05
> An: users@tomcat.apache.org
> Betreff: Re: AW: Slow startup first time after reboot
> 
> Well, like taking your car into the shop It decided to stop failing for a 
> while
> on my local machine.  Finally this morning I got it to start doing the slow 
> load
> again inside Eclipse.  I suspended the thread several different times and got
> the same thread dump.  I'm not sure what it's doing here.  It's opening a jar,
> but it's trying to open a "RandomAccessFile".  With the word "random', it
> sounds like this could be possibly related to that JVM random noise
> seed.  But the 'fix' that was recommended did not have an effect unless I
> added the parameter incorrectly.  Does this thread dump provide any
> insights?
> 
> Thread [main] (Suspended)
>      owns: JarResourceSet  (id=73)
>      owns: StandardRoot  (id=74)
>      owns: StandardContext  (id=75)
>      owns: StandardHost  (id=21)
>      owns: StandardEngine  (id=22)
>      owns: StandardService  (id=23)
>      owns: Object  (id=24)
>      owns: StandardServer  (id=25)
>      RandomAccessFile.open0(String, int) line: not available [native method]
>      RandomAccessFile.open(String, int) line: 346
>      RandomAccessFile.(File, String, boolean) line: 260
>      RandomAccessFile.(File, String) line: 215
>      ZipFile$Source.(ZipFile$Source$Key, boolean) line: 1271
>      ZipFile$Source.get(File, boolean) line: 1237
>      ZipFile$CleanableResource.(ZipFile, File, int) line: 727
>      ZipFile$CleanableResource.get(ZipFile, File, int) line: 844
>      JarFile(ZipFile).(File, int, Charset) line: 247
>      JarFile(ZipFile).(File, int) line: 177
>      JarFile.(File, boolean, int, Runtime$Version) line: 346
>      GeneratedConstructorAccessor14.newInstance(Object[]) line: not
> available
>      DelegatingConstructorAccessorImpl.newInstance(Object[]) line: 45
>      Constructor.newInstance(Object...) line: 490
>      Jre9Compat.jarFileNewInstance(File) line: 206
>      Jre9Compat(JreCompat).jarFileNewInstance(String) line: 226
>   JarResourceSet(AbstractSingleArchiveResourceSet).initInternal() line: 140
>      JarResourceSet(LifecycleBase).init() line: 136
>      JarResourceSet(LifecycleBase).start() line: 173
>      StandardRoot.startInternal() line: 726
>      StandardRoot(LifecycleBase).start() line: 183
>      StandardContext.resourcesStart() line: 4885
>      StandardContext.startInternal() line: 5023
>      StandardContext(LifecycleBase).start() line: 183
>      StandardHost(ContainerBase).addChildInternal(Container) line: 726
>      StandardHost(ContainerBase).addChild(Container) line: 698
>      StandardHost.addChild(Container) line: 696
>      HostConfig.deployDirectory(ContextName, File) line: 1185
>      HostConfig$DeployDirectory.run() line: 1933
>      Executors$RunnableAdapter.call() line: 515
>      FutureTask.run() line: 264
>      InlineExecutorService.execute(Runnable) line: 75
>   InlineExecutorService(AbstractExecutorService).submit(Runnable) line: 118
>      HostConfig.deployDirectories(File, String[]) line: 1095
>      HostConfig.deployApps() line: 477
>      HostConfig.start() line: 1618
>      HostConfig.lifecycleEvent(LifecycleEvent) line: 319
>      StandardHost(LifecycleBase).fireLifecycleEvent(String, Object)
> line: 123
>      StandardHost(LifecycleBase).setStateInternal(LifecycleState,
> Object, boolean) line: 423
>      StandardHost(LifecycleBase).setState(LifecycleState) line: 366
>      StandardHost(ContainerBase).startInternal() line: 946
>      StandardHost.startInternal() line: 835
>      StandardHost(LifecycleBase).start() line: 183
>      ContainerBase$StartChild.call() line: 1396
>      ContainerBase$StartChild.call() line: 1386
>      FutureTask.run() line: 264
>      InlineExecutorService.execute(Runnable) line: 75
>   InlineExecutorService(AbstractExecutorService).submit(Callable)
> line: 140
>      StandardEngine(ContainerBase).startInternal() line: 919
>      StandardEngine.startInternal() line: 263
>      StandardEngine(LifecycleBase).start() line: 183
>      StandardService.startInternal() line: 432
>      StandardService(LifecycleBase).start() line: 183
>      StandardServer.startInternal() line: 927
>      StandardServer(LifecycleBase).start() line: 183
>      Catalina.start() line: 772
>      NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line:
> not available [native method]
>      NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
>      DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
>      Method.invoke(Object, Object...) line: 566
>      Bootstrap.start() line: 345
>      Bootstrap.main(String[]) line: 476


RandomAccessFile only means that the file is opened and allows random access to 
it.
It is not read in a sequential way but the class files can be access as needed.
Is it a war file or a normal web folder?
How many jars are in the WEB-INF\lib direc

Re: AW: Slow startup first time after reboot

[Jerry Malcolm]

Well, like taking your car into the shop It decided to stop failing 
for a while on my local machine.  Finally this morning I got it to start 
doing the slow load again inside Eclipse.  I suspended the thread 
several different times and got the same thread dump.  I'm not sure what 
it's doing here.  It's opening a jar, but it's trying to open a 
"RandomAccessFile".  With the word "random', it sounds like this could 
be possibly related to that JVM random noise seed.  But the 'fix' that 
was recommended did not have an effect unless I added the parameter 
incorrectly.  Does this thread dump provide any insights?


Thread [main] (Suspended)
    owns: JarResourceSet  (id=73)
    owns: StandardRoot  (id=74)
    owns: StandardContext  (id=75)
    owns: StandardHost  (id=21)
    owns: StandardEngine  (id=22)
    owns: StandardService  (id=23)
    owns: Object  (id=24)
    owns: StandardServer  (id=25)
    RandomAccessFile.open0(String, int) line: not available [native 
method]

    RandomAccessFile.open(String, int) line: 346
    RandomAccessFile.(File, String, boolean) line: 260
    RandomAccessFile.(File, String) line: 215
    ZipFile$Source.(ZipFile$Source$Key, boolean) line: 1271
    ZipFile$Source.get(File, boolean) line: 1237
    ZipFile$CleanableResource.(ZipFile, File, int) line: 727
    ZipFile$CleanableResource.get(ZipFile, File, int) line: 844
    JarFile(ZipFile).(File, int, Charset) line: 247
    JarFile(ZipFile).(File, int) line: 177
    JarFile.(File, boolean, int, Runtime$Version) line: 346
    GeneratedConstructorAccessor14.newInstance(Object[]) line: not 
available

    DelegatingConstructorAccessorImpl.newInstance(Object[]) line: 45
    Constructor.newInstance(Object...) line: 490
    Jre9Compat.jarFileNewInstance(File) line: 206
    Jre9Compat(JreCompat).jarFileNewInstance(String) line: 226
 JarResourceSet(AbstractSingleArchiveResourceSet).initInternal() line: 140
    JarResourceSet(LifecycleBase).init() line: 136
    JarResourceSet(LifecycleBase).start() line: 173
    StandardRoot.startInternal() line: 726
    StandardRoot(LifecycleBase).start() line: 183
    StandardContext.resourcesStart() line: 4885
    StandardContext.startInternal() line: 5023
    StandardContext(LifecycleBase).start() line: 183
    StandardHost(ContainerBase).addChildInternal(Container) line: 726
    StandardHost(ContainerBase).addChild(Container) line: 698
    StandardHost.addChild(Container) line: 696
    HostConfig.deployDirectory(ContextName, File) line: 1185
    HostConfig$DeployDirectory.run() line: 1933
    Executors$RunnableAdapter.call() line: 515
    FutureTask.run() line: 264
    InlineExecutorService.execute(Runnable) line: 75
 InlineExecutorService(AbstractExecutorService).submit(Runnable) line: 118
    HostConfig.deployDirectories(File, String[]) line: 1095
    HostConfig.deployApps() line: 477
    HostConfig.start() line: 1618
    HostConfig.lifecycleEvent(LifecycleEvent) line: 319
    StandardHost(LifecycleBase).fireLifecycleEvent(String, Object) 
line: 123
    StandardHost(LifecycleBase).setStateInternal(LifecycleState, 
Object, boolean) line: 423

    StandardHost(LifecycleBase).setState(LifecycleState) line: 366
    StandardHost(ContainerBase).startInternal() line: 946
    StandardHost.startInternal() line: 835
    StandardHost(LifecycleBase).start() line: 183
    ContainerBase$StartChild.call() line: 1396
    ContainerBase$StartChild.call() line: 1386
    FutureTask.run() line: 264
    InlineExecutorService.execute(Runnable) line: 75
 InlineExecutorService(AbstractExecutorService).submit(Callable) 
line: 140

    StandardEngine(ContainerBase).startInternal() line: 919
    StandardEngine.startInternal() line: 263
    StandardEngine(LifecycleBase).start() line: 183
    StandardService.startInternal() line: 432
    StandardService(LifecycleBase).start() line: 183
    StandardServer.startInternal() line: 927
    StandardServer(LifecycleBase).start() line: 183
    Catalina.start() line: 772
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: 
not available [native method]

    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
    Method.invoke(Object, Object...) line: 566
    Bootstrap.start() line: 345
    Bootstrap.main(String[]) line: 476

On 9/29/2022 4:29 AM, Mark Thomas wrote:

On 29/09/2022 04:05, Jerry Malcolm wrote:
Hi, Mark, Thanks for the info. Getting several thread dumps is not 
going to be easy in this specific situation.  This problem only 
occurs on initial boot up of a newly-imaged linux EC2 which is built 
and launched automatically by the AWS autoscaling service.  By the 
time i can see that an image has been created and launched, get the 
newly assigned IP address from AWS, then go through the 
initialization of a puTTY session, then get the PID for tomcat, I'm 
pretty much out of time to try to get meaningful thread dumps. I have 
no doubt that the thread dump might tell us something.  B

RE: Install CA signed certificate on Tomcat 9

[Janardhanan, Veni]

C:\>"C:\Program Files\RedHat\java-11-openjdk-11.0.13-1\bin\keytool" -list 
-keystore C:\SSL\myserver.keystore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

tomcat, Sep 8, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 
8B:1D:5B:59:86:39:A5:CD:AB:2A:4A:45:13:2B:82:A1:44:CD:8A:E7:20:96:5A:02:0F:73:E3:5A:A6:DB:B6:FD
tomcat1, Sep 29, 2022, trustedCertEntry,
Certificate fingerprint (SHA-256): 
1F:A1:D5:1A:AD:5C:57:6C:B8:90:D8:CA:D1:89:2D:E1:1E:1F:7E:78:D2:19:72:CE:CC:3B:25:03:DE:0F:E1:B6

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to 
PKCS12 which is an industry standard format using "keytool -importkeystore 
-srckeystore C:\SSL\myserver.keystore -destkeystore C:\SSL\myserver.keystore 
-deststoretype pkcs12".

From: Rob Sargent 
Sent: Friday, September 30, 2022 6:45 PM
To: Tomcat Users List 
Subject: Re: Install CA signed certificate on Tomcat 9

> On Sep 30, 2022, at 5: 17 AM, Janardhanan, Veni  
> wrote: > >  > Hi Chris, > > I executed the command that you asked and below 
> is it’s output. > Paste the text of the output. Not a screen
ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
ZjQcmQRYFpfptBannerEnd





> On Sep 30, 2022, at 5:17 AM, Janardhanan, Veni 
> mailto:vjanardha...@trueblue.com>> wrote:

>

> 

> Hi Chris,

>

> I executed the command that you asked and below is it’s output.

>



Paste the text of the output. Not a screen shot



>

>

>

Re: Install CA signed certificate on Tomcat 9

[Rob Sargent]

> On Sep 30, 2022, at 5:17 AM, Janardhanan, Veni  
> wrote:
> 
> 
> Hi Chris,
>  
> I executed the command that you asked and below is it’s output.
>  

Paste the text of the output. Not a screen shot

> 
> 
>  

Re: tomcat 9.0.67 not supported

[Mark Thomas]

On 30/09/2022 12:54, zhta...@163.com wrote:

Hi,
 I used the latest version of tomcat (9.0.67) to deploy jsp applications. I found 
that thetag is not supported, but 9.0.65 does. Is it no longer 
supported in later versions?


https://bz.apache.org/bugzilla/show_bug.cgi?id=66277

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Install CA signed certificate on Tomcat 9

[Janardhanan, Veni]

Hi Chris,

I executed the command that you asked and below is it’s output.

[cid:image001.png@01D8D4E9.CF1FDCF0]

SSLHostConfig details :




I am from Bengaluru IST time zone and can work with you on a screenshare if we 
need to, please let me know.

A bit of background : Few months back I upgraded our existing SAP BO BI 
Platform 4.2 to 4.3, on the upgraded server I successfully installed CA signed 
certificate. This time I did a fresh install of SAP BI 4.3 on a windows box and 
duplicated same steps to install CA certificate on Tomcat during my 4.3 upgrade 
but this time Tomcat errors out and am unable to get the Central Management 
Console (CMC) login page.

What works now : am able to access CMC login page using the self-signed 
certificate and can login successfully but as ‘Not Secure’ site.

Thanks,
Veni

From: Christopher Schultz 
Sent: Friday, September 30, 2022 3:23 AM
To: users@tomcat.apache.org
Subject: Re: Install CA signed certificate on Tomcat 9

Veni, On 9/29/22 13: 21, Janardhanan, Veni wrote: > Hi, > > My Tomcat version 
is 9. I am trying to install a CA signed certificate on Tomcat, tomcat error 
log says Invalid Keystore format. > Followed instructions given in ‍ ‍ ‍ ‍ ‍ ‍
ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
ZjQcmQRYFpfptBannerEnd

Veni,



On 9/29/22 13:21, Janardhanan, Veni wrote:

> Hi,

>

> My Tomcat version is 9. I am trying to install a CA signed certificate on 
> Tomcat, tomcat error log says Invalid Keystore format.

> Followed instructions given in 
> https://urldefense.com/v3/__https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html*Installing_a_Certificate_from_a_Certificate_Authority__;Iw!!B5G2v5lgyw!ItNasjh8dSW9-I0Yf5oKgweZWrpL81ZZ3ef0g-qXo0X0budUzFCWkouE0e9b1a19GI7Pz2g2DIzzeaI6lvZvtSql3o1c$



What kind of keystore are you using? Please post the output of this command:



$ keytool -list -keystore [path to keeystore]



Please post your  with all secrets removed.



-chris





-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: 
users-h...@tomcat.apache.org

AW: AW: MaxRequestWorkers error

[Thomas Hoffmann (Speed4Trade GmbH)]

Hello,

> -Ursprüngliche Nachricht-
> Von: Kumawat, Priyanka 
> Gesendet: Donnerstag, 29. September 2022 22:28
> An: Tomcat Users List 
> Betreff: RE: AW: MaxRequestWorkers error
> 
> Hi Thomas,
> 
> -> How much memory did you configure for your java environment? -->
> 
> -Xms4096m -Xmx8192m

This depends on the application and the server memory.
Every application is different. Depends on what they keep in memory. Some 
application eat a lot of memory, some less.
Maybe you can take a look at the requirements or ask developer if possible.

> -> Parameter If the java process eats too much memory, you can write a
> memory dump when an OOM exception occurs.
> 
> Do we need to run this cmd with the startup.sh tomcat cmd , or can we run
> this directly on server , before restarting the tomcat during OOM error.
> 
> -XX:+HeapDumpOnOutOfMemoryError -
> XX:HeapDumpPath=/opt/tomcat/apache-tomcat-8.5.5/logs/javadump.dmp
> 

All the -X... stuff are startup parameters / variables. Usually set via 
setenv.sh.
Please check e.g. 
https://crunchify.com/how-to-change-jvm-heap-setting-xms-xmx-of-tomcat/ 

> Also can we use the cmd's also for generating heap and thread dumps ,
> please suggest are these cmd's useful for these error condition.
> 
> kill -3 PID
> 
> /opt/tomcat/jdk1.8.0_121/bin
> ./jmap -F -dump:format=b,file=/tmp/heap.bin PID ./jmap -F -
> dump:format=b,file=/tmp/heap.hpof PID
> 

Kill and jmap can dump the memory during runtime, yes.
However you need tools like MAT https://www.eclipse.org/mat/ and the dumps in 
general are useful for the developers of the app.


> The below is the current memory foot print on the prod server now -
> 
> tomcat@[/tmp]$ free -m
>  total   used   free sharedbuffers cached
> Mem: 32109  31422687 69   1185  15452
> -/+ buffers/cache:  14783  17325
> Swap:16383  0  16383
> tomcat@[/tmp]$ free -m
>  total   used   free sharedbuffers cached
> Mem: 32109  31422687 69   1185  15452
> -/+ buffers/cache:  14783  17325
> Swap:16383  0  16383
> tomcat@[/tmp]$ free -m
>  total   used   free sharedbuffers cached
> Mem: 32109  31422687 69   1185  15452
> -/+ buffers/cache:  14783  17325
> Swap:16383  0  16383
> 
> 
> > > You can also open a JMX port and inspect the memory using during
> > > runtime, e.g. with VirtualVM
> 
> We are still trying this option to get run , this is asking to have java 1.8 
> or
> higher , can we use any other option to check the Memory during runtime?.


VisualVM is best in my point of view. Don’t you have Java 8 or higher running?


> 
> Please suggest . Thankyou !
> 
> 
> Thanks & Regards,
> 
> Priyanka Kumawat | Middleware Admin
> T +91.7879364483
> EMail - priyanka.kuma...@dxc.com
> DL - ams-leveraged-webadmin-offsh...@dxc.com
> 
> DXC Technology
> 
> 
> 
> 
> 
> 
> 
> 
> -Original Message-
> From: Thomas Hoffmann (Speed4Trade GmbH)
> 
> Sent: 28 September 2022 12:02
> To: Tomcat Users List 
> Subject: AW: AW: MaxRequestWorkers error
> 
> 
> 
> > -Ursprüngliche Nachricht-
> > Von: Rob Sargent 
> > Gesendet: Dienstag, 27. September 2022 23:26
> > An: users@tomcat.apache.org
> > Betreff: Re: AW: MaxRequestWorkers error
> >
> >
> >
> > On 9/27/22 14:37, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > >
> > > How much memory did you configure for your java environment? -->
> Xmx
> > > Parameter If the java process eats too much memory, you can write a
> > memory dump when an OOM exception occurs.
> > > E.g. with the parameters: -XX:+HeapDumpOnOutOfMemoryError -
> > XX:HeapDumpPath=/var/logs/javadump.dmp"
> > > You can also open a JMX port and inspect the memory using during
> > > runtime, e.g. with VirtualVM
> > >
> https://clicktime.symantec.com/15tStaehQas9svtdpJ83n?h=sL3qdTIGckj1F
> > > A6lY59OUlcDtGbnfGtyeXufk7TPq70=&u=https://visualvm.github.io/
> > >
> > Pretty sure Thomas mean "VisualVM"
> 
> Of course :)
> Link was right, name was wrong.
> 
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> B
> KK
> KKCB  [  X  ܚX KK[XZ[  \ \  ][  X  ܚX P X ]  \X K ܙ B
> ܈Y][ۘ[  [X[  K[XZ[  \ \  Z[ X ]  \X K ܙ B
> 
> 
> DXC Technology Company -- This message is transmitted to you by or on
> behalf of DXC Technology Company or one of its affiliates. It is intended
> exclusively for the addressee. The substance of this message, along with any
> attachments, may contain proprietary, confidential or privileged information
> or information that is otherwise legally exempt from disclosure. Any
> unauthorized review, use, disclos