Re: Slow startup first time after reboot
Neil, Sadly, that line doesn't appear either with or without the java.security.egd option. That appears to be a lump sum of 4-5 minutes for the SecureRandom seed thing. I'm getting a total accumulation of ~5 minutes. But it's made up of a bunch of ~15-sec web starts. See example below. Same two lines from Catalina.out from the first boot and a second boot. One other clue I've noted. The quick boot occurs on an immediate stop/start. I have noticed a couple of times in Eclipse that if I stop TC and leave it stopped for a while, it goes back to the long startup. It sounds like something related to loading modules/jarFiles. On an immediate restart the JVM wouldn't have time to unload modules before they are needed again. But a new loading of some module on each web app might be what's killing it (??) JVM Version: 11.0.16+8-LTS JVM Vendor: Red Hat, Inc. Auto-startup of TC on EC2 startup: 28-Sep-2022 00:15:55.612 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/domains/wridz.com/webapps/idmanager] has finished in [16,991] ms [1 example... ~20 more like it] 28-Sep-2022 00:18:40.818 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [30] milliseconds Reboot of TC 28-Sep-2022 02:28:45.854 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/domains/wridz.com/webapps/idmanager] has finished in [737] ms [1 example... ~20 more like it] 28-Sep-2022 02:28:51.476 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [19795] milliseconds On 9/27/2022 8:32 PM, Neil Aggarwal wrote: Are there perhaps some log levels I could change that would provide more detailed information about what step it's hung up on for loading these web apps? I just tested this on a dev sever. I removed the java.security.egd option and rebooted my server. Once I waited for Tomcat to finish starting up, catalina.out had this line: WARNING [main] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [222,741] milliseconds. Check if your log has something similar. Thank you, Neil -- Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com We offer 30 year loans on single family houses! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Slow startup first time after reboot
> Are there perhaps > some log levels I could change that would provide more detailed > information about what step it's hung up on for loading these web apps? I just tested this on a dev sever. I removed the java.security.egd option and rebooted my server. Once I waited for Tomcat to finish starting up, catalina.out had this line: WARNING [main] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [222,741] milliseconds. Check if your log has something similar. Thank you, Neil -- Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com We offer 30 year loans on single family houses! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow startup first time after reboot
Hi Neil, Your suggestion sounded very encouraging. Unfortunately, after adding the JVM parameter you suggested, there was no change. Are there perhaps some log levels I could change that would provide more detailed information about what step it's hung up on for loading these web apps? Thanks again. Jerry On 9/27/2022 4:50 PM, Neil Aggarwal wrote: Is this related to the random number generator? Take a look at this: https://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmrand.html I read somewhere to use this so I put in in my JVM args: -Djava.security.egd=file:/dev/./urandom Thank you, Neil -- Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com We offer 30 year loans on single family houses! -Original Message- From: Jerry Malcolm [mailto:techst...@malcolms.com] Sent: Tuesday, September 27, 2022 4:43 PM To: users@tomcat.apache.org Subject: Slow startup first time after reboot I'm running Tomcat 9.0.56 in a production environment on AWS Linux2. My environment is 4 virtual hosts with about 9-10 web applications on each. The web apps are already expanded in the webapps folder. So there are no war file expansions involved. The first time after I reboot the server each web app takes ~15-20 seconds to start up which is an eternity to get the full Tomcat up. But if I stop TC and restart it, everything starts up in just a few seconds. This same situation happens on my local windows development machine with eclipse. After starting eclipse, the first TC start I watch the stdout log creep along saying it loaded webapp xyz in 18 seconds... then loads the next web app in 16 seconds, etc. But if I stop TC and restart, immediate quick load. No errors or anything. No messages about taglib searches that I've seen. Simply starting a web app msg, wait 18 seconds completed. It's now gotten so bad in production TC taking so long to start that the heath checks are failing, terminating the server, spinning up a new one, and the process repeats again. Any ideas what might be causing this? Thanks. Jerry - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Slow startup first time after reboot
Is this related to the random number generator? Take a look at this: https://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmrand.html I read somewhere to use this so I put in in my JVM args: -Djava.security.egd=file:/dev/./urandom Thank you, Neil -- Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com We offer 30 year loans on single family houses! -Original Message- From: Jerry Malcolm [mailto:techst...@malcolms.com] Sent: Tuesday, September 27, 2022 4:43 PM To: users@tomcat.apache.org Subject: Slow startup first time after reboot I'm running Tomcat 9.0.56 in a production environment on AWS Linux2. My environment is 4 virtual hosts with about 9-10 web applications on each. The web apps are already expanded in the webapps folder. So there are no war file expansions involved. The first time after I reboot the server each web app takes ~15-20 seconds to start up which is an eternity to get the full Tomcat up. But if I stop TC and restart it, everything starts up in just a few seconds. This same situation happens on my local windows development machine with eclipse. After starting eclipse, the first TC start I watch the stdout log creep along saying it loaded webapp xyz in 18 seconds... then loads the next web app in 16 seconds, etc. But if I stop TC and restart, immediate quick load. No errors or anything. No messages about taglib searches that I've seen. Simply starting a web app msg, wait 18 seconds completed. It's now gotten so bad in production TC taking so long to start that the heath checks are failing, terminating the server, spinning up a new one, and the process repeats again. Any ideas what might be causing this? Thanks. Jerry - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Slow startup first time after reboot
I'm running Tomcat 9.0.56 in a production environment on AWS Linux2. My environment is 4 virtual hosts with about 9-10 web applications on each. The web apps are already expanded in the webapps folder. So there are no war file expansions involved. The first time after I reboot the server each web app takes ~15-20 seconds to start up which is an eternity to get the full Tomcat up. But if I stop TC and restart it, everything starts up in just a few seconds. This same situation happens on my local windows development machine with eclipse. After starting eclipse, the first TC start I watch the stdout log creep along saying it loaded webapp xyz in 18 seconds... then loads the next web app in 16 seconds, etc. But if I stop TC and restart, immediate quick load. No errors or anything. No messages about taglib searches that I've seen. Simply starting a web app msg, wait 18 seconds completed. It's now gotten so bad in production TC taking so long to start that the heath checks are failing, terminating the server, spinning up a new one, and the process repeats again. Any ideas what might be causing this? Thanks. Jerry - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: MaxRequestWorkers error
On 9/27/22 14:37, Thomas Hoffmann (Speed4Trade GmbH) wrote: How much memory did you configure for your java environment? --> Xmx Parameter If the java process eats too much memory, you can write a memory dump when an OOM exception occurs. E.g. with the parameters: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/logs/javadump.dmp" You can also open a JMX port and inspect the memory using during runtime, e.g. with VirtualVM https://visualvm.github.io/ Pretty sure Thomas mean "VisualVM" - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: MaxRequestWorkers error
Hello! > -Ursprüngliche Nachricht- > Von: Kumawat, Priyanka > Gesendet: Dienstag, 27. September 2022 21:41 > An: Tomcat Users List > Betreff: RE: MaxRequestWorkers error > > Hello Christopher , > > Thank you for your assistance !.Please find the mpm event given under > httpd.mpm.conf file on the server as below , the max request worker was > increased 50 more , approx two months before , that time we faced the > same issue with Max request error and Memory got exhausted on server , > the root cause was concluded that time was insufficient memory part of > solution 16GB memory was added to the server along with the Max worker > increase, the tomcat application heap size was also been adjusted - > > But we have started seeing the same kind of problems again , the tomcat > went to crash status with Out of memory errors , and Apache error logs with > Max request errors , whole production environment due to this . > > It occurred twice recently - 16th Sept and 20th Sept. > > ** > > Error details - > > : [Tue Sep 20 10:25:23.518687 2022] [mpm_event:error] [pid 29112:tid > 140343038482176] AH00484: server reached MaxRequestWorkers setting, > consider raising the MaxRequestWorkers setting [Tue Sep 20 10:48:08.134955 > 2022] [mpm_event:error] [pid 23940:tid 140413844547328] AH00484: server > reached MaxRequestWorkers setting, consider raising the > MaxRequestWorkers setting > > > [root@ustlswh552 logs]# uptime > 05:08:35 up 72 days, 5:39, 2 users, load average: 0.22, 0.13, 0.07 > > /opt/tomcat/apache-tomcat-8.5.5/logs/localhost.2022-09-16.log > > 16-Sep-2022 15:49:00.672 SEVERE [ajp-nio-8009-exec-117] > org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for > servlet [ws] in context with path [/thq] threw exception [Request processing > failed; nested exception is java.lang.OutOfMemoryError: unable to create > new native thread] with root cause > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) How much memory did you configure for your java environment? --> Xmx Parameter If the java process eats too much memory, you can write a memory dump when an OOM exception occurs. E.g. with the parameters: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/logs/javadump.dmp" You can also open a JMX port and inspect the memory using during runtime, e.g. with VirtualVM https://visualvm.github.io/ Other options are: - Activate AccessLog and check the request time. Are they ok or do they take too much time? - Activate tomcat-manager and check the number of active requests and sessions. Are they in the expected range? > > ** > > > # event MPM > # StartServers: initial number of server processes to start # > MinSpareThreads: minimum number of worker threads which are kept spare > # MaxSpareThreads: maximum number of worker threads which are kept > spare # ThreadsPerChild: constant number of worker threads in each server > process # MaxRequestWorkers: maximum number of worker threads # > MaxConnectionsPerChild: maximum number of connections a server process > serves > # before terminating > > StartServers 3 > MinSpareThreads 75 > MaxSpareThreads250 > ThreadsPerChild 50 > MaxRequestWorkers 500 > MaxConnectionsPerChild 0 > > > > Thanks & Regards, > > Priyanka Kumawat | Middleware Admin > T +91.7879364483 > EMail - priyanka.kuma...@dxc.com > DL - ams-leveraged-webadmin-offsh...@dxc.com > > DXC Technology > > > > > > -Original Message- > From: Christopher Schultz > Sent: 28 September 2022 00:19 > To: Tomcat Users List > Subject: Re: MaxRequestWorkers error > > Koustav, > > On 9/27/22 11:09, Naha, Koustav wrote: > > We have Tomcat and Apache installed in our production environment > > since > > 5/6 years. Everything was going fine until we started getting > > application not responding status from users, upon checking we found > > out that there was a MaxRequest error as below, > > > > * In Apache log we have found this error. > > > > [Tue Sep 20 10:48:08.134955 2022] [mpm_event:error] [pid 23940:tid > > 140413844547328] AH00484: server reached MaxRequestWorkers setting, > > consider raising the MaxRequestWorkers setting > > > > * we have seen the below error on tomcat logs > > Your attachment has been removed from the mailing list. Can you please > provide a text-only description of the problem? > > > We increased 16gb memory on both servers , now we have 32 Gb memory > in > > each servers. But the issue still is there. > > You could have 1TiB of memory on your server and it would not solve the > issue you are reporting. > > > Suddenly we start getting the error and we have to restart Tomcat and > > Apache. > > > > FAQ: > > > > 1.
[ANN] Apache Tomcat 10.0.26 available
The Apache Tomcat team announces the immediate availability of Apache Tomcat 10.0.26. This release is targeted at Jakarta EE 9. Applications that run on Tomcat 9 and earlier will not run on Tomcat 10 without changes. Java EE applications designed for Tomcat 9 and earlier may be placed in the $CATALINA_BASE/webapps-javaee directory and Tomcat will automatically convert them to Jakarta EE and copy them to the webapps directory. This conversion is performed using the Apache Tomcat migration tool for Jakarta EE tool which is also available as a separate download for off-line use. Apache Tomcat 10 is an open source software implementation of the Jakarta Servlet, Jakarta Server Pages, Jakarta Expression Language, Jakarta WebSocket, Jakarta Authentication and Jakarta Annotations specifications. The notable changes compared to 10.0.23 include: - Add support for authenticating WebSocket clients with an HTTP forward proxy when establishing a connection to a WebSocket endpoint via a forward proxy that requires authentication. Based on a patch provided by Joe Mokos. - Various fixes for edge case bugs in EL processing - Improve host header handling for HTTP/2 requests Please refer to the change log for the complete list of changes: http://tomcat.apache.org/tomcat-10.0-doc/changelog.html Downloads: http://tomcat.apache.org/download-10.cgi Migration guides from Apache Tomcat 7.0.x, 8.5.x and 9.0.x: http://tomcat.apache.org/migration.html Enjoy! - The Apache Tomcat team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: MaxRequestWorkers error
Hello Christopher , Thank you for your assistance !.Please find the mpm event given under httpd.mpm.conf file on the server as below , the max request worker was increased 50 more , approx two months before , that time we faced the same issue with Max request error and Memory got exhausted on server , the root cause was concluded that time was insufficient memory part of solution 16GB memory was added to the server along with the Max worker increase, the tomcat application heap size was also been adjusted - But we have started seeing the same kind of problems again , the tomcat went to crash status with Out of memory errors , and Apache error logs with Max request errors , whole production environment due to this . It occurred twice recently - 16th Sept and 20th Sept. ** Error details - : [Tue Sep 20 10:25:23.518687 2022] [mpm_event:error] [pid 29112:tid 140343038482176] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting [Tue Sep 20 10:48:08.134955 2022] [mpm_event:error] [pid 23940:tid 140413844547328] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting [root@ustlswh552 logs]# uptime 05:08:35 up 72 days, 5:39, 2 users, load average: 0.22, 0.13, 0.07 /opt/tomcat/apache-tomcat-8.5.5/logs/localhost.2022-09-16.log 16-Sep-2022 15:49:00.672 SEVERE [ajp-nio-8009-exec-117] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [ws] in context with path [/thq] threw exception [Request processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) ** # event MPM # StartServers: initial number of server processes to start # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestWorkers: maximum number of worker threads # MaxConnectionsPerChild: maximum number of connections a server process serves # before terminating StartServers 3 MinSpareThreads 75 MaxSpareThreads250 ThreadsPerChild 50 MaxRequestWorkers 500 MaxConnectionsPerChild 0 Thanks & Regards, Priyanka Kumawat | Middleware Admin T +91.7879364483 EMail - priyanka.kuma...@dxc.com DL - ams-leveraged-webadmin-offsh...@dxc.com DXC Technology -Original Message- From: Christopher Schultz Sent: 28 September 2022 00:19 To: Tomcat Users List Subject: Re: MaxRequestWorkers error Koustav, On 9/27/22 11:09, Naha, Koustav wrote: > We have Tomcat and Apache installed in our production environment > since > 5/6 years. Everything was going fine until we started getting > application not responding status from users, upon checking we found > out that there was a MaxRequest error as below, > > * In Apache log we have found this error. > > [Tue Sep 20 10:48:08.134955 2022] [mpm_event:error] [pid 23940:tid > 140413844547328] AH00484: server reached MaxRequestWorkers setting, > consider raising the MaxRequestWorkers setting > > * we have seen the below error on tomcat logs Your attachment has been removed from the mailing list. Can you please provide a text-only description of the problem? > We increased 16gb memory on both servers , now we have 32 Gb memory in > each servers. But the issue still is there. You could have 1TiB of memory on your server and it would not solve the issue you are reporting. > Suddenly we start getting the error and we have to restart Tomcat and > Apache. > > FAQ: > > 1. Need permeant Solution on this as restarting the instances has > became too frequent these days. > 2. We increased MaxRequestWorker value 400 from 200. > > > StartServers 3 > MinSpareThreads 75 > MaxSpareThreads250 > ThreadsPerChild 25 > MaxRequestWorkers 400 > MaxConnectionsPerChild 0 > Your error message says "[mpm_event:error]" which suggests you are using the "event" MPM and not the "worker" MPM. You have configured the "worker" MPM above. Do you have a different config section for mpm_event_module? Another relevant setting is "ServerLimit". > 3. Database and network team reported no unusual activity during the issue. Good. That suggests you don't actually have a "problem". You are just hitting the limits imposed by your configuration. > 4. Code is refreshed every month , no issue found on coding side. > 5. While the issue starts the server becomes low on memory. If you are running httpd and Tomcat on the same machine, my
Re: MaxRequestWorkers error
Koustav, On 9/27/22 11:09, Naha, Koustav wrote: We have Tomcat and Apache installed in our production environment since 5/6 years. Everything was going fine until we started getting application not responding status from users, upon checking we found out that there was a MaxRequest error as below, * In Apache log we have found this error. [Tue Sep 20 10:48:08.134955 2022] [mpm_event:error] [pid 23940:tid 140413844547328] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting * we have seen the below error on tomcat logs Your attachment has been removed from the mailing list. Can you please provide a text-only description of the problem? We increased 16gb memory on both servers , now we have 32 Gb memory in each servers. But the issue still is there. You could have 1TiB of memory on your server and it would not solve the issue you are reporting. Suddenly we start getting the error and we have to restart Tomcat and Apache. FAQ: 1. Need permeant Solution on this as restarting the instances has became too frequent these days. 2. We increased MaxRequestWorker value 400 from 200. StartServers 3 MinSpareThreads 75 MaxSpareThreads250 ThreadsPerChild 25 MaxRequestWorkers 400 MaxConnectionsPerChild 0 Your error message says "[mpm_event:error]" which suggests you are using the "event" MPM and not the "worker" MPM. You have configured the "worker" MPM above. Do you have a different config section for mpm_event_module? Another relevant setting is "ServerLimit". 3. Database and network team reported no unusual activity during the issue. Good. That suggests you don't actually have a "problem". You are just hitting the limits imposed by your configuration. 4. Code is refreshed every month , no issue found on coding side. 5. While the issue starts the server becomes low on memory. If you are running httpd and Tomcat on the same machine, my guess is that the memory increase is due to additional load on the application, not the web server. The web server should be using minimal memory compared to your application. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
MaxRequestWorkers error
Hi All, Need expert advices on this issue. We have Tomcat and Apache installed in our production environment since 5/6 years. Everything was going fine until we started getting application not responding status from users, upon checking we found out that there was a MaxRequest error as below, * In Apache log we have found this error. [Tue Sep 20 10:48:08.134955 2022] [mpm_event:error] [pid 23940:tid 140413844547328] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting * we have seen the below error on tomcat logs [cid:image001.png@01D8D2AF.EB134CD0] We increased 16gb memory on both servers , now we have 32 Gb memory in each servers. But the issue still is there. Suddenly we start getting the error and we have to restart Tomcat and Apache. FAQ: 1. Need permeant Solution on this as restarting the instances has became too frequent these days. 2. We increased MaxRequestWorker value 400 from 200. StartServers 3 MinSpareThreads 75 MaxSpareThreads250 ThreadsPerChild 25 MaxRequestWorkers 400 MaxConnectionsPerChild 0 1. Database and network team reported no unusual activity during the issue. 2. Code is refreshed every month , no issue found on coding side. 3. While the issue starts the server becomes low on memory. Thanks and Regards, Koustav Naha DXC Technology Company -- This message is transmitted to you by or on behalf of DXC Technology Company or one of its affiliates. It is intended exclusively for the addressee. The substance of this message, along with any attachments, may contain proprietary, confidential or privileged information or information that is otherwise legally exempt from disclosure. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate any part of this message. If you have received this message in error, please destroy and delete all copies and notify the sender by return e-mail. Regardless of content, this e-mail shall not operate to bind DXC Technology Company or any of its affiliates to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose.
