Excessive threads crashing tomcat
Hi I have an issue where my applicaion servers have recently started failing with OOM - cannot create native thread errors. However the dump files all show he majority of threads (over 90%) like the following: pool-21088-thread-1 - Thread t@36407 java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for 2bc1d4f0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Locked ownable synchronizers: - None My first assumption is that these are for a connection pool, however I am seeing close to 1000 (one thousand) and these never appear to timeout, or close. I could of course increase the number of threads the OS allows, but this of course has memory implications etc and is not really solving the root cause. Can anyone identify what these are? I see some posts online indicating these are the connection pool waiting for new requests. But this doesnt seem to match my configuation for acceptable connections Any help would be greatly appreciated Thanks *Mitchell * Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Re: Excessive threads crashing tomcat
Tomcat7 Java1.7_025 On 21 October 2014 20:06, Smith, Mitchell mitchell.sm...@cwc.com wrote: Hi I have an issue where my applicaion servers have recently started failing with OOM - cannot create native thread errors. However the dump files all show he majority of threads (over 90%) like the following: pool-21088-thread-1 - Thread t@36407 java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for 2bc1d4f0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Locked ownable synchronizers: - None My first assumption is that these are for a connection pool, however I am seeing close to 1000 (one thousand) and these never appear to timeout, or close. I could of course increase the number of threads the OS allows, but this of course has memory implications etc and is not really solving the root cause. Can anyone identify what these are? I see some posts online indicating these are the connection pool waiting for new requests. But this doesnt seem to match my configuation for acceptable connections Any help would be greatly appreciated Thanks *Mitchell * Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
maxActiveConnections usage
Hi, I was looking at using the maxActiveConnections in the application context, what I find is. When I reach the limit, as expected the server will not create a new session, however does return a http 200 status code. I am wondering would it be possible for me to change this behavior to return say a 300 range code to redirect to (in a production environment) load balancer. I can see some potential issues with this, like the possibility of an infinite loop, but for now I am just exploring possible ways to prevent my tomcat servers becoming overloaded. Does anyone have any thoughts or ideas for this type of solution? Regards -- *Mitchell Smith * The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Re: maxActiveConnections usage
On 13 August 2013 13:24, Daniel Mikusa dmik...@gopivotal.com wrote: On Aug 13, 2013, at 6:43 AM, Smith, Mitchell mitchell.sm...@cwc.com wrote: Hi, What version of Tomcat are you using? 6.0.x or 7.0.x? Tomcat7 I was looking at using the maxActiveConnections in the application context, Please include the relevant configuration from your server.xml or context.xml file. Context.xml Context !-- Default set of monitored resources -- WatchedResourceWEB-INF/web.xml/WatchedResource !-- Uncomment this to disable session persistence across Tomcat restarts -- Manager pathname= *maxActiveConnections=150 */ !-- Uncomment this to enable Comet connection tacking (provides events on session expiration as well as webapp lifecycle) -- !-- Valve className=org.apache.catalina.valves.CometConnectionManagerValve / -- /Context what I find is. When I reach the limit, as expected the server will not create a new session, however does return a http 200 status code. I am wondering would it be possible for me to change this behavior to return say a 300 range code to redirect to (in a production environment) load balancer. I can see some potential issues with this, like the possibility of an infinite loop, but for now I am just exploring possible ways to prevent my tomcat servers becoming overloaded. How do you have your load balancer setup now? What software / hardware are you using? What algorithm is it using to distribute the load? Its an apache httpd server, distributing load using byRequest, with a redundant fail-over node configured howevr this only works if the tomcat server fails to accept the initial connection Does anyone have any thoughts or ideas for this type of solution? Perhaps you need to adjust the load balancer so it's spreading out the connections more evenly? Perhaps you need to add another node to handle the load? Its a recent issue, typical user response to a slow application (for whatever reason) is to open another session, however the jvmRoute is stored in the browser which forces the 2nd,3rd,4th session to the same host. We are looking at a application solution to provide a new session without the jvmRoute in separate tabs, but the ability to reject and redirect a user (even to a error page) when the server limit we set is reached would prevent a larger scale outage. Dan Regards -- *Mitchell Smith * - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Re: maxActiveConnections usage
Dan, I am currently developing/testing with the latest version: 7.0.42 *Note this is different than the configuration you specified above, which appears like it's trying to restrict the number of sessions that the manager will create* Yes, maxActiveSessions is the parameter I am using. As I know the memory footprint of a session in the application I have set the parameter to the number of sessions I would support per application instance. Thus ensuring that an instance is not overloaded by sessions. My idea is based on the understanding that I have a fixed user base, and my infrastructure is designed to scale for this user base. The issue occurs if the LB fails to balance (for what ever reason), the tomcat instance could block new sessions and pass the user to a static (helpful) error page and/or redirect them back to the base URL of the LB. What I am struggling is the ability to catch the IllegalStateException that would be thrown by the Manager, and forward to the error/redirect as a result of this. - Mitchell On 13 August 2013 15:18, Daniel Mikusa dmik...@gopivotal.com wrote: On Aug 13, 2013, at 9:33 AM, Smith, Mitchell mitchell.sm...@cwc.com wrote: On 13 August 2013 13:24, Daniel Mikusa dmik...@gopivotal.com wrote: On Aug 13, 2013, at 6:43 AM, Smith, Mitchell mitchell.sm...@cwc.com wrote: Hi, What version of Tomcat are you using? 6.0.x or 7.0.x? Tomcat7 Please include the complete version number. There are 42 different Tomcat 7 releases. The more specific you can be the better response you'll get from the list. I was looking at using the maxActiveConnections in the application context, Please include the relevant configuration from your server.xml or context.xml file. Context.xml Context !-- Default set of monitored resources -- WatchedResourceWEB-INF/web.xml/WatchedResource !-- Uncomment this to disable session persistence across Tomcat restarts -- Manager pathname= *maxActiveConnections=150 Are you referring to maxActiveSessions? https://tomcat.apache.org/tomcat-7.0-doc/config/manager.html */ !-- Uncomment this to enable Comet connection tacking (provides events on session expiration as well as webapp lifecycle) -- !-- Valve className=org.apache.catalina.valves.CometConnectionManagerValve / -- /Context what I find is. When I reach the limit, as expected the server will not create a new session, however does return a http 200 status code. I am wondering would it be possible for me to change this behavior to return say a 300 range code to redirect to (in a production environment) load balancer. I can see some potential issues with this, like the possibility of an infinite loop, but for now I am just exploring possible ways to prevent my tomcat servers becoming overloaded. How do you have your load balancer setup now? What software / hardware are you using? What algorithm is it using to distribute the load? Its an apache httpd server, distributing load using byRequest, with a redundant fail-over node configured howevr this only works if the tomcat server fails to accept the initial connection Does anyone have any thoughts or ideas for this type of solution? Perhaps you need to adjust the load balancer so it's spreading out the connections more evenly? Perhaps you need to add another node to handle the load? Its a recent issue, typical user response to a slow application (for whatever reason) is to open another session, however the jvmRoute is stored in the browser which forces the 2nd,3rd,4th session to the same host. We are looking at a application solution to provide a new session without the jvmRoute in separate tabs, but the ability to reject and redirect a user (even to a error page) when the server limit we set is reached would prevent a larger scale outage. You might try maxConnections on the HTTP connector. https://tomcat.apache.org/tomcat-7.0-doc/config/http.html or the AJP connector, depending on how your proxy is connecting to Tomcat. https://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will not accept any more connections until the number of connections falls below this value. Note this is different than the configuration you specified above, which appears like it's trying to restrict the number of sessions that the manager will create. Dan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- *Mitchell Smith * Developer *Cable Wireless Communications* Tel: +44 (0) 20 7315 4454 Mobile: +44 (0) 7921619263 www.cwc.com The information contained
Set context of web app to / tomcat7
Hi I am trying to set the context of the standard jira web app to / ; so my url would be http://hostname:8080/ and this would load the jira application. I have read the documentation, and found a couple of posts on this ( http://stackoverflow.com/questions/7276989/howto-set-the-context-path-of-a-web-application-in-tomcat-7-0) but cannot get this to work. Can anyone help me get this working? Or point me in the right direction. Thanks Mitchell The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Is it possible to expire jvmRoute cookie
Hi, I have a jvmRoute appended to my JSESSIONID to enable sticky sessions at an application level. I see often that users do not close browsers, resulting (even if the tomcat session has expired) the user to be forwarded back to the instance. Is it possible to set an expiry for the jvmRoute / JSESSIONID to be expired at the browser so after x amount of time since last request the jvmRoute is not transmitted back to the application apache load balancer. This would prevent users constantly being directed to the same application node. Regards -- *Mitchell * The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Exception in thread http-bio-8080-exec-3 Exception in thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
Hi, I am currently upgrading to use Java 7 and tomcat7. However when I try to execute a login in my WebApp I get the following error. Exception in thread http-bio-8080-exec-3 Exception in thread ContainerBackgroundProcessor[StandardEngine[Catalina]] I am using: Java jdk7 (update4) Tomcat 7.0.27 My Application is an Icefaces3.0.1 JSF2 application. Any detail on what this error means would be greatly appreciated. Thanks -- *Mitchell Smith * The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
java.lang.IllegalStateException Question
Hi, Tomcat 5.5.35 JAVA 5 I am seeing: [ERROR] org.apache.catalina.core.ContainerBase.[Catalina].[localhost] - Exception Processing ErrorPage[exceptionType=java.lang.Exception, location=/base/errors/runtimeError.iface] java.lang.IllegalStateException at org.apache.coyote.Response.reset(Response.java:297) I have looked in to this issue and found multiple comments relating to the maxPostSize Attribute in the connector. Can someone provide me with a simple explaination of the cause of this error, by default this value is 2mb, does this error indicate that somewhere a large data packet is being sent to my web app? Or could it be that when communicating to a webservice layer it is posting / receiving a large packet? Background, this error is appearing in the frontend application servers, the webapplication utalises a set of axis2 webservices. Any info or assistance would be greatly appreciated. Thanks -- *Mitchell Smith * The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Tomcat 5.5.35 as Windows Service Java Opts
Hi, I am trying to configure some additional parameters for a tomcat installation Tomcat5.5.35 Windows Server 2003 Due to the nature of the WebService to be hosted on this tomcat installation I would like to increase the Eden heap size. With my previous Linux Implementations adding -XX:NewSize=1234m to my java opts worked, however if i try to add this to the service in Windows, the value appears to be ignored. [image: image.png] It appears in the VM arguments, but the value is not propogated to the Heap Configurations. [image: image.png] [image: image.png] *Max: 838,912 kbytes* * * I appreciate any assistance in this matter. Regards -- *Mitchell Smith* The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Apache Tomcat 7 JMX Monitoring
Hi, I am configuring an application server: Tomcat 7, Java 1.7 I previously used Apache Tomcat 5.5, and JDK1.5. I had JMX monitoring enabled as below. -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false With this implementation JMX exports JMX RMI connection objects exposed on a 'Random Port', meaning that for monitoring of a remote tomcat to be available, any firewall must allow access to all ports. I would like to have the JMX RMI objects returned on a set port. Do newer versions of tomcat address this issue, allowing me to set the port on which the JXM monitoring will respond. Any information would be greatly appreciated Thanks -- *Mitchell Smith * The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ
Concurrent Low Pause GC - Eden Space Size (Apache Tomcat/5.5.27)
Hi, Tomcat Version 5.5.27 OS: Linux Red Hat Linux 5.3 Enterprise 64bit I have a tomcat server setup, with the following configuration added: -XX:+UseLargePages -Xms14336m -Xmx14336m -XX:NewRatio=4 -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=51 -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=75 -XX:CMSIncrementalDutyCycleMin=20 -XX:CMSIncrementalSafetyFactor=10 -XX:CMSIncrementalOffset=0 -XX:CMSExpAvgFactor=25 When I check the JVM output via jConsole the Eden space only has an available max space of 10496kbytes, I believe that my configuration of -(XX:NewRatio=4) should effectively set this parameter to 20% of the total available heap (-Xmx14336m) - should be approx 2936000kbytes. This problem, creates an issue that the Eden space GC runs frequently, and the time available between these minor GC's is to short for the Tenured space to be effectively collected. Any Help or suggestions would be greatly appreciated. Thank You, -- Mitchell Smith The information contained in this email (and any attachments) is confidential and may be privileged. If you are not the intended recipient and have received this email in error, please notify the sender immediately by reply email and delete the message and any attachments. If you are not the named addressee, you must not copy, disclose, forward or otherwise use the information contained in this email. Cable Wireless Communications Plc and its affiliates reserve the right to monitor all email communications through their networks to ensure regulatory compliance. Cable Wireless Communications Plc is a company registered in England Wales with number: 07130199 and offices located at 3rd Floor, 26 Red Lion Square, London WC1R 4HQ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
