Re: Tomcat Thread issue
-Christopher Schultz ch...@christopherschultz.net wrote: - To: Tomcat Users List users@tomcat.apache.org From: Christopher Schultz ch...@christopherschultz.net Date: 04/24/2015 07:14PM Subject: Re: Tomcat Thread issue -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 4/24/15 3:19 AM, Felix Schumacher wrote: Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul subhro.p...@tcs.com: -Subhro Paul subhro.p...@tcs.com wrote: - To: users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat Users List users@tomcat.apache.org From: Daniel Mikusa dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat Thread issue On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116 ) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper .java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 13) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp atcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD ispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis patcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary .java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer vice(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper .java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 13) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV alve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV alve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j ava:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j ava:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa lve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav a:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java: 555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal ve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav a:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java :857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce ss(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:48 9
Re: Tomcat Thread issue
Am 29. April 2015 14:54:36 MESZ, schrieb Subhro Paul subhro.p...@tcs.com: -Christopher Schultz ch...@christopherschultz.net wrote: - To: Tomcat Users List users@tomcat.apache.org From: Christopher Schultz ch...@christopherschultz.net Date: 04/24/2015 07:14PM Subject: Re: Tomcat Thread issue -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 4/24/15 3:19 AM, Felix Schumacher wrote: Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul subhro.p...@tcs.com: -Subhro Paul subhro.p...@tcs.com wrote: - To: users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat Users List users@tomcat.apache.org From: Daniel Mikusa dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat Thread issue On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116 ) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper .java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 13) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp atcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD ispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis patcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary .java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer vice(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper .java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 13) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV alve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV alve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j ava:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j ava:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa lve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav a:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java: 555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal ve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav a:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java :857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce ss(Http11Protocol.java:588
Re: Tomcat Thread issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 4/24/15 3:19 AM, Felix Schumacher wrote: Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul subhro.p...@tcs.com: -Subhro Paul subhro.p...@tcs.com wrote: - To: users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat Users List users@tomcat.apache.org From: Daniel Mikusa dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat Thread issue On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116 ) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper .java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 13) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp atcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD ispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis patcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary .java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer vice(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper .java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 13) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV alve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV alve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j ava:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j ava:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa lve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav a:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java: 555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal ve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav a:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java :857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce ss(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:48 9) at java.lang.Thread.run(Thread.java:701) Locked ownable synchronizers: - None This was coming for different threads. Once we restarted the servers, the website back to normal again but we got
Re: Tomcat Thread issue
-Subhro Paul subhro.p...@tcs.com wrote: - To: users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat Users List users@tomcat.apache.org From: Daniel Mikusa dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat Thread issue On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:701) Locked ownable synchronizers: - None This was coming for different threads
Re: Tomcat Thread issue
Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul subhro.p...@tcs.com: -Subhro Paul subhro.p...@tcs.com wrote: - To: users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat Users List users@tomcat.apache.org From: Daniel Mikusa dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat Thread issue On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:701) Locked ownable
Re: Tomcat Thread issue
-Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat Users List users@tomcat.apache.org From: Daniel Mikusa dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat Thread issue On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:701) Locked ownable synchronizers: - None This was coming for different threads. Once we restarted the servers, the website back to normal again but we got the below exception in the log : Apr 22, 2015 11:15:28 AM
Re: Tomcat Thread issue
On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote: Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump: Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. http-8080-1 - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock 37749687 (a java.util.Vector) owned by http-8080-116 t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:701) Locked ownable synchronizers: - None This was coming for different threads. Once we restarted the servers, the website back to normal again but we got the below exception in the log : Apr 22, 2015 11:15:28 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: A web application appears to have started a thread named [http-8080-1] but has failed to stop it. This is very likely to create a memory leak. This
