Re: Tomcat Thread issue

2015-04-29 Thread Subhro Paul
-Christopher Schultz ch...@christopherschultz.net wrote: -
To: Tomcat Users List users@tomcat.apache.org
From: Christopher Schultz ch...@christopherschultz.net
Date: 04/24/2015 07:14PM
Subject: Re: Tomcat Thread issue

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Felix,

On 4/24/15 3:19 AM, Felix Schumacher wrote:
 Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
 subhro.p...@tcs.com:
 
 
 -Subhro Paul subhro.p...@tcs.com wrote: - To:
 users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com 
 Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
 
 -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat
 Users List users@tomcat.apache.org From: Daniel Mikusa
 dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat
 Thread issue
 
 On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
 subhro.p...@tcs.com wrote:
 
 Dear Team,
 
 One of our client's website stopped working yesterday. We
 observed
 that
 Tomcat servers were not working properly during that time. We
 have
 checked
 the memory usage of the server was fine but in the Catalina.out
 log
 we
 found it was already reached to max thread which is 512 though
 the
 number
 of connections to the server was normal. We took a thread dump
 from
 the
 server using VisualVM and we got the below message from
 threaddump:
 
 
 Since a thread dump is a point in time snapshot, you should
 always take multiple thread dumps, with a few seconds in between
 each one.  This gives you additional perspective as to what's
 happening with the threads over a period of time.
 
 
 
 http-8080-1 - Thread t@22
 
 java.lang.Thread.State: BLOCKED
 
 at java.util.Vector$1.nextElement(Vector.java:320)
 
 - waiting to lock 37749687 (a java.util.Vector) owned
 by
 http-8080-116 t@161
 
 at 
 org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
)


 
at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)



 
Look at what header.jsp is doing.  It seems to be doing something with
 the Vector class which is causing the thread to block.
 
 
 
 at 
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)


 
at
 
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)


 
at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)


 
at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)


 
at
 
 org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:646)


 
at
 
 org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:551)


 
at
 
 org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:488)


 
at
 
 org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
.java:968)


 
at
 
 org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer
vice(my_005fbill_jsp.java:126)


 
at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)


 
at
 
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)


 
at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)


 
at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)


 
at
 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)


 
at
 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)


 
at
 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:127)


 
at
 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)


 
at
 
 org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa
lve.java:269)


 
at
 
 org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav
a:81)


 
at
 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
555)


 
at
 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)


 
at
 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:298)


 
at
 
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:857)


 
at
 
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:588)


 
at
 
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:48
9

Re: Tomcat Thread issue

2015-04-29 Thread Felix Schumacher


Am 29. April 2015 14:54:36 MESZ, schrieb Subhro Paul subhro.p...@tcs.com:
-Christopher Schultz ch...@christopherschultz.net wrote: -
To: Tomcat Users List users@tomcat.apache.org
From: Christopher Schultz ch...@christopherschultz.net
Date: 04/24/2015 07:14PM
Subject: Re: Tomcat Thread issue

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Felix,

On 4/24/15 3:19 AM, Felix Schumacher wrote:
 Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
 subhro.p...@tcs.com:
 
 
 -Subhro Paul subhro.p...@tcs.com wrote: - To:
 users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com 
 Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
 
 -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat
 Users List users@tomcat.apache.org From: Daniel Mikusa
 dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat
 Thread issue
 
 On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
 subhro.p...@tcs.com wrote:
 
 Dear Team,
 
 One of our client's website stopped working yesterday. We
 observed
 that
 Tomcat servers were not working properly during that time. We
 have
 checked
 the memory usage of the server was fine but in the Catalina.out
 log
 we
 found it was already reached to max thread which is 512 though
 the
 number
 of connections to the server was normal. We took a thread dump
 from
 the
 server using VisualVM and we got the below message from
 threaddump:
 
 
 Since a thread dump is a point in time snapshot, you should
 always take multiple thread dumps, with a few seconds in between
 each one.  This gives you additional perspective as to what's
 happening with the threads over a period of time.
 
 
 
 http-8080-1 - Thread t@22
 
 java.lang.Thread.State: BLOCKED
 
 at java.util.Vector$1.nextElement(Vector.java:320)
 
 - waiting to lock 37749687 (a java.util.Vector) owned
 by
 http-8080-116 t@161
 
 at 

org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
)


 
at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)



 
Look at what header.jsp is doing.  It seems to be doing something with
 the Vector class which is causing the thread to block.
 
 
 
 at 
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 

org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)


 
at
 

org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)


 
at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)


 
at
 

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)


 
at
 

org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:646)


 
at
 

org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:551)


 
at
 

org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:488)


 
at
 

org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
.java:968)


 
at
 

org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer
vice(my_005fbill_jsp.java:126)


 
at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 

org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)


 
at
 

org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)


 
at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)


 
at
 

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)


 
at
 

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)


 
at
 

org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)


 
at
 

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:127)


 
at
 

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)


 
at
 

org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa
lve.java:269)


 
at
 

org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav
a:81)


 
at
 

org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
555)


 
at
 

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)


 
at
 

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:298)


 
at
 

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:857)


 
at
 

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:588

Re: Tomcat Thread issue

2015-04-24 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Felix,

On 4/24/15 3:19 AM, Felix Schumacher wrote:
 Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
 subhro.p...@tcs.com:
 
 
 -Subhro Paul subhro.p...@tcs.com wrote: - To:
 users@tomcat.apache.org From: Subhro Paul subhro.p...@tcs.com 
 Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
 
 -Daniel Mikusa dmik...@pivotal.io wrote: - To: Tomcat
 Users List users@tomcat.apache.org From: Daniel Mikusa
 dmik...@pivotal.io Date: 04/23/2015 05:01PM Subject: Re: Tomcat
 Thread issue
 
 On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
 subhro.p...@tcs.com wrote:
 
 Dear Team,
 
 One of our client's website stopped working yesterday. We
 observed
 that
 Tomcat servers were not working properly during that time. We
 have
 checked
 the memory usage of the server was fine but in the Catalina.out
 log
 we
 found it was already reached to max thread which is 512 though
 the
 number
 of connections to the server was normal. We took a thread dump
 from
 the
 server using VisualVM and we got the below message from
 threaddump:
 
 
 Since a thread dump is a point in time snapshot, you should
 always take multiple thread dumps, with a few seconds in between
 each one.  This gives you additional perspective as to what's
 happening with the threads over a period of time.
 
 
 
 http-8080-1 - Thread t@22
 
 java.lang.Thread.State: BLOCKED
 
 at java.util.Vector$1.nextElement(Vector.java:320)
 
 - waiting to lock 37749687 (a java.util.Vector) owned
 by
 http-8080-116 t@161
 
 at 
 org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
)


 
at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)



 
Look at what header.jsp is doing.  It seems to be doing something with
 the Vector class which is causing the thread to block.
 
 
 
 at 
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)


 
at
 
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)


 
at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)


 
at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)


 
at
 
 org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:646)


 
at
 
 org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:551)


 
at
 
 org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:488)


 
at
 
 org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
.java:968)


 
at
 
 org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer
vice(my_005fbill_jsp.java:126)


 
at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)


 
at
 
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)


 
at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)


 
at
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 
 at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)


 
at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)


 
at
 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)


 
at
 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)


 
at
 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:127)


 
at
 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)


 
at
 
 org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa
lve.java:269)


 
at
 
 org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav
a:81)


 
at
 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
555)


 
at
 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)


 
at
 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:298)


 
at
 
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:857)


 
at
 
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:588)


 
at
 
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:48
9)


 
at java.lang.Thread.run(Thread.java:701)
 
 
 
 Locked ownable synchronizers:
 
 -  None
 
 
 
 This was coming for different threads. Once we restarted the
 servers,
 the
 website back to normal again but we got

Re: Tomcat Thread issue

2015-04-24 Thread Subhro Paul


-Subhro Paul subhro.p...@tcs.com wrote: -
To: users@tomcat.apache.org
From: Subhro Paul subhro.p...@tcs.com
Date: 04/23/2015 06:20PM
Subject: Re: Tomcat Thread issue

-Daniel Mikusa dmik...@pivotal.io wrote: -
To: Tomcat Users List users@tomcat.apache.org
From: Daniel Mikusa dmik...@pivotal.io
Date: 04/23/2015 05:01PM
Subject: Re: Tomcat Thread issue

On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote:

 Dear Team,

 One of our client's website stopped working yesterday. We observed that
 Tomcat servers were not working properly during that time. We have checked
 the memory usage of the server was fine but in the Catalina.out log we
 found it was already reached to max thread which is 512 though the number
 of connections to the server was normal. We took a thread dump from the
 server using VisualVM and we got the below message from threaddump:


Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This gives
you additional perspective as to what's happening with the threads over a
period of time.



 http-8080-1 - Thread t@22

    java.lang.Thread.State: BLOCKED

             at java.util.Vector$1.nextElement(Vector.java:320)

             - waiting to lock 37749687 (a java.util.Vector) owned by
 http-8080-116 t@161

             at
 org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)

             at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)


Look at what header.jsp is doing.  It seems to be doing something with the
Vector class which is causing the thread to block.



             at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

             at
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

             at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

             at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

             at
 org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)

             at
 org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)

             at
 org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)

             at
 org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)

             at
 org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)

             at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

             at
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

             at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

             at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

             at
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

             at
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

             at
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

             at
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

             at
 org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)

             at
 org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)

             at
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

             at
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

             at
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

             at
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)

             at
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

             at
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

             at java.lang.Thread.run(Thread.java:701)



    Locked ownable synchronizers:

 -          None



 This was coming for different threads

Re: Tomcat Thread issue

2015-04-24 Thread Felix Schumacher


Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul subhro.p...@tcs.com:


-Subhro Paul subhro.p...@tcs.com wrote: -
To: users@tomcat.apache.org
From: Subhro Paul subhro.p...@tcs.com
Date: 04/23/2015 06:20PM
Subject: Re: Tomcat Thread issue

-Daniel Mikusa dmik...@pivotal.io wrote: -
To: Tomcat Users List users@tomcat.apache.org
From: Daniel Mikusa dmik...@pivotal.io
Date: 04/23/2015 05:01PM
Subject: Re: Tomcat Thread issue

On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com
wrote:

 Dear Team,

 One of our client's website stopped working yesterday. We observed
that
 Tomcat servers were not working properly during that time. We have
checked
 the memory usage of the server was fine but in the Catalina.out log
we
 found it was already reached to max thread which is 512 though the
number
 of connections to the server was normal. We took a thread dump from
the
 server using VisualVM and we got the below message from threaddump:


Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This
gives
you additional perspective as to what's happening with the threads over
a
period of time.



 http-8080-1 - Thread t@22

    java.lang.Thread.State: BLOCKED

             at java.util.Vector$1.nextElement(Vector.java:320)

             - waiting to lock 37749687 (a java.util.Vector) owned
by
 http-8080-116 t@161

             at
 org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)

             at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)


Look at what header.jsp is doing.  It seems to be doing something with
the
Vector class which is causing the thread to block.



             at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

             at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at

org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

             at

org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

             at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

             at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

             at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

             at

org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)

             at

org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)

             at

org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)

             at

org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)

             at

org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)

             at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

             at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at

org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

             at

org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

             at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

             at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

             at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

             at

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

             at

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

             at

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

             at

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

             at

org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)

             at

org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)

             at

org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

             at

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

             at

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

             at

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)

             at

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

             at

org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

             at java.lang.Thread.run(Thread.java:701)



    Locked ownable

Tomcat Thread issue

2015-04-23 Thread Subhro Paul
Dear Team, 

One of our client's website stopped working yesterday. We observed that Tomcat 
servers were not working properly during that time. We have checked the memory 
usage of the server was fine but in the Catalina.out log we found it was 
already reached to max thread which is 512 though the number of connections to 
the server was normal. We took a thread dump from the server using VisualVM and 
we got the below message from threaddump: 

http-8080-1 - Thread t@22

   java.lang.Thread.State: BLOCKED

    at java.util.Vector$1.nextElement(Vector.java:320)

    - waiting to lock 37749687 (a java.util.Vector) owned by 
http-8080-116 t@161

    at 
org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)

    at 
org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)

    at 
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

    at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

    at 
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)

    at 
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)

    at 
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)

    at 
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)

    at 
org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)

    at 
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

    at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

    at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

    at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

    at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

    at 
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)

    at 
org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)

    at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

    at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

    at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

    at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)

    at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

    at java.lang.Thread.run(Thread.java:701)

 

   Locked ownable synchronizers:

-  None

  

This was coming for different threads. Once we restarted the servers, the 
website back to normal again but we got the below exception in the log :

 

Apr 22, 2015 11:15:28 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: A web application appears to have started a thread named [http-8080-1] 
but has failed to stop it. This is very likely to create a memory leak. 

 

So, we want to know while the thread is blocked is it like deadlock condition 
for which all threads were unavailable? Current thread count I about 190 but 
after few days this will reach to 500+ again even if the concurrent users are 
not high. Memory usage of the server was normal during this issue. This problem 
is happening from last 2 3 months.



Thanks  Regards,

Subhro Paul

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it 

Re: Tomcat Thread issue

2015-04-23 Thread Subhro Paul
-Daniel Mikusa dmik...@pivotal.io wrote: -
To: Tomcat Users List users@tomcat.apache.org
From: Daniel Mikusa dmik...@pivotal.io
Date: 04/23/2015 05:01PM
Subject: Re: Tomcat Thread issue

On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote:

 Dear Team,

 One of our client's website stopped working yesterday. We observed that
 Tomcat servers were not working properly during that time. We have checked
 the memory usage of the server was fine but in the Catalina.out log we
 found it was already reached to max thread which is 512 though the number
 of connections to the server was normal. We took a thread dump from the
 server using VisualVM and we got the below message from threaddump:


Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This gives
you additional perspective as to what's happening with the threads over a
period of time.



 http-8080-1 - Thread t@22

    java.lang.Thread.State: BLOCKED

             at java.util.Vector$1.nextElement(Vector.java:320)

             - waiting to lock 37749687 (a java.util.Vector) owned by
 http-8080-116 t@161

             at
 org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)

             at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)


Look at what header.jsp is doing.  It seems to be doing something with the
Vector class which is causing the thread to block.



             at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

             at
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

             at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

             at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

             at
 org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)

             at
 org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)

             at
 org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)

             at
 org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)

             at
 org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)

             at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

             at
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

             at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

             at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

             at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

             at
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

             at
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

             at
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

             at
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

             at
 org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)

             at
 org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)

             at
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

             at
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

             at
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

             at
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)

             at
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

             at
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

             at java.lang.Thread.run(Thread.java:701)



    Locked ownable synchronizers:

 -          None



 This was coming for different threads. Once we restarted the servers, the
 website back to normal again but we got the below exception in the log :



 Apr 22, 2015 11:15:28 AM

Re: Tomcat Thread issue

2015-04-23 Thread Daniel Mikusa
On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul subhro.p...@tcs.com wrote:

 Dear Team,

 One of our client's website stopped working yesterday. We observed that
 Tomcat servers were not working properly during that time. We have checked
 the memory usage of the server was fine but in the Catalina.out log we
 found it was already reached to max thread which is 512 though the number
 of connections to the server was normal. We took a thread dump from the
 server using VisualVM and we got the below message from threaddump:


Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This gives
you additional perspective as to what's happening with the threads over a
period of time.



 http-8080-1 - Thread t@22

java.lang.Thread.State: BLOCKED

 at java.util.Vector$1.nextElement(Vector.java:320)

 - waiting to lock 37749687 (a java.util.Vector) owned by
 http-8080-116 t@161

 at
 org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)

 at
 org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)


Look at what header.jsp is doing.  It seems to be doing something with the
Vector class which is causing the thread to block.



 at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

 at
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

 at
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

 at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

 at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

 at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

 at
 org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)

 at
 org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)

 at
 org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)

 at
 org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)

 at
 org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)

 at
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

 at
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)

 at
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

 at
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

 at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

 at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

 at
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

 at
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

 at
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

 at
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

 at
 org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)

 at
 org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)

 at
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

 at
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

 at
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

 at
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)

 at
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

 at
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

 at java.lang.Thread.run(Thread.java:701)



Locked ownable synchronizers:

 -  None



 This was coming for different threads. Once we restarted the servers, the
 website back to normal again but we got the below exception in the log :



 Apr 22, 2015 11:15:28 AM org.apache.catalina.loader.WebappClassLoader
 clearReferencesThreads

 SEVERE: A web application appears to have started a thread named
 [http-8080-1] but has failed to stop it. This is very likely to create a
 memory leak.


This