Re: Tomcat Thread issue

2015-04-29 Thread Felix Schumacher


Am 29. April 2015 14:54:36 MESZ, schrieb Subhro Paul :
>-Christopher Schultz  wrote: -
>To: Tomcat Users List 
>From: Christopher Schultz 
>Date: 04/24/2015 07:14PM
>Subject: Re: Tomcat Thread issue
>
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA256
>
>Felix,
>
>On 4/24/15 3:19 AM, Felix Schumacher wrote:
>> Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
>> :
>>> 
>>> 
>>> -Subhro Paul  wrote: - To:
>>> users@tomcat.apache.org From: Subhro Paul  
>>> Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
>>> 
>>> -Daniel Mikusa  wrote: ----- To: Tomcat
>>> Users List  From: Daniel Mikusa
>>>  Date: 04/23/2015 05:01PM Subject: Re: Tomcat
>>> Thread issue
>>> 
>>> On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
>>>  wrote:
>>> 
>>>> Dear Team,
>>>> 
>>>> One of our client's website stopped working yesterday. We
>>>> observed
>>> that
>>>> Tomcat servers were not working properly during that time. We
>>>> have
>>> checked
>>>> the memory usage of the server was fine but in the Catalina.out
>>>> log
>>> we
>>>> found it was already reached to max thread which is 512 though
>>>> the
>>> number
>>>> of connections to the server was normal. We took a thread dump
>>>> from
>>> the
>>>> server using VisualVM and we got the below message from
>>>> threaddump:
>>>> 
>>> 
>>> Since a thread dump is a point in time snapshot, you should
>>> always take multiple thread dumps, with a few seconds in between
>>> each one.  This gives you additional perspective as to what's
>>> happening with the threads over a period of time.
>>> 
>>> 
>>>> 
>>>> "http-8080-1" - Thread t@22
>>>> 
>>>> java.lang.Thread.State: BLOCKED
>>>> 
>>>> at java.util.Vector$1.nextElement(Vector.java:320)
>>>> 
>>>> - waiting to lock <37749687> (a java.util.Vector) owned
>>> by
>>>> "http-8080-116" t@161
>>>> 
>>>> at 
>>>>
>org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
>)
>>>>
>>>>
>>>> 
>at
>>>> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>>>>
>>>
>>>
>>>> 
>Look at what header.jsp is doing.  It seems to be doing something with
>>> the Vector class which is causing the thread to block.
>>> 
>>> 
>>>> 
>>>> at 
>>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>
>>>>
>>>> 
>at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>> 
>>>> at
>>>> 
>>>
>org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
>.java:377)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
>org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
>13)
>>>>
>>>>
>>> 
>at
>>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>>>
>>>>
>>>> 
>at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>> 
>>>> at
>>>> 
>>>
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
>icationFilterChain.java:290)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
>ilterChain.java:206)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
>org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
>atcher.java:646)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
>org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
>ispatcher.java:551)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
>org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
>patcher.java:488)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
>org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
>.java:968)
>>>>
>>>>
>>> 
>at
>>>> 
>>>
&

Re: Tomcat Thread issue

2015-04-29 Thread Subhro Paul
-Christopher Schultz  wrote: -
To: Tomcat Users List 
From: Christopher Schultz 
Date: 04/24/2015 07:14PM
Subject: Re: Tomcat Thread issue

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Felix,

On 4/24/15 3:19 AM, Felix Schumacher wrote:
> Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
> :
>> 
>> 
>> -Subhro Paul  wrote: - To:
>> users@tomcat.apache.org From: Subhro Paul  
>> Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
>> 
>> -Daniel Mikusa  wrote: - To: Tomcat
>> Users List  From: Daniel Mikusa
>>  Date: 04/23/2015 05:01PM Subject: Re: Tomcat
>> Thread issue
>> 
>> On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
>>  wrote:
>> 
>>> Dear Team,
>>> 
>>> One of our client's website stopped working yesterday. We
>>> observed
>> that
>>> Tomcat servers were not working properly during that time. We
>>> have
>> checked
>>> the memory usage of the server was fine but in the Catalina.out
>>> log
>> we
>>> found it was already reached to max thread which is 512 though
>>> the
>> number
>>> of connections to the server was normal. We took a thread dump
>>> from
>> the
>>> server using VisualVM and we got the below message from
>>> threaddump:
>>> 
>> 
>> Since a thread dump is a point in time snapshot, you should
>> always take multiple thread dumps, with a few seconds in between
>> each one.  This gives you additional perspective as to what's
>> happening with the threads over a period of time.
>> 
>> 
>>> 
>>> "http-8080-1" - Thread t@22
>>> 
>>> java.lang.Thread.State: BLOCKED
>>> 
>>> at java.util.Vector$1.nextElement(Vector.java:320)
>>> 
>>> - waiting to lock <37749687> (a java.util.Vector) owned
>> by
>>> "http-8080-116" t@161
>>> 
>>> at 
>>> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
)
>>>
>>>
>>> 
at
>>> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>>>
>>
>>
>>> 
Look at what header.jsp is doing.  It seems to be doing something with
>> the Vector class which is causing the thread to block.
>> 
>> 
>>> 
>>> at 
>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
>>>
>>>
>> 
at
>>> 
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
>>>
>>>
>> 
at
>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:646)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:551)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:488)
>>>
>>>
>> 
at
>>> 
>> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
.java:968)
>>>
>>>
>> 
at
>>> 
>> org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer
vice(my_005fbill_jsp.java:126)
>>>
>>>
>> 
at
>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
>>>
>>>
>> 
at
>>> 
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
>>>
>>>
>> 
at
>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.

Re: Tomcat Thread issue

2015-04-24 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Felix,

On 4/24/15 3:19 AM, Felix Schumacher wrote:
> Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
> :
>> 
>> 
>> -Subhro Paul  wrote: - To:
>> users@tomcat.apache.org From: Subhro Paul  
>> Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
>> 
>> -Daniel Mikusa  wrote: - To: Tomcat
>> Users List  From: Daniel Mikusa
>>  Date: 04/23/2015 05:01PM Subject: Re: Tomcat
>> Thread issue
>> 
>> On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
>>  wrote:
>> 
>>> Dear Team,
>>> 
>>> One of our client's website stopped working yesterday. We
>>> observed
>> that
>>> Tomcat servers were not working properly during that time. We
>>> have
>> checked
>>> the memory usage of the server was fine but in the Catalina.out
>>> log
>> we
>>> found it was already reached to max thread which is 512 though
>>> the
>> number
>>> of connections to the server was normal. We took a thread dump
>>> from
>> the
>>> server using VisualVM and we got the below message from
>>> threaddump:
>>> 
>> 
>> Since a thread dump is a point in time snapshot, you should
>> always take multiple thread dumps, with a few seconds in between
>> each one.  This gives you additional perspective as to what's
>> happening with the threads over a period of time.
>> 
>> 
>>> 
>>> "http-8080-1" - Thread t@22
>>> 
>>> java.lang.Thread.State: BLOCKED
>>> 
>>> at java.util.Vector$1.nextElement(Vector.java:320)
>>> 
>>> - waiting to lock <37749687> (a java.util.Vector) owned
>> by
>>> "http-8080-116" t@161
>>> 
>>> at 
>>> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
)
>>>
>>>
>>> 
at
>>> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>>>
>>
>>
>>> 
Look at what header.jsp is doing.  It seems to be doing something with
>> the Vector class which is causing the thread to block.
>> 
>> 
>>> 
>>> at 
>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
>>>
>>>
>> 
at
>>> 
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
>>>
>>>
>> 
at
>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:646)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:551)
>>>
>>>
>> 
at
>>> 
>> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:488)
>>>
>>>
>> 
at
>>> 
>> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
.java:968)
>>>
>>>
>> 
at
>>> 
>> org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer
vice(my_005fbill_jsp.java:126)
>>>
>>>
>> 
at
>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
>>>
>>>
>> 
at
>>> 
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
>>>
>>>
>> 
at
>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>>
>>>
>>> 
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>> 
>>> at
>>> 
>> org.apa

Re: Tomcat Thread issue

2015-04-24 Thread Felix Schumacher


Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul :
>
>
>-Subhro Paul  wrote: -
>To: users@tomcat.apache.org
>From: Subhro Paul 
>Date: 04/23/2015 06:20PM
>Subject: Re: Tomcat Thread issue
>
>-Daniel Mikusa  wrote: -
>To: Tomcat Users List 
>From: Daniel Mikusa 
>Date: 04/23/2015 05:01PM
>Subject: Re: Tomcat Thread issue
>
>On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul 
>wrote:
>
>> Dear Team,
>>
>> One of our client's website stopped working yesterday. We observed
>that
>> Tomcat servers were not working properly during that time. We have
>checked
>> the memory usage of the server was fine but in the Catalina.out log
>we
>> found it was already reached to max thread which is 512 though the
>number
>> of connections to the server was normal. We took a thread dump from
>the
>> server using VisualVM and we got the below message from threaddump:
>>
>
>Since a thread dump is a point in time snapshot, you should always take
>multiple thread dumps, with a few seconds in between each one.  This
>gives
>you additional perspective as to what's happening with the threads over
>a
>period of time.
>
>
>>
>> "http-8080-1" - Thread t@22
>>
>>    java.lang.Thread.State: BLOCKED
>>
>>             at java.util.Vector$1.nextElement(Vector.java:320)
>>
>>             - waiting to lock <37749687> (a java.util.Vector) owned
>by
>> "http-8080-116" t@161
>>
>>             at
>> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)
>>
>>             at
>> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>>
>
>Look at what header.jsp is doing.  It seems to be doing something with
>the
>Vector class which is causing the thread to block.
>
>
>>
>>             at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>
>>             at
>javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>
>>             at
>>
>org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>>
>>             at
>>
>org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>>
>>             at
>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>
>>             at
>javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
>>
>>             at
>>
>org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
>>
>>             at
>>
>org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)
>>
>>             at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>
>>             at
>javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>
>>             at
>>
>org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>>
>>             at
>>
>org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>>
>>             at
>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>
>>             at
>javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>
>>             at
>>
>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>
>>             at
>>
>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>
>>             at
>>
>org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>
>>             at
>>
>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>
>>             at
>>
>org.apache.catalina.

Re: Tomcat Thread issue

2015-04-24 Thread Subhro Paul


-Subhro Paul  wrote: -
To: users@tomcat.apache.org
From: Subhro Paul 
Date: 04/23/2015 06:20PM
Subject: Re: Tomcat Thread issue

-Daniel Mikusa  wrote: -
To: Tomcat Users List 
From: Daniel Mikusa 
Date: 04/23/2015 05:01PM
Subject: Re: Tomcat Thread issue

On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul  wrote:

> Dear Team,
>
> One of our client's website stopped working yesterday. We observed that
> Tomcat servers were not working properly during that time. We have checked
> the memory usage of the server was fine but in the Catalina.out log we
> found it was already reached to max thread which is 512 though the number
> of connections to the server was normal. We took a thread dump from the
> server using VisualVM and we got the below message from threaddump:
>

Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This gives
you additional perspective as to what's happening with the threads over a
period of time.


>
> "http-8080-1" - Thread t@22
>
>    java.lang.Thread.State: BLOCKED
>
>             at java.util.Vector$1.nextElement(Vector.java:320)
>
>             - waiting to lock <37749687> (a java.util.Vector) owned by
> "http-8080-116" t@161
>
>             at
> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)
>
>             at
> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>

Look at what header.jsp is doing.  It seems to be doing something with the
Vector class which is causing the thread to block.


>
>             at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>
>             at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>
>             at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
>             at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
>
>             at
> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
>
>             at
> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
>
>             at
> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
>
>             at
> org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)
>
>             at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>
>             at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>
>             at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
>             at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>
>             at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>
>             at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>
>             at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>
>             at
> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
>
>             at
> org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)
>
>             at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>
>             at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>
>             at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>
>             at
> org.apache.coyote.http11.Http11Processor.process(

Re: Tomcat Thread issue

2015-04-23 Thread Subhro Paul
-Daniel Mikusa  wrote: -
To: Tomcat Users List 
From: Daniel Mikusa 
Date: 04/23/2015 05:01PM
Subject: Re: Tomcat Thread issue

On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul  wrote:

> Dear Team,
>
> One of our client's website stopped working yesterday. We observed that
> Tomcat servers were not working properly during that time. We have checked
> the memory usage of the server was fine but in the Catalina.out log we
> found it was already reached to max thread which is 512 though the number
> of connections to the server was normal. We took a thread dump from the
> server using VisualVM and we got the below message from threaddump:
>

Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This gives
you additional perspective as to what's happening with the threads over a
period of time.


>
> "http-8080-1" - Thread t@22
>
>    java.lang.Thread.State: BLOCKED
>
>             at java.util.Vector$1.nextElement(Vector.java:320)
>
>             - waiting to lock <37749687> (a java.util.Vector) owned by
> "http-8080-116" t@161
>
>             at
> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)
>
>             at
> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>

Look at what header.jsp is doing.  It seems to be doing something with the
Vector class which is causing the thread to block.


>
>             at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>
>             at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>
>             at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
>             at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
>
>             at
> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
>
>             at
> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
>
>             at
> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
>
>             at
> org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)
>
>             at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>
>             at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>
>             at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>
>             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>
>             at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
>             at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>
>             at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>
>             at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>
>             at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>
>             at
> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
>
>             at
> org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)
>
>             at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>
>             at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>
>             at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>
>             at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>
>             at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt

Re: Tomcat Thread issue

2015-04-23 Thread Daniel Mikusa
On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul  wrote:

> Dear Team,
>
> One of our client's website stopped working yesterday. We observed that
> Tomcat servers were not working properly during that time. We have checked
> the memory usage of the server was fine but in the Catalina.out log we
> found it was already reached to max thread which is 512 though the number
> of connections to the server was normal. We took a thread dump from the
> server using VisualVM and we got the below message from threaddump:
>

Since a thread dump is a point in time snapshot, you should always take
multiple thread dumps, with a few seconds in between each one.  This gives
you additional perspective as to what's happening with the threads over a
period of time.


>
> "http-8080-1" - Thread t@22
>
>java.lang.Thread.State: BLOCKED
>
> at java.util.Vector$1.nextElement(Vector.java:320)
>
> - waiting to lock <37749687> (a java.util.Vector) owned by
> "http-8080-116" t@161
>
> at
> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116)
>
> at
> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>

Look at what header.jsp is doing.  It seems to be doing something with the
Vector class which is causing the thread to block.


>
> at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>
> at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
>
> at
> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
>
> at
> org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126)
>
> at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
>
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>
> at
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>
> at
> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
>
> at
> org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81)
>
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>
> at java.lang.Thread.run(Thread.java:701)
>
>
>
>Locked ownable synchronizers:
>
> -  None
>
>
>
> This was coming for different threads. Once we restarted the servers, the
> website back to normal again but we got the below exception in the log :
>
>
>
> Apr 22, 2015 11:15:28 AM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
>
> SEVERE: A web application appears to