Am 29. April 2015 14:54:36 MESZ, schrieb Subhro Paul <subhro.p...@tcs.com>: >-----Christopher Schultz <ch...@christopherschultz.net> wrote: ----- >To: Tomcat Users List <users@tomcat.apache.org> >From: Christopher Schultz <ch...@christopherschultz.net> >Date: 04/24/2015 07:14PM >Subject: Re: Tomcat Thread issue > >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA256 > >Felix, > >On 4/24/15 3:19 AM, Felix Schumacher wrote: >> Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul >> <subhro.p...@tcs.com>: >>> >>> >>> -----Subhro Paul <subhro.p...@tcs.com> wrote: ----- To: >>> users@tomcat.apache.org From: Subhro Paul <subhro.p...@tcs.com> >>> Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue >>> >>> -----Daniel Mikusa <dmik...@pivotal.io> wrote: ----- To: Tomcat >>> Users List <users@tomcat.apache.org> From: Daniel Mikusa >>> <dmik...@pivotal.io> Date: 04/23/2015 05:01PM Subject: Re: Tomcat >>> Thread issue >>> >>> On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul >>> <subhro.p...@tcs.com> wrote: >>> >>>> Dear Team, >>>> >>>> One of our client's website stopped working yesterday. We >>>> observed >>> that >>>> Tomcat servers were not working properly during that time. We >>>> have >>> checked >>>> the memory usage of the server was fine but in the Catalina.out >>>> log >>> we >>>> found it was already reached to max thread which is 512 though >>>> the >>> number >>>> of connections to the server was normal. We took a thread dump >>>> from >>> the >>>> server using VisualVM and we got the below message from >>>> threaddump: >>>> >>> >>> Since a thread dump is a point in time snapshot, you should >>> always take multiple thread dumps, with a few seconds in between >>> each one. This gives you additional perspective as to what's >>> happening with the threads over a period of time. >>> >>> >>>> >>>> "http-8080-1" - Thread t@22 >>>> >>>> java.lang.Thread.State: BLOCKED >>>> >>>> at java.util.Vector$1.nextElement(Vector.java:320) >>>> >>>> - waiting to lock <37749687> (a java.util.Vector) owned >>> by >>>> "http-8080-116" t@161 >>>> >>>> at >>>> >org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116 >) >>>> >>>> >>>> >at >>>> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) >>>> >>> >>> >>>> >Look at what header.jsp is doing. It seems to be doing something with >>> the Vector class which is causing the thread to block. >>> >>> >>>> >>>> at >>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) >>>> >>>> >>>> >at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >>>> >>>> at >>>> >>> >org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper >.java:377) >>>> >>>> >>> >at >>>> >>> >org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 >13) >>>> >>>> >>> >at >>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) >>>> >>>> >>>> >at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >>>> >>>> at >>>> >>> >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl >icationFilterChain.java:290) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF >ilterChain.java:206) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp >atcher.java:646) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD >ispatcher.java:551) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis >patcher.java:488) >>>> >>>> >>> >at >>>> >>> >org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary >.java:968) >>>> >>>> >>> >at >>>> >>> >org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer >vice(my_005fbill_jsp.java:126) >>>> >>>> >>> >at >>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) >>>> >>>> >>>> >at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >>>> >>>> at >>>> >>> >org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper >.java:377) >>>> >>>> >>> >at >>>> >>> >org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3 >13) >>>> >>>> >>> >at >>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) >>>> >>>> >>>> >at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >>>> >>>> at >>>> >>> >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl >icationFilterChain.java:290) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF >ilterChain.java:206) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV >alve.java:233) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.StandardContextValve.invoke(StandardContextV >alve.java:191) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j >ava:127) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j >ava:102) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa >lve.java:269) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav >a:81) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java: >555) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal >ve.java:109) >>>> >>>> >>> >at >>>> >>> >org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav >a:298) >>>> >>>> >>> >at >>>> >>> >org.apache.coyote.http11.Http11Processor.process(Http11Processor.java >:857) >>>> >>>> >>> >at >>>> >>> >org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce >ss(Http11Protocol.java:588) >>>> >>>> >>> >at >>>> >>> >org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:48 >9) >>>> >>>> >>> >at java.lang.Thread.run(Thread.java:701) >>>> >>>> >>>> >>>> Locked ownable synchronizers: >>>> >>>> - ÿ ÿ ÿ ÿ ÿNone >>>> >>>> >>>> >>>> This was coming for different threads. Once we restarted the >>>> servers, >>> the >>>> website back to normal again but we got the below exception in >>>> the >>> log : >>>> >>>> >>>> >>>> Apr 22, 2015 11:15:28 AM >>>> org.apache.catalina.loader.WebappClassLoader >>>> clearReferencesThreads >>>> >>>> SEVERE: A web application appears to have started a thread >>>> named [http-8080-1] but has failed to stop it. This is very >>>> likely to >>> create a >>>> memory leak. >>>> >>>> >>> This means your app started a thread and failed to properly stop >>> it. ˜If you stopped the process completely, this is not going >to >>> cause any problems because the entire process would have exited >>> and so this thread would go away any way. ˜If you're doing >>> hot redeploy's (i.e. deplying apps without restarting the >>> process), this could cause problems. ˜Either way, when you >>> have a moment you should try to investigate what is creating >>> that thread and setup something to properly close it. >>> >>> >>>> >>>> >>>> So, we want to know while the thread is blocked is it like >>>> deadlock condition for which all threads were unavailable? >>> >>> >>> The thread dump simply lists the thread as "blocked". ˜This >>> generally means that at some point it will complete, so it's not >>> technically a deadlock. If you had multiple thread dumps, you >>> could watch the thread across all of them and see how it >>> progresses. >>> >>> >>>> Current thread count I about 190 but after few days this will >>>> reach >>> to >>>> 500+ again even if the concurrent users are not high. Memory >>>> usage of >>> the >>>> server was normal during this issue. This problem is happening >>>> from >>> last 2 >>>> 3 months. >>>> >>> >>> Look at the header.jsp, especially anything that has changed in >>> the last 2 - 3 months. >>> >>> Dan >>> >>> >>>> >>>> >>>> >>>> Thanks & Regards, >>>> >>>> Subhro Paul >>> >>> >>> Dear Team, >>> >>> Thanks for the reply. >>> >>> Next thing I want to know, creating a vector and adding objects >>> to the same in header.jsp which is loaded >>> and˜referred˜by˜almost˜all the JSP's(more >>> than 1000) in an application, will that leads to memory leak. >>> >>> Thanks & Regards, Subhro Paul >>> >>> >>> >>> Hi all, >>> >>> Is there any way to understand why the vector object was locked >>> by thread "http-8080-116" for long? From the thread dump we can >>> see there is around 140 threads were blocked because of the >>> thread "http-8080-116" acquired locked on the vector object.˜ >> >> Look at the full thread dump. There you will see where the lock is >> held. We can't tell you just from looking at the stacktrace. > >+1 > >My question would be what that Vector is being used for. If all >threads have to consult that Vector, and its synchronized (which is >it: it's a java.util.Vector), then every thread will be waiting on >every other thread. > >If this Vector is filled with read-only data, and the contents don't >need to be changed, then consider using another concrete List >implementation there. If you use ArrayList or LinkedList or whatever, >your page accesses will speed-up significantly. > >If I were you, I'd investigate the possibility of switching >implementations. I would also wrap the unsynchronized, read-only list >in a formally-read-only list implementation like this: > >public class Startup implements ServletContextListener { >ÿÿpublic void contextInitialized(...) { >ÿÿ ÿ List<Thing> listOfThings = new ArrayList<>(...); >ÿÿ ÿ // fill listOfThings with things >ÿÿ ÿ listOfThings = >java.util.Collections.unmodifiableList(listOfThings) >; >ÿÿ ÿ getServletContext().setAttribute("listOfThings"); >ÿÿ} >} > >This will give you a structure that will throw an error if you try to >modify it. > >//// > >If you ARE trying to modify it, you have to ask yourself why you are >using a single structure that every request coming into the >application is going to have to fight-over. > >In either case, can you explain the use-case here? > >- -chris >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v2 >Comment: GPGTools - http://gpgtools.org > > >Hi Team, >ÿ >While staring tomcat we can observeÿÿthrough Jconsole thatÿsome threads >are automatically created by Tomcat. Once we put load on the server, >depending on the load Tomcat increases the thread count but if we stop >putting load on the server we can see theÿstillÿThread count on the >server remain same even after long time. My assumption was it would >destroy some threads once there is a little load or no load which is >not the actual scenario. We can see that the unused threads are in >sleep mode. Is it a normal feature of Tomcat or some configuration is >required in Tomcat to destroy inactive threads. FYI we are using >“HTTP 1.1” connector.
Have a look at https://tomcat.apache.org/tomcat-7.0-doc/config/executor.html Regards Felix > >Thanks & Regards, >Subhro Paul > >=====-----=====-----===== >Notice: The information contained in this e-mail >message and/or attachments to it may contain >confidential or privileged information. If you are >not the intended recipient, any dissemination, use, >review, distribution, printing or copying of the >information contained in this e-mail message >and/or attachments to it are strictly prohibited. If >you have received this communication in error, >please notify us by reply e-mail or telephone and >immediately and permanently delete the message >and any attachments. Thank you --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org