On 05/10/2015 12:58, Yogesh Patel wrote: > Hi Mark Thomas, > > in image it shows Tomcat Manager screen: > Under "ajp-apr-10003" Section in tomcat manager: > Max threads: 200 Current thread count:200 Current thread busy :200 Keeped > alive socket count :0 > > *Stage Time BSent BRecv Client VHost > Request* > S 2884466 ms 0 KB 184063 KB machineip host POST > /solr430/update?wt=xml&version=2.2 HTTP/1.1
And the other 199 processors? Mark > > .......... > ............ > > > On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.pa...@highq.com> wrote: > >> Hi , let me try sending image using attachment , if still its not viewable >> then i will find another way. >> >> On 5 October 2015 at 16:06, Mark Thomas <ma...@apache.org> wrote: >> >>> On 05/10/2015 11:28, Yogesh Patel wrote: >>>> Hi Thomas , >>>> Please see this image ...have a look at Time column >>> >>> The list strips images. If you really want us to look at the image (not >>> that I think it will be remotely relevant) upload it somewhere and post >>> the URL (make sure it is publicly accessible and doe snot require any >>> form of registration to view it). >>> >>> Mark >>> >>> >>>> >>>> >>>> >>>> >>>> On 5 October 2015 at 14:50, Mark Thomas <ma...@apache.org >>>> <mailto:ma...@apache.org>> wrote: >>>> >>>> On 05/10/2015 10:09, Yogesh Patel wrote: >>>> > Hi Thomas, >>>> > >>>> > Connector configuration is like : >>>> > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" /> >>>> >>>> That means you are using the BIO AJP connector. >>>> >>>> You don't have a problem with long running requests. >>>> >>>> You have a problem with thread starvation. >>>> >>>> AJP uses persistent connections. >>>> >>>> BIO requires one thread per connection, regardless of whether or nor >>>> that connection is processing a request or waiting for the next one. >>>> >>>> httpd will (eventually, assuming mostly default config) create one >>> AJP >>>> connection for each httpd thread. If you have more httpd threads >>> than >>>> Tomcat threads you will eventually reach the point where httpd can't >>>> create a Tomcat thread it requires and at that point it will appear >>>> to hang. >>>> >>>> There are several possible fixes: >>>> a) disable connection re-use in httpd for AJP connections >>>> b) use the NIO AJP connector in Tomcat >>>> c) increase maxThreads in Tomcat so it is > max threads in httpd >>>> >>>> For more explanation read this: >>>> >>> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf >>>> < >>> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf >>>> >>>> >>>> from slide 29 to 33 >>>> >>>> Mark >>>> >>>> >>>> >>>> > >>>> > On 5 October 2015 at 14:17, Mark Thomas <ma...@apache.org >>>> <mailto:ma...@apache.org>> wrote: >>>> > >>>> >> On 05/10/2015 09:07, Yogesh Patel wrote: >>>> >>> Thanks Mark Thomas , >>>> >>> >>>> >>> Our application is access by Apache TO Tomcat using AJP >>>> Connector. >>>> >> >>>> >> OK. That answers one of the questions I asked. However, you >>> haven't >>>> >> provided the connector configuration. >>>> >> >>>> >>> Problem : >>>> >>> Our application was mostly hanged,after looking at tomcat >>>> manager it >>>> >>> shown there are so many long running threads shown. >>>> >> >>>> >> The Tomcat Manager app does not show long running threads. It >>>> does show >>>> >> you how many requests are busy but again, a busy thread is not >>>> >> necessarily processing a request. >>>> >> >>>> >>> We want to recognize why so many threads are running since >>>> long time. >>>> >> >>>> >> Threads are always "running". The key question is are those >>> threads >>>> >> 'idle' or 'busy'. An idle thread is waiting to be allocated work. >>>> A busy >>>> >> thread has been allocated work but the manager application can't >>>> >> distinguish if that work is 'wait for a request to process' or >>>> >> 'processing a request'. >>>> >> >>>> >>> We want to detect such thread and want to kill these stucked >>>> thread. >>>> >> >>>> >> You continue to make the (increasingly unlikely) assumption that >>> 200 >>>> >> busy threads mean you have 200 threads processing long running >>>> requests. >>>> >> Had you provided the connector configuration I asked for (by that >>>> I mean >>>> >> the <Connector ... /> elements in server.xml) then I'd be in a >>> better >>>> >> position to tell you what is wrong. >>>> >> >>>> >> If you do have 200 long running requests then the >>>> >> StuckThreadDetectionValve is how you detect them. That fact that >>> that >>>> >> valve is not reporting any long running requests should be a big >>> clue >>>> >> that your assumptions about what is going on are not correct. >>>> >> >>>> >> If, and it is a big if, you did have 200 concurrent long running >>>> >> requests then there is no guaranteed way to stop them. If your >>>> >> application checks for interrupts (most don't) then the >>>> >> StuckThreadDetectionValve can interrupt them which should >>>> terminate the >>>> >> processing of that request but the time it would take to code the >>>> >> application to handle that properly would normally be better >>> spent >>>> >> fixing the root cause of the long running request. >>>> >> >>>> >> Mark >>>> >> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> On 5 October 2015 at 13:11, Mark Thomas <ma...@apache.org >>>> <mailto:ma...@apache.org>> wrote: >>>> >>> >>>> >>>> On 05/10/2015 07:54, Yogesh Patel wrote: >>>> >>>>> We are facing issues with long running thread in >>>> tomcat . we >>>> >> are >>>> >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy threads >>>> : 200, >>>> >>>>> application gets stucked due to these long running threads. >>>> >>>> >>>> >>>> What makes you think you have issues with long running threads? >>>> A busy >>>> >>>> thread isn't necessarily processing a request. Configuration >>> errors >>>> >>>> leading to thread starvation are a more typical cause of the >>>> symptom you >>>> >>>> describe rather than long running threads in the application. >>>> >>>> >>>> >>>>> We implemented StuckThreadDetectionValve in >>>> server.xml( <Valve >>>> >>>>> >>>> >>>>> >>>> className="org.apache.catalina.valves.StuckThreadDetectionValve" >>>> >>>>> >>>> >>>>> threshold="60" />), but it could not help out. >>>> >>>> >>>> >>>> Why not? Again, this suggests that long running threads aren't >>> the >>>> >> problem. >>>> >>>> >>>> >>>>> So we implemented custom StuckThreadDetectionValve by >>>> extending >>>> >>>>> StuckThreadDetectionValve from >>>> >>>>> catalina, it only goes to "constructor","initInternal",and in >>>> >>>> "invoke"(when >>>> >>>>> action fires), it does not go to function >>>> "getStuckThreadNames()" even >>>> >>>>> after threshold time.How to achieve the same.Is there any way >>>> to detect >>>> >>>>> stucked thread and kill them? >>>> >>>> >>>> >>>> First of all your invoke() method fails to call super.invoke() >>>> so the >>>> >>>> Valve is never going to do anything. >>>> >>>> >>>> >>>> Secondly, if the original valve didn't work adding a bunch of >>>> >>>> System.out.println() lines isn't going to magically make it >>> work. >>>> >>>> >>>> >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat >>>> code so it >>>> >>>> is no surprise that you never see a call to that method. That >>>> method is >>>> >>>> intended for use via JMX. >>>> >>>> >>>> >>>> I suggest you start again and tell us more about the problem >>>> you are >>>> >>>> trying to solve (lack of response) and your configuration. In >>>> particular >>>> >>>> we need to know your connector configuration and how users >>>> access the >>>> >>>> application. Do they connect directly to Tomcat or do they go >>> via a >>>> >>>> reverse proxy? >>>> >>>> >>>> >>>> Mark >>>> >>>> >>>> >>>> >>>> >>>>> >>>> >>>>> My custom Valve is like following : >>>> >>>>> >>>> >>>>> public class StuckThreadDetection extends >>>> StuckThreadDetectionValve >>>> >>>>> { >>>> >>>>> StuckThreadDetection stuckThreadDetection; >>>> >>>>> >>>> >>>>> public StuckThreadDetection() >>>> >>>>> { >>>> >>>>> System.out.println("in StuckThreadDetection constructor"); >>>> >>>>> >>>> >>>>> } >>>> >>>>> >>>> >>>>> public void invoke(Request request, Response response) throws >>>> >>>> IOException, >>>> >>>>> ServletException >>>> >>>>> { >>>> >>>>> System.out.println("in invoke..."); >>>> >>>>> >>>> >>>>> getNext().invoke(request, response); >>>> >>>>> } >>>> >>>>> >>>> >>>>> @Override >>>> >>>>> protected void initInternal() throws LifecycleException >>>> >>>>> { >>>> >>>>> System.out.println("In initInternal"); >>>> >>>>> super.initInternal(); >>>> >>>>> } >>>> >>>>> >>>> >>>>> @Override >>>> >>>>> public String[] getStuckThreadNames() >>>> >>>>> { >>>> >>>>> System.out.println("in getStuckThreadNames..."); >>>> >>>>> String[] listStuckedThread = this.getStuckThreadNames(); >>>> >>>>> >>>> >>>>> for (String threadName : listStuckedThread) >>>> >>>>> { >>>> >>>>> System.out.println(threadName); >>>> >>>>> } >>>> >>>>> >>>> >>>>> return listStuckedThread; >>>> >>>>> >>>> >>>>> } >>>> >>>>> >>>> >>>>> @Override >>>> >>>>> public String getInfo() >>>> >>>>> { >>>> >>>>> System.out.println("In getInfo"); >>>> >>>>> return super.getInfo(); >>>> >>>>> } >>>> >>>>> } >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> --------------------------------------------------------------------- >>>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> <mailto:users-unsubscr...@tomcat.apache.org> >>>> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> <mailto:users-h...@tomcat.apache.org> >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> >> >>> --------------------------------------------------------------------- >>>> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> <mailto:users-unsubscr...@tomcat.apache.org> >>>> >> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> <mailto:users-h...@tomcat.apache.org> >>>> >> >>>> >> >>>> > >>>> > >>>> >>>> >>>> >>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> <mailto:users-unsubscr...@tomcat.apache.org> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> <mailto:users-h...@tomcat.apache.org> >>>> >>>> >>>> >>>> >>>> -- >>>> /Thanks & Regards,/ >>>> / / >>>> / Yogesh Patel/ >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >> >> -- >> *Thanks & Regards,* >> >> * Yogesh Patel* >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
