Good luck. It could be something stupid like backup is running during the day, or some other periodic process. There's no sense troubleshooting a tomcat slowdown until you know it's tomcat that's using the CPU.
If you're running Windows, you need to run perfmon on the the process and chart process CPU and total cpu. If you're running UNIX, you need to install sysstat and also use top or some other tool. Really, this is basic troubleshooting. Determine where the problem is, and dig deeper to find the solution. Notice I didn't say: Assume where the problem is and dig deeper to justify your assumptions. George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 > -----Original Message----- > From: neilgoldsmith [mailto:ne...@avaya.com] > Sent: Friday, August 21, 2009 9:21 AM > To: users@tomcat.apache.org > Subject: RE: Tracking down a Tomcat slowdown > > > I don't have direct access to their system, but they assure me this > server is > only used for this purpose and nothing else is running on their > servers. > They have 2 servers (exact same config) running Tomcat and load balance > 1:1 > between the 2. When I looked at the log for the 2nd server, it had not > experienced the slowdown although in theory it should have been just as > busy > as the 1st. I don't know if I can conclude anything from that though. > > > > George Sexton wrote: > > > > So, I take it you've monitored the machine and made sure there are no > > other > > processes using up CPU/IO Bandwidth? > > > > George Sexton > > MH Software, Inc. > > http://www.mhsoftware.com/ > > Voice: 303 438 9585 > > > > > >> -----Original Message----- > >> From: neilgoldsmith [mailto:ne...@avaya.com] > >> Sent: Friday, August 21, 2009 9:11 AM > >> To: users@tomcat.apache.org > >> Subject: Tracking down a Tomcat slowdown > >> > >> > >> We have a servlet that handles incoming voice calls at customer > >> locations, so > >> you can have bursts of busy time (much like web traffic). A > customer > >> of > >> ours has been running this servlet (24/7) for a few months and can > go > >> weeks > >> at a time with no problems, but periodically they will experience a > big > >> slowdown on the app server which does eventually recover (slows down > >> for > >> maybe an hour). We had them increase their maxThreads from 150 to > 300 > >> and > >> increase the heap size: > >> > >> (their current server.xml settings) > >> <Connector port="8080" maxHttpHeaderSize="8192" > >> maxThreads="300" minSpareThreads="25" maxSpareThreads="75" > >> enableLookups="false" redirectPort="8443" acceptCount="100" > >> connectionTimeout="20000" disableUploadTimeout="true" /> > >> > >> JAVA_OPTS=%JAVA_OPTS% - > Djava.util.logging.manager=org.apache.juli.Cla > >> ssLoaderLogManager > >> - > >> > Djava.util.logging.config.file="%CATALINA_BASE%\conf\logging.properties > >> " > >> -server -Xms1024m -Xmx1536m > >> > >> It seems to have improved, but the problem cropped up again. > >> > >> We use log4j to perform our logging and if you notice below, we have > 2 > >> lines > >> of code logging some output one right after another. typically, > there > >> is > >> just ms's in the log between these 2 log stmts, but when the > slowdown > >> occurs, you can see there is about 2 secs delay which leads me to > >> believe > >> the appserver has exhausted its threads and the servlet cannot get a > >> thread > >> to execute on. We have disabled logging in the past to make sure > the > >> logging itself wasn't the problem and we still encountered the > >> slowdown. > >> Note that the slowdown isn't just here in the log, but the entire > >> servlet > >> slows down, this part of the code is just a good indicator of how it > is > >> affecting the servlet. > >> > >> These 2 log stmts follow one another and use log4j internally: > >> LOG("ICConnectorManager.doGet: Wait time to return response:" + > >> (System.currentTimeMillis() - waitStart) + "ms for request ID:" + > >> reqID); > >> LOG("ICConnectorManager.doGet: Looking up " + reqID + " and > returning > >> response back to caller:" + sResponse); > >> > >> This is the output from those 2 log stmts, you can see the timestamp > >> shows 2 > >> seconds between. > >> 10/08/2009 09:06:24 DEBUG - ICConnectorManager.doGet: Wait time to > >> return > >> response:10125ms for request ID:14252-592 > >> 10/08/2009 09:06:26 DEBUG - ICConnectorManager.doGet: Looking up > 14252- > >> 592 > >> and returning response back to caller: > >> > >> > >> My first question, what is the best tool to monitor this so we can > get > >> an > >> accurate description of when the problems occur and what might be at > >> fault? > >> They just started running perfmon (on a Windows system), but as of > yet > >> I > >> have not seen any data from it. Is there a better monitoring tool? > >> > >> Second question, what might we do to improve Tomcat performance? > >> Should we > >> continue to tweak the max threads and heap size? Any other Tomcat > >> settings > >> that we should look at that might directly relate to this server > >> slowdown? > >> > >> thank you > >> > >> > >> -- > >> View this message in context: http://www.nabble.com/Tracking-down-a- > >> Tomcat-slowdown-tp25081707p25081707.html > >> Sent from the Tomcat - User mailing list archive at Nabble.com. > >> > >> > >> -------------------------------------------------------------------- > - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > -- > View this message in context: http://www.nabble.com/Tracking-down-a- > Tomcat-slowdown-tp25081707p25081898.html > Sent from the Tomcat - User mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
