Scott, The config change suggested did not solve this issue. Those waiting threads are likely a side affect and not cause. Note that this issue (accumulation of waiting threads, leading to full degradation of performance) can be duplicated by running our link checker against all 12 web sites. This activity, it seems, would be similar to a crawler indexing our sites. The assumption is that this activity is responsible for initiating the downward spiral in performance ending in no response from the server. From testing it seems this is not necessarily related to the level of concurrent activity. If we link check one site at a time, after a number of sites, we end up in this state. Interesting to note that if we link check the same site repeatedly, the server does not progress to this bad state. Instead some extra threads are created on the first test, subsequent tests do not increase the thread count. Of course when we link check all at once, threads pile up, performance deg! rades
When the server gets in this bad state CPU utilization is tapped. Our environment is running on a 2 CPU box. While in this bad state, one CPU will me maxed (90+ %) the other near (0%). At pretty short intervals (few seconds) the load is swapped from CPU1 to CPU2 and then back again, continuing to swap in this fashion. I assume the JVM is utilizing one CPU at a time. In worst state, thread count (in JVM stack trace) is high, usually over 200. No dead locks are indicated in the JVM stack trace. When the system is healthy, we typically see 80 threads. I took 3 traces a few minutes apart as system performance went from bad to worse. The majority of threads are ?waiting on monitor entry?. The thread traces appear to be all executing, or waiting at a low level in the code (below our application). There are only a few threads in the runnable state. These runnable threads appear to be in normal processing based on thread stack traces, nothing stands out. I?d like to attach the 3 stack traces but these are large. Can I send these via email to get more input? Allen View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3915560#3915560 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3915560 ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ JBoss-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jboss-user
