A couple of thoughts.
1. In my experience turning off pooling will make matters worse, not better.
2. You show threads blocked, but they aren't using CPU. What is using
all the CPU? Look at the threads that are actually running. Chances are
the clue to the bottleneck will be there. The two threads you have shown
say "blocked" so I doubt either of them is the cause.
3. If the bottleneck can't be alleviated then you need to increase your
pool size to allow for that.
[EMAIL PROTECTED] wrote:
Hi,
I am sort of going to get a heart attack in the next hours, if I can’t
resolve this quickly.
We recently deployed a new Version of a intranet-application of a
large logistic company here in Germany. It is completely written in
cocoon and performs extraordinary nicely. Until at one point the
System starts thrashing. CPU Usage climbs from 12% to 80% and no
request are served anymore.
Here a little more details on the Setup:
- Windows 2003 Server
- 4 GB Ram … 1,5 GB reserved for the appserver
- Bea Weblogic 9.2
- JRockit 1.5 VM
- MySQL 5.1 DB Backend
During normal operation a thread dump is about 30k big … during
thrashing this increases to 1,5MB.
Yesterday we had similar Problems which were unfortunately related to
too much logging (100 Threads blocked because they were locked from
the Cocoon Log system). After removing a lot of the Log Output the
system performed nicely even under heavier load. Until the whole thing
blew up again. This time the Thread Dump showed a lot of Threads
blocked at:
"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'" id=35 idx=0x80 tid=5092 prio=5 alive, in native,
blocked, daemon
-- Blocked trying to get lock: java/lang/[EMAIL PROTECTED]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1630)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1731)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1277)[optimized]
at
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(InstrumentedResourceLimitingPool.java:411)[inlined]
at
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(PoolableComponentHandler.java:212)[optimized]
at
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.java:425)[inlined]
at
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(ExcaliburComponentSelector.java:307)[inlined]
at
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedComponentSelector.java:301)[optimized]
at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.recycle(AbstractProcessingPipeline.java:743)[optimized]
at
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(InstrumentedResourceLimitingPool.java:407)[inlined]
at
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(PoolableComponentHandler.java:212)[optimized]
at
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.java:425)[inlined]
at
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(ExcaliburComponentSelector.java:307)[inlined]
at
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedComponentSelector.java:301)[optimized]
at
org/apache/cocoon/components/treeprocessor/InvokeContext.dispose(InvokeContext.java:283)
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:179)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTypeNode.java:139)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[optimized]
at org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]
at
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[optimized]
at
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:104)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(CloseHibernateSessionFilter.java:25)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[inlined]
at
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3242)[optimized]
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2010)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1916)[optimized]
at
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
As far as I understand the thing, the problem are pool limitations to
some of the sitemap components. In general this should not be a
problem. Unfortunately my application uses JavaFlow and a lot of Flows
have to send Data to other Systems. They all use internal pipelines to
generate this output. My guess now is, that if a Flow is executed to
produce output, it blocks some resources. Maybe some resources it
needs itself to finish the request, causing a deadlock. This would be
the case in the following Thread-Dump:
"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default
(self-tuning)'" id=36 idx=0x84 tid=5100 prio=5 alive, in native,
native_blocked, daemon
at
jrockit/vm/Allocator.allocLargeObjectOrArray(IIIZ)Ljava/lang/Object;(Native
Method)
at
jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:831)[inlined]
at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:947)[inlined]
at jrockit/vm/Allocator.allocSlowCase(Allocator.java:793)[optimized]
at
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java:63)[optimized]
at
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java:46)
at
org/apache/cocoon/environment/AbstractEnvironment.getOutputStream(AbstractEnvironment.java:490)
at
org/apache/cocoon/environment/http/HttpEnvironment.getOutputStream(HttpEnvironment.java:263)
at
org/apache/cocoon/environment/wrapper/EnvironmentWrapper.getOutputStream(EnvironmentWrapper.java:311)
at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:577)[optimized]
at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:481)[optimized]
at
org/apache/cocoon/components/treeprocessor/sitemap/SerializeNode.invoke(SerializeNode.java:121)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.handleCocoonRedirect(ConcreteTreeProcessor.java:299)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.access$000(ConcreteTreeProcessor.java:48)[inlined]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor$TreeProcessorRedirector.cocoonRedirect(ConcreteTreeProcessor.java:340)[inlined]
at
org/apache/cocoon/environment/ForwardRedirector.redirect(ForwardRedirector.java:60)[optimized]
at
org/apache/cocoon/components/flow/java/AbstractContinuable.sendPage(AbstractContinuable.java:99)[optimized]
at
de/transa/disco/flow/im/InstantMessengerFlow.doCheckMessageClosed(InstantMessengerFlow.java:88)
at
sun/reflect/GeneratedMethodAccessor1853.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown
Source)
at
sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:585)[optimized]
at
org/apache/cocoon/components/flow/java/JavaInterpreter.callFunction(JavaInterpreter.java:173)[optimized]
at
org/apache/cocoon/components/treeprocessor/sitemap/CallFunctionNode.invoke(CallFunctionNode.java:139)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTypeNode.java:139)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[optimized]
at org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]
at
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[optimized]
at
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:104)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(CloseHibernateSessionFilter.java:25)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[inlined]
at
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3242)[optimized]
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2010)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1916)[optimized]
at
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
Causing the System to immediately run out of resources.
The only solution I can think of would be to turn off pooling of
components. Where and how can I do this? Is this simply done by
setting the pool-max properties in the core.properties to 0 or do I
have to do more?
We currently have 4 CPUs and 4GB Ram, but this could be extended if we
need to (Virtual environment).
Regards,
Chris
[ C h r i s t o f e r D u t z ]
C-Ware IT-Service
Inhaber
Dipl. Inf. Christofer Dutz
Karlstraße. 104, 64285 Darmstadt
fon: 0 61 51 / 27315 - 61
fax: 0 61 51 / 27315 - 64
mobil: 0171 / 7 444 2 33
email: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
http://www.c-ware.de <http://www.c-ware.de/>
FA Darmstadt: 07 813 60581
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
