We finally got some thread dumps from our production server. It shows
something very different than what we were seeing in testing. First,
this happens under light load after running for days. To summarize,
many threads are waiting for the ResourceLimitingPool and several are
waiting for the class loader. This system hasn't had the pools tuned so
I'm not surprised about pool contention, but I don't believe that is the
issue. That is because the thread holding the lock is simply waiting for
the class loader.
We took two traces and both were similar, but not identical. Different
threads were holding the class loader lock in both. However, in both
cases the threads holding the class loader lock were called from Castor
while creating the portal layout.
So far, we have been speculating that the problem is due to a problem
with the NPTL threads on Enterprise Linux 3. However, I'm wondering if
perhaps castor is having problems and simply calling the class loader
over and over.
I'd appreciate any ideas.
We see many threads like this one...
"http-8080-Processor155" daemon prio=1 tid=0x083e3378 nid=0x1e8b waiting
for monitor entry [22dc0000..22dc187c]
at
org.apache.avalon.excalibur.pool.ResourceLimitingPool.get(ResourceLimitingPool.java:262)
- waiting to lock <0x60cd9970> (a java.lang.Object)
at
org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
at
org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
at
org.apache.avalon.excalibur.component.ExcaliburComponentSelector.select(ExcaliburComponentSelector.java:213)
at
org.apache.cocoon.components.ExtendedComponentSelector.select(ExtendedComponentSelector.java:260)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.addTransformer(AbstractProcessingPipeline.java:267)
at
org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.addTransformer(AbstractCachingProcessingPipeline.java:143)
at
org.apache.cocoon.components.treeprocessor.sitemap.TransformNode.invoke(TransformNode.java:59)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:130)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:89)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:240)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:180)
at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:243)
at org.apache.cocoon.Cocoon.process(Cocoon.java:606)
at
org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java:1119)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)3)
at java.lang.Thread.run(Thread.java:534))
The thread holding the lock for this is
"http-8080-Processor33" daemon prio=1 tid=0x0833f308 nid=0x1e8b waiting
for monitor entry [24d09000..24d0a87c]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:260)
- waiting to lock <0x60b19148> (a sun.misc.Launcher$AppClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:235)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1255)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1189)
at
org.apache.excalibur.xml.xslt.XSLTProcessorImpl.getTransformerFactory(XSLTProcessorImpl.java:453)
at
org.apache.excalibur.xml.xslt.XSLTProcessorImpl.initialize(XSLTProcessorImpl.java:152)
at
org.apache.avalon.framework.container.ContainerUtil.initialize(ContainerUtil.java:283)
at
org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:277)
at
org.apache.avalon.excalibur.pool.ResourceLimitingPool.newPoolable(ResourceLimitingPool.java:672)
at
org.apache.avalon.excalibur.pool.ResourceLimitingPool.get(ResourceLimitingPool.java:402)
- locked <0x613264d8> (a java.lang.Object)
at
org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
at
org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
at
org.apache.avalon.excalibur.component.ExcaliburComponentManager.lookup(ExcaliburComponentManager.java:270)
at
org.apache.cocoon.components.CocoonComponentManager.lookup(CocoonComponentManager.java:310)
at
org.apache.avalon.excalibur.component.ExcaliburComponentManager.lookup(ExcaliburComponentManager.java:192)
at
org.apache.cocoon.components.CocoonComponentManager.lookup(CocoonComponentManager.java:310)
at
org.apache.avalon.framework.service.WrapperServiceManager.lookup(WrapperServiceManager.java:107)
at
org.apache.cocoon.transformation.TraxTransformer.configure(TraxTransformer.java:248)
at
org.apache.avalon.framework.container.ContainerUtil.configure(ContainerUtil.java:240)
at
org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:269)
at
org.apache.avalon.excalibur.pool.ResourceLimitingPool.newPoolable(ResourceLimitingPool.java:672)
at
org.apache.avalon.excalibur.pool.ResourceLimitingPool.get(ResourceLimitingPool.java:402)
- locked <0x60cd9970> (a java.lang.Object)
at
org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
at
org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
at
org.apache.avalon.excalibur.component.ExcaliburComponentSelector.select(ExcaliburComponentSelector.java:213)
at
org.apache.cocoon.components.ExtendedComponentSelector.select(ExtendedComponentSelector.java:260)
at
org.apache.cocoon.components.ExtendedComponentSelector.select(ExtendedComponentSelector.java:270)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.addTransformer(AbstractProcessingPipeline.java:267)
at
org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.addTransformer(AbstractCachingProcessingPipeline.java:143)
at
org.apache.cocoon.components.treeprocessor.sitemap.TransformNode.invoke(TransformNode.java:59)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.ActTypeNode.invoke(ActTypeNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:130)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:89)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:240)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:180)
at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:243)
at
org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:130)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:89)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:240)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:180)
at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:243)
at org.apache.cocoon.Cocoon.process(Cocoon.java:606)
at
org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java:1119)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at
java.lang.Thread.run(Thread.java:534)ntrolRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534))
and the thread holding that lock is
"http-8080-Processor26" daemon prio=1 tid=0x0821b148 nid=0x1e8b waiting
for monitor entry [2cafd000..2caff87c]
at java.lang.String.replace(String.java:1555)
at java.net.URLClassLoader$1.run(URLClassLoader.java:190)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:187)
at java.lang.ClassLoader.loadClass(ClassLoader.java:289)
- locked <0x60b1d920> (a sun.misc.Launcher$ExtClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:282)
- locked <0x60b19148> (a sun.misc.Launcher$AppClassLoader)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274)
- locked <0x60b19148> (a sun.misc.Launcher$AppClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:235)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1255)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1189)
at
org.exolab.castor.xml.util.ClassDescriptorResolverImpl.loadClass(ClassDescriptorResolverImpl.java:521)
at
org.exolab.castor.xml.util.ClassDescriptorResolverImpl.resolve(ClassDescriptorResolverImpl.java:330)
at
org.exolab.castor.xml.MarshalFramework.searchInheritance(MarshalFramework.java:306)
at
org.exolab.castor.xml.UnmarshalHandler.startElement(UnmarshalHandler.java:1692)
at
org.exolab.castor.xml.UnmarshalHandler.startElement(UnmarshalHandler.java:1292)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
Source)
at
org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.exolab.castor.xml.Unmarshaller.unmarshal(Unmarshaller.java:605)
at
org.apache.cocoon.components.persistence.CastorSourceConverter.getObject(CastorSourceConverter.java:79)
at
org.apache.cocoon.portal.profile.impl.MapProfileLS.loadProfile(MapProfileLS.java:141)
at
org.apache.cocoon.portal.profile.impl.AuthenticationProfileManager.loadProfile(AuthenticationProfileManager.java:420)
at
org.apache.cocoon.portal.profile.impl.AuthenticationProfileManager.getOrCreateProfile(AuthenticationProfileManager.java:306)
at
org.apache.cocoon.portal.profile.impl.AuthenticationProfileManager.loadProfile(AuthenticationProfileManager.java:125)
at
org.apache.cocoon.portal.profile.impl.AbstractUserProfileManager.getPortalLayout(AbstractUserProfileManager.java:345)
at
org.apache.cocoon.portal.impl.PageLabelManager.initializeLabels(PageLabelManager.java:251)
at
org.apache.cocoon.portal.impl.PageLabelManager.getPageLabelEvents(PageLabelManager.java:190)
at
org.apache.cocoon.portal.event.aspect.impl.PageLabelEventAspect.process(PageLabelEventAspect.java:84)
at
org.apache.cocoon.portal.event.impl.DefaultEventAspectContext.invokeNext(DefaultEventAspectContext.java:58)
at
org.apache.cocoon.portal.event.aspect.impl.FullScreenCopletEventAspect.process(FullScreenCopletEventAspect.java:105)
at
org.apache.cocoon.portal.event.impl.DefaultEventAspectContext.invokeNext(DefaultEventAspectContext.java:58)
at
org.apache.cocoon.portal.event.aspect.impl.AbstractContentEventAspect.process(AbstractContentEventAspect.java:131)
at
org.apache.cocoon.portal.event.impl.DefaultEventAspectContext.invokeNext(DefaultEventAspectContext.java:58)
at
org.apache.cocoon.portal.event.aspect.impl.AbstractContentEventAspect.process(AbstractContentEventAspect.java:131)
at
org.apache.cocoon.portal.event.impl.DefaultEventAspectContext.invokeNext(DefaultEventAspectContext.java:58)
at
org.apache.cocoon.portal.event.aspect.impl.ConvertableEventAspect.process(ConvertableEventAspect.java:88)
at
org.apache.cocoon.portal.event.impl.DefaultEventAspectContext.invokeNext(DefaultEventAspectContext.java:58)
at
org.apache.cocoon.portal.event.impl.DefaultEventManager.processEvents(DefaultEventManager.java:274)
at
org.apache.cocoon.portal.impl.PortalManagerImpl.process(PortalManagerImpl.java:51)
at
org.apache.cocoon.portal.impl.PortletPortalManager.process(PortletPortalManager.java:198)
at
org.apache.cocoon.portal.generation.PortalGenerator.generate(PortalGenerator.java:58)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:530)
at
org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:172)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:468)
at
org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:120)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.ActTypeNode.invoke(ActTypeNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:130)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:89)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:240)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:180)
at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:243)
at
org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:46)
at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:130)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:138)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:68)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:89)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:240)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:180)
at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:243)
at org.apache.cocoon.Cocoon.process(Cocoon.java:606)
at
org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java:1119)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at
java.lang.Thread.run(Thread.java:534)hreads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534))
