Maybe the following thread helps: http://mail-archives.apache.org/mod_mbox/cxf-users/201201.mbox/%[email protected]%3E
Oli ________________________________________ From: dhogan [[email protected]] Sent: 08 April 2013 23:37 To: [email protected] Subject: WebSphere 8.5 hanging CXF STS implementation? Hi- I am working on getting an STS (CXF 2.7.2) implementation working on a variety of containers. Thus far, I've been successful on Tomcat, JBoss EAP 4, EAP 5, and AS 7, WebLogic 11g and 12c, all on Ubuntu 10.04. Now I'm working on WebSphere 8.5 (last in the list!!), and can't seem to get things working. Perhaps others have run into similar issues, and can shed some light? I've followed the directions for WebSphere from here: http://cxf.apache.org/docs/application-server-specific-configuration-guide.html I publish an STS with e.g. UsernameTokens over the Symmetric binding, and attempt to do an issue operation. The invocation hangs. Eventually, WebSphere produces the following log: [4/8/13 13:54:47:845 PDT] 00000025 ThreadMonitor W WSVR0605W: Thread "WebContainer : 0" (000000c7) has been active for 661856 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung. at com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.wrapChildDOMNode(SOAPElement.java:3223) at com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.insertBefore(SOAPElement.java:1896) at org.apache.xerces.dom.ParentNode.internalInsertBefore(Unknown Source) at org.apache.xerces.dom.ParentNode.replaceChild(Unknown Source) at org.apache.xml.security.encryption.XMLCipher.decryptElement(XMLCipher.java:1627) at org.apache.xml.security.encryption.XMLCipher.doFinal(XMLCipher.java:980) at org.apache.ws.security.processor.ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337) at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197) at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104) at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64) at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396) at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274) at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:198) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1224) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456) at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97) at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91) at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:95) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:895) at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1814) Triggering a system dump from the WebSphere console generates the following information: 3XMTHREADINFO "WebContainer : 0" J9VMThread:0x0BC1E400, j9thread_t:0x0CA5CF78, java/lang/Thread:0x7A926E18, state:CW, prio=5 3XMTHREADINFO1 (native thread ID:0x20C0, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO2 (native stack address range from:0x02697000, to:0x026D8000, size:0x41000) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at org/apache/xerces/dom/ElementNSImpl.<init>(Bytecode PC:9(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl.<init>(DOMElementImpl.java:97(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl_MS.<init>(DOMElementImpl_MS.java:133(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.<init>(SOAPElement.java:465(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.createChildElement(SOAPElement.java:3152(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.wrapChildDOMNode(SOAPElement.java:3218(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.insertBefore(SOAPElement.java:1896(Compiled Code)) 4XESTACKTRACE at org/apache/xerces/dom/ParentNode.internalInsertBefore(Bytecode PC:91(Compiled Code)) 4XESTACKTRACE at org/apache/xerces/dom/ParentNode.replaceChild(Bytecode PC:12) 4XESTACKTRACE at org/apache/xml/security/encryption/XMLCipher.decryptElement(XMLCipher.java:1627) 4XESTACKTRACE at org/apache/xml/security/encryption/XMLCipher.doFinal(XMLCipher.java:980) 4XESTACKTRACE at org/apache/ws/security/processor/ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337) 4XESTACKTRACE at org/apache/ws/security/processor/ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197) 4XESTACKTRACE at org/apache/ws/security/processor/ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104) 4XESTACKTRACE at org/apache/ws/security/processor/ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64) 4XESTACKTRACE at org/apache/ws/security/WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396) 4XESTACKTRACE at org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274) 4XESTACKTRACE at org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93) 4XESTACKTRACE at org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) 5XESTACKTRACE (entered lock: org/apache/cxf/phase/PhaseInterceptorChain@0x7D08A1D8, entry count: 1) 4XESTACKTRACE at org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) 4XESTACKTRACE at org/apache/cxf/transport/http/AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) 4XESTACKTRACE at org/apache/cxf/transport/servlet/ServletController.invokeDestination(ServletController.java:218) 4XESTACKTRACE at org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:198) 4XESTACKTRACE at org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:137) 4XESTACKTRACE at org/apache/cxf/transport/servlet/CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158) 4XESTACKTRACE at org/apache/cxf/transport/servlet/AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243) 4XESTACKTRACE at org/apache/cxf/transport/servlet/AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163) 4XESTACKTRACE at javax/servlet/http/HttpServlet.service(HttpServlet.java:595) 4XESTACKTRACE at org/apache/cxf/transport/servlet/AbstractHTTPServlet.service(AbstractHTTPServlet.java:219) 4XESTACKTRACE at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1224) 4XESTACKTRACE at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:774) 4XESTACKTRACE at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:456) 4XESTACKTRACE at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97) 4XESTACKTRACE at org/forgerock/openam/validation/ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91) 4XESTACKTRACE at com/sun/identity/setup/AMSetupFilter.doFilter(AMSetupFilter.java:95) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928) 4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025) 4XESTACKTRACE at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87) 4XESTACKTRACE at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:895) 4XESTACKTRACE at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662) 4XESTACKTRACE at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:195) 4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459) 4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526) 4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312) 4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:87) 4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) 4XESTACKTRACE at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) 4XESTACKTRACE at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) 4XESTACKTRACE at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138) 4XESTACKTRACE at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204) 4XESTACKTRACE at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775) 4XESTACKTRACE at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905) 4XESTACKTRACE at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1814(Compiled Code)) 3XMTHREADINFO3 Native callstack: 4XENATIVESTACK (0x003085C0 [libj9prt26.so+0xf5c0]) 4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2]) 4XENATIVESTACK (0x003082B1 [libj9prt26.so+0xf2b1]) 4XENATIVESTACK (0x003083A5 [libj9prt26.so+0xf3a5]) 4XENATIVESTACK (0x00307EB4 [libj9prt26.so+0xeeb4]) 4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2]) 4XENATIVESTACK (0x00307E23 [libj9prt26.so+0xee23]) 4XENATIVESTACK (0x00302A5D [libj9prt26.so+0x9a5d]) 4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0x00AE7410) 4XENATIVESTACK j9thread_monitor_wait+0x85 (0x00F84B21 [libj9thr26.so+0x8b21]) 4XENATIVESTACK (0x002C9019 [libj9vm26.so+0x5c019]) 4XENATIVESTACK (0x002C9354 [libj9vm26.so+0x5c354]) 4XENATIVESTACK (0x00272D01 [libj9vm26.so+0x5d01]) 4XENATIVESTACK (0x01593386 [libj9jit26.so+0x5cf386]) 4XENATIVESTACK (0x002D0924 [libj9vm26.so+0x63924]) 4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2]) 4XENATIVESTACK (0x002D0876 [libj9vm26.so+0x63876]) 4XENATIVESTACK (0x00F851E5 [libj9thr26.so+0x91e5]) 4XENATIVESTACK (0x0083496E [libpthread.so.0+0x596e]) 4XENATIVESTACK clone+0x5e (0x001DDA4E [libc.so.6+0xcda4e]) Note that the dump log indicates that the invocation progressed a bit further. This is not a mistake, and I've noticed the same behavior several times. The box is pegged (CPU at >99%) once this occurs, and it seems like the disparity is a function of time differences when the log entry and dumps are generated. Additional dump state(see below) seems to indicate that the invoking thread has entered a monitor's wait-set, but is never notified out of the waiting state. I'm not sure if this is a red-herring, or if it is a function of the invocation simply no longer making progress (i.e. starving because CPU is pegged - though this too, seems unlikely). ... 2LKREGMON &newThread->threadNameMutex lock (0x0C04EC98): <unowned> 2LKREGMON JVM_RawMonitor lock (0x0C04ECF0): <unowned> 2LKREGMON Thread public flags mutex lock (0x0C04ED48): <unowned> 3LKNOTIFYQ Waiting to be notified: 3LKWAITNOTIFY "WebContainer : 0" (0x0BC1E400) 2LKREGMON &(classLoader->mutex) lock (0x0C04EDA0): <unowned> 2LKREGMON JIT-QueueSlotMonitor-1103 lock (0x0C04F0B8): <unowned> 2LKREGMON &(classLoader->mutex) lock (0x0C04F2C8): <unowned> ... Anyone have any suggestions or similar experiences? Thanks Dirk -- View this message in context: http://cxf.547215.n5.nabble.com/WebSphere-8-5-hanging-CXF-STS-implementation-tp5725997.html Sent from the cxf-user mailing list archive at Nabble.com.
