Hi-
I am working on getting an STS (CXF 2.7.2) implementation working on a
variety of containers.
Thus far, I've been successful on Tomcat, JBoss EAP 4, EAP 5, and AS 7,
WebLogic 11g
and 12c, all on Ubuntu 10.04. Now I'm working on WebSphere 8.5 (last in the
list!!), and can't seem to get things working.
Perhaps others have run into similar issues, and can shed some light?
I've followed the directions for WebSphere from here:
http://cxf.apache.org/docs/application-server-specific-configuration-guide.html
I publish an STS with e.g. UsernameTokens over the Symmetric binding, and
attempt to
do an issue operation. The invocation hangs. Eventually, WebSphere produces
the following log:
[4/8/13 13:54:47:845 PDT] 00000025 ThreadMonitor W WSVR0605W: Thread
"WebContainer : 0" (000000c7) has been active for 661856 milliseconds and
may be hung. There is/are 1 thread(s) in total in the server that may be
hung.
at
com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.wrapChildDOMNode(SOAPElement.java:3223)
at
com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.insertBefore(SOAPElement.java:1896)
at org.apache.xerces.dom.ParentNode.internalInsertBefore(Unknown
Source)
at org.apache.xerces.dom.ParentNode.replaceChild(Unknown Source)
at
org.apache.xml.security.encryption.XMLCipher.decryptElement(XMLCipher.java:1627)
at
org.apache.xml.security.encryption.XMLCipher.doFinal(XMLCipher.java:980)
at
org.apache.ws.security.processor.ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337)
at
org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197)
at
org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104)
at
org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64)
at
org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274)
at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:198)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1224)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
at
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at
org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:95)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:895)
at
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at
com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
at
com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at
com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at
com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at
com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1814)
Triggering a system dump from the WebSphere console generates the following
information:
3XMTHREADINFO "WebContainer : 0" J9VMThread:0x0BC1E400,
j9thread_t:0x0CA5CF78, java/lang/Thread:0x7A926E18, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x20C0, native priority:0x5,
native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x02697000,
to:0x026D8000, size:0x41000)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at
org/apache/xerces/dom/ElementNSImpl.<init>(Bytecode PC:9(Compiled Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl.<init>(DOMElementImpl.java:97(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl_MS.<init>(DOMElementImpl_MS.java:133(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.<init>(SOAPElement.java:465(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.createChildElement(SOAPElement.java:3152(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.wrapChildDOMNode(SOAPElement.java:3218(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.insertBefore(SOAPElement.java:1896(Compiled
Code))
4XESTACKTRACE at
org/apache/xerces/dom/ParentNode.internalInsertBefore(Bytecode
PC:91(Compiled Code))
4XESTACKTRACE at
org/apache/xerces/dom/ParentNode.replaceChild(Bytecode PC:12)
4XESTACKTRACE at
org/apache/xml/security/encryption/XMLCipher.decryptElement(XMLCipher.java:1627)
4XESTACKTRACE at
org/apache/xml/security/encryption/XMLCipher.doFinal(XMLCipher.java:980)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64)
4XESTACKTRACE at
org/apache/ws/security/WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
4XESTACKTRACE at
org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274)
4XESTACKTRACE at
org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93)
4XESTACKTRACE at
org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
5XESTACKTRACE (entered lock:
org/apache/cxf/phase/PhaseInterceptorChain@0x7D08A1D8, entry count: 1)
4XESTACKTRACE at
org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
4XESTACKTRACE at
org/apache/cxf/transport/http/AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invokeDestination(ServletController.java:218)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:198)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:137)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163)
4XESTACKTRACE at
javax/servlet/http/HttpServlet.service(HttpServlet.java:595)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.service(AbstractHTTPServlet.java:219)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1224)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:774)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:456)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
4XESTACKTRACE at
org/forgerock/openam/validation/ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
4XESTACKTRACE at
com/sun/identity/setup/AMSetupFilter.doFilter(AMSetupFilter.java:95)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
4XESTACKTRACE at
com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:895)
4XESTACKTRACE at
com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662)
4XESTACKTRACE at
com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:195)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
4XESTACKTRACE at
com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
4XESTACKTRACE at
com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
4XESTACKTRACE at
com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
4XESTACKTRACE at
com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905)
4XESTACKTRACE at
com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1814(Compiled Code))
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK (0x003085C0 [libj9prt26.so+0xf5c0])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x003082B1 [libj9prt26.so+0xf2b1])
4XENATIVESTACK (0x003083A5 [libj9prt26.so+0xf3a5])
4XENATIVESTACK (0x00307EB4 [libj9prt26.so+0xeeb4])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x00307E23 [libj9prt26.so+0xee23])
4XENATIVESTACK (0x00302A5D [libj9prt26.so+0x9a5d])
4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0x00AE7410)
4XENATIVESTACK j9thread_monitor_wait+0x85 (0x00F84B21
[libj9thr26.so+0x8b21])
4XENATIVESTACK (0x002C9019 [libj9vm26.so+0x5c019])
4XENATIVESTACK (0x002C9354 [libj9vm26.so+0x5c354])
4XENATIVESTACK (0x00272D01 [libj9vm26.so+0x5d01])
4XENATIVESTACK (0x01593386 [libj9jit26.so+0x5cf386])
4XENATIVESTACK (0x002D0924 [libj9vm26.so+0x63924])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x002D0876 [libj9vm26.so+0x63876])
4XENATIVESTACK (0x00F851E5 [libj9thr26.so+0x91e5])
4XENATIVESTACK (0x0083496E [libpthread.so.0+0x596e])
4XENATIVESTACK clone+0x5e (0x001DDA4E [libc.so.6+0xcda4e])
Note that the dump log indicates that the invocation progressed a bit
further. This is not a mistake, and I've noticed the same behavior several
times. The box is pegged (CPU at >99%) once this occurs, and it seems like
the disparity is a function of time differences when the log entry and dumps
are generated. Additional dump state(see below) seems to indicate that the
invoking thread has entered a monitor's wait-set, but is never notified out
of the waiting state. I'm not sure if this is a red-herring, or if it is a
function of the invocation simply no longer making progress (i.e. starving
because CPU is pegged - though this too, seems unlikely).
...
2LKREGMON &newThread->threadNameMutex lock (0x0C04EC98): <unowned>
2LKREGMON JVM_RawMonitor lock (0x0C04ECF0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0C04ED48): <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "WebContainer : 0" (0x0BC1E400)
2LKREGMON &(classLoader->mutex) lock (0x0C04EDA0): <unowned>
2LKREGMON JIT-QueueSlotMonitor-1103 lock (0x0C04F0B8): <unowned>
2LKREGMON &(classLoader->mutex) lock (0x0C04F2C8): <unowned>
...
Anyone have any suggestions or similar experiences?
Thanks
Dirk
--
View this message in context:
http://cxf.547215.n5.nabble.com/WebSphere-8-5-hanging-CXF-STS-implementation-tp5725997.html
Sent from the cxf-user mailing list archive at Nabble.com.
