I have a theory. It's related to multiple MTOM attachments, and someone is closing the main stream thinking that they are closing a cache stream.
On Mon, Dec 6, 2010 at 2:49 PM, Daniel Kulp <[email protected]> wrote: > > Well, in both cases, it looks like something in the middle is closing the > stream. I have NO idea what would cause that. > > Tracing through the calls in the stack trace, it's basically saying the call > to the underlying NIO ByteChannel's isOpen method is returning false. So > SOMETHING is closing it. > > How reproducible is it? Here is my thought on how to debug: > > Add an interceptor at the very beginning of the IN chain and fairly early > (before AttachmentOut) of the OUT chain that wrappers the input/output streams > that just override the "close()" method. That override would do a "new > Throwable().fillInStackTrace()" and sticks it on the message using a private > key. > > An separate interceptor on the fault out chain would grab those and print > those if they are there. THAT way, if something is calling close that > shouldn't be, we can know where. If they are both null, then we know it's > not us calling close and I'm not sure what would be the issue. > > Dan > > > On Saturday 04 December 2010 8:57:09 am Benson Margulies wrote: >> Possibly related on the server side: >> >> 2010-12-04 08:49:29,676 org.apache.cxf.phase.PhaseInterceptorChain - >> Interceptor for >> {http://jdd.basistech.com/}DocumentDatabaseService#{http://jdd.basistech.co >> m/}addDocument has thrown exception, unwinding now >> org.apache.cxf.interceptor.Fault: Could not write attachments. >> at >> org.apache.cxf.interceptor.AttachmentOutInterceptor.handleMessage(Attachme >> ntOutInterceptor.java:71) at >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha >> in.java:247) at >> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outgoing >> ChainInterceptor.java:77) at >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha >> in.java:247) at >> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiation >> Observer.java:113) at >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(Je >> ttyHTTPDestination.java:311) at >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHT >> TPDestination.java:280) at >> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandl >> er.java:72) at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.ja >> va:922) at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.jav >> a:860) at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:1 >> 17) at >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHa >> ndlerCollection.java:247) at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java >> :114) at org.eclipse.jetty.server.Server.handle(Server.java:352) >> at >> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java: >> 596) at >> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnect >> ion.java:1066) at >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at >> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at >> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoin >> t.java:510) at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEnd >> Point.java:34) at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint >> .java:40) at >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java >> :450) at java.lang.Thread.run(Thread.java:680) >> Caused by: org.eclipse.jetty.io.EofException >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:148) >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:92) >> at >> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputS >> tream.java:46) at >> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputS >> tream.java:55) at >> org.apache.cxf.attachment.AttachmentSerializer.writeProlog(AttachmentSeria >> lizer.java:142) at >> org.apache.cxf.interceptor.AttachmentOutInterceptor.handleMessage(Attachme >> ntOutInterceptor.java:69) ... 23 more >> 2010-12-04 08:49:29,683 org.apache.cxf.phase.PhaseInterceptorChain - >> Interceptor for >> {http://jdd.basistech.com/}DocumentDatabaseService#{http://jdd.basistech.co >> m/}addDocument has thrown exception, unwinding now >> org.apache.cxf.binding.soap.SoapFault: Error writing to XMLStreamWriter. >> at >> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor$SoapOutEndingIn >> terceptor.handleMessage(SoapOutInterceptor.java:288) at >> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor$SoapOutEndingIn >> terceptor.handleMessage(SoapOutInterceptor.java:270) at >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha >> in.java:247) at >> org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(A >> bstractFaultChainInitiatorObserver.java:99) at >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha >> in.java:303) at >> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outgoing >> ChainInterceptor.java:77) at >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha >> in.java:247) at >> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiation >> Observer.java:113) at >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(Je >> ttyHTTPDestination.java:311) at >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHT >> TPDestination.java:280) at >> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandl >> er.java:72) at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.ja >> va:922) at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.jav >> a:860) at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:1 >> 17) at >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHa >> ndlerCollection.java:247) at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java >> :114) at org.eclipse.jetty.server.Server.handle(Server.java:352) >> at >> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java: >> 596) at >> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnect >> ion.java:1066) at >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at >> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at >> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoin >> t.java:510) at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEnd >> Point.java:34) at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint >> .java:40) at >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java >> :450) at java.lang.Thread.run(Thread.java:680) >> Caused by: com.ctc.wstx.exc.WstxIOException: null >> at >> com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:143 >> 1) at >> com.ctc.wstx.sw.BaseStreamWriter.writeEndDocument(BaseStreamWriter.java:55 >> 3) at >> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor$SoapOutEndingIn >> terceptor.handleMessage(SoapOutInterceptor.java:282) ... 26 more >> Caused by: org.eclipse.jetty.io.EofException >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:148) >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:92) >> at >> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputS >> tream.java:46) at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:100) >> at >> com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:225) >> at >> com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:198) >> at >> com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:142 >> 9) ... 28 more >> >> On Sat, Dec 4, 2010 at 7:19 AM, Benson Margulies <[email protected]> > wrote: >> > My server, which is a JAX-WS endpoint launched via the CXF API for the >> > purpose, seems perfectly happy. Yet the client, after 30K successful >> > interactions, gets the following backtrace. MTOM is in use here. 2.3.0 >> > is in use here. >> > >> > >> > >> > >> > Exception in thread "main" javax.xml.ws.WebServiceException: Could not >> > send Message. >> > at >> > org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135) >> > at $Proxy20.addDocument(Unknown Source) >> > at >> > com.basistech.jug.FileHarvesterService$HarvestOutputToDatabase.collect(F >> > ileHarvesterService.java:116) at >> > com.basistech.jug.FileHarvester.process(FileHarvester.java:178) at >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:137) at >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:149) at >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:149) at >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:149) at >> > com.basistech.jug.FileHarvester.harvest(FileHarvester.java:106) at >> > com.basistech.jug.FileHarvesterService.run(FileHarvesterService.java:70) >> > at >> > com.basistech.jug.FileHarvesterService.main(FileHarvesterService.java:13 >> > 8) Caused by: java.net.SocketException: SocketException invoking >> > http://localhost:6006/documentDatabase: Unexpected end of file from >> > server >> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> > Method) at >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA >> > ccessorImpl.java:39) at >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons >> > tructorAccessorImpl.java:27) at >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513) at >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapExcepti >> > on(HTTPConduit.java:1995) at >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTP >> > Conduit.java:1980) at >> > org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) >> > at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:662) >> > at >> > org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingI >> > nterceptor.handleMessage(MessageSenderInterceptor.java:62) at >> > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC >> > hain.java:247) at >> > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516) at >> > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313) at >> > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265) at >> > org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73) at >> > org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124) >> > ... 10 more >> > Caused by: java.net.SocketException: Unexpected end of file from server >> > at >> > sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) at >> > sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) at >> > sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:766) at >> > sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) at >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnec >> > tion.java:1200) at >> > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) >> > at >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResp >> > onseInternal(HTTPConduit.java:2102) at >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResp >> > onse(HTTPConduit.java:2071) at >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTP >> > Conduit.java:1925) ... 19 more > > -- > Daniel Kulp > [email protected] > http://dankulp.com/blog >
