On Monday 06 December 2010 7:44:04 pm Benson Margulies wrote: > I have a theory. It's related to multiple MTOM attachments, and > someone is closing the main stream thinking that they are closing a > cache stream.
Well, that should still be tracked. I think the MTOM things, at close(), check if there is another attachment and if so, they don't close the underlying stream. At least that's the theory. Dan > On Mon, Dec 6, 2010 at 2:49 PM, Daniel Kulp <[email protected]> wrote: > > Well, in both cases, it looks like something in the middle is closing the > > stream. I have NO idea what would cause that. > > > > Tracing through the calls in the stack trace, it's basically saying the > > call to the underlying NIO ByteChannel's isOpen method is returning > > false. So SOMETHING is closing it. > > > > How reproducible is it? Here is my thought on how to debug: > > > > Add an interceptor at the very beginning of the IN chain and fairly early > > (before AttachmentOut) of the OUT chain that wrappers the input/output > > streams that just override the "close()" method. That override would > > do a "new Throwable().fillInStackTrace()" and sticks it on the message > > using a private key. > > > > An separate interceptor on the fault out chain would grab those and print > > those if they are there. THAT way, if something is calling close that > > shouldn't be, we can know where. If they are both null, then we know > > it's not us calling close and I'm not sure what would be the issue. > > > > Dan > > > > On Saturday 04 December 2010 8:57:09 am Benson Margulies wrote: > >> Possibly related on the server side: > >> > >> 2010-12-04 08:49:29,676 org.apache.cxf.phase.PhaseInterceptorChain - > >> Interceptor for > >> {http://jdd.basistech.com/}DocumentDatabaseService#{http://jdd.basistech > >> .co m/}addDocument has thrown exception, unwinding now > >> org.apache.cxf.interceptor.Fault: Could not write attachments. > >> at > >> org.apache.cxf.interceptor.AttachmentOutInterceptor.handleMessage(Attach > >> me ntOutInterceptor.java:71) at > >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC > >> ha in.java:247) at > >> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outgoi > >> ng ChainInterceptor.java:77) at > >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC > >> ha in.java:247) at > >> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiati > >> on Observer.java:113) at > >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest( > >> Je ttyHTTPDestination.java:311) at > >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(Jetty > >> HT TPDestination.java:280) at > >> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHan > >> dl er.java:72) at > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler. > >> ja va:922) at > >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.j > >> av a:860) at > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java > >> :1 17) at > >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Context > >> Ha ndlerCollection.java:247) at > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.ja > >> va > >> > >> :114) at org.eclipse.jetty.server.Server.handle(Server.java:352) > >> > >> at > >> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.jav > >> a: 596) at > >> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConne > >> ct ion.java:1066) at > >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at > >> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at > >> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) > >> at > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndP > >> oin t.java:510) at > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelE > >> nd Point.java:34) at > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoi > >> nt .java:40) at > >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.ja > >> va > >> > >> :450) at java.lang.Thread.run(Thread.java:680) > >> > >> Caused by: org.eclipse.jetty.io.EofException > >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:148) > >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:92) > >> at > >> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutpu > >> tS tream.java:46) at > >> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutpu > >> tS tream.java:55) at > >> org.apache.cxf.attachment.AttachmentSerializer.writeProlog(AttachmentSer > >> ia lizer.java:142) at > >> org.apache.cxf.interceptor.AttachmentOutInterceptor.handleMessage(Attach > >> me ntOutInterceptor.java:69) ... 23 more > >> 2010-12-04 08:49:29,683 org.apache.cxf.phase.PhaseInterceptorChain - > >> Interceptor for > >> {http://jdd.basistech.com/}DocumentDatabaseService#{http://jdd.basistech > >> .co m/}addDocument has thrown exception, unwinding now > >> org.apache.cxf.binding.soap.SoapFault: Error writing to XMLStreamWriter. > >> at > >> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor$SoapOutEnding > >> In terceptor.handleMessage(SoapOutInterceptor.java:288) at > >> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor$SoapOutEnding > >> In terceptor.handleMessage(SoapOutInterceptor.java:270) at > >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC > >> ha in.java:247) at > >> org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage > >> (A bstractFaultChainInitiatorObserver.java:99) at > >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC > >> ha in.java:303) at > >> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outgoi > >> ng ChainInterceptor.java:77) at > >> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC > >> ha in.java:247) at > >> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiati > >> on Observer.java:113) at > >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest( > >> Je ttyHTTPDestination.java:311) at > >> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(Jetty > >> HT TPDestination.java:280) at > >> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHan > >> dl er.java:72) at > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler. > >> ja va:922) at > >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.j > >> av a:860) at > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java > >> :1 17) at > >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Context > >> Ha ndlerCollection.java:247) at > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.ja > >> va > >> > >> :114) at org.eclipse.jetty.server.Server.handle(Server.java:352) > >> > >> at > >> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.jav > >> a: 596) at > >> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConne > >> ct ion.java:1066) at > >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at > >> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at > >> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) > >> at > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndP > >> oin t.java:510) at > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelE > >> nd Point.java:34) at > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoi > >> nt .java:40) at > >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.ja > >> va > >> > >> :450) at java.lang.Thread.run(Thread.java:680) > >> > >> Caused by: com.ctc.wstx.exc.WstxIOException: null > >> at > >> com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1 > >> 43 1) at > >> com.ctc.wstx.sw.BaseStreamWriter.writeEndDocument(BaseStreamWriter.java: > >> 55 3) at > >> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor$SoapOutEnding > >> In terceptor.handleMessage(SoapOutInterceptor.java:282) ... 26 more > >> Caused by: org.eclipse.jetty.io.EofException > >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:148) > >> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:92) > >> at > >> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutpu > >> tS tream.java:46) at > >> com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:100) at > >> com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:225) > >> at > >> com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:198) > >> at > >> com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1 > >> 42 9) ... 28 more > >> > >> On Sat, Dec 4, 2010 at 7:19 AM, Benson Margulies <[email protected]> > > > > wrote: > >> > My server, which is a JAX-WS endpoint launched via the CXF API for the > >> > purpose, seems perfectly happy. Yet the client, after 30K successful > >> > interactions, gets the following backtrace. MTOM is in use here. 2.3.0 > >> > is in use here. > >> > > >> > > >> > > >> > > >> > Exception in thread "main" javax.xml.ws.WebServiceException: Could not > >> > send Message. > >> > at > >> > org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135 > >> > ) at $Proxy20.addDocument(Unknown Source) > >> > at > >> > com.basistech.jug.FileHarvesterService$HarvestOutputToDatabase.collect > >> > (F ileHarvesterService.java:116) at > >> > com.basistech.jug.FileHarvester.process(FileHarvester.java:178) at > >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:137) at > >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:149) at > >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:149) at > >> > com.basistech.jug.FileHarvester.harvestDir(FileHarvester.java:149) at > >> > com.basistech.jug.FileHarvester.harvest(FileHarvester.java:106) at > >> > com.basistech.jug.FileHarvesterService.run(FileHarvesterService.java:7 > >> > 0) at > >> > com.basistech.jug.FileHarvesterService.main(FileHarvesterService.java: > >> > 13 8) Caused by: java.net.SocketException: SocketException invoking > >> > http://localhost:6006/documentDatabase: Unexpected end of file from > >> > server > >> > at > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > >> > at > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo > >> > rA ccessorImpl.java:39) at > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo > >> > ns tructorAccessorImpl.java:27) at > >> > java.lang.reflect.Constructor.newInstance(Constructor.java:513) at > >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapExcep > >> > ti on(HTTPConduit.java:1995) at > >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HT > >> > TP Conduit.java:1980) at > >> > org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56 > >> > ) at > >> > org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:662) > >> > at > >> > org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndin > >> > gI nterceptor.handleMessage(MessageSenderInterceptor.java:62) at > >> > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercept > >> > orC hain.java:247) at > >> > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516) at > >> > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313) at > >> > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265) at > >> > org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73) at > >> > org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124 > >> > ) ... 10 more > >> > Caused by: java.net.SocketException: Unexpected end of file from > >> > server at > >> > sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) at > >> > sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) at > >> > sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:766) at > >> > sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) at > >> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConn > >> > ec tion.java:1200) at > >> > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) > >> > at > >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRe > >> > sp onseInternal(HTTPConduit.java:2102) at > >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRe > >> > sp onse(HTTPConduit.java:2071) at > >> > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HT > >> > TP Conduit.java:1925) ... 19 more > > > > -- > > Daniel Kulp > > [email protected] > > http://dankulp.com/blog -- Daniel Kulp [email protected] http://dankulp.com/blog
