Hi On 24/01/12 09:34, Towers, Vanessa wrote:
UNCLASSIFIEDHi Jeff, Here are some more details. Writing the file out: The service itself is getting a DataHandler from the MultipartBody param and the file is written out by getting an InputStream from the DataHandler. Uploading: For my integration tests, I have tried two approaches. I wonder if you could comment on these approaches in terms of good or bad. Approach 1 - Using org.apache.commons.httpclient.* File f = new File("src/test/resources/massive.mpg"); Part[] parts = { ... new FilePart(f.getName(), f) }; PostMethod filePost = new PostMethod(address); filePost.setRequestEntity(new MultipartRequestEntity(parts, filePost.getParams())); HttpClient client = new HttpClient(); int result = client.executeMethod(filePost); Approach 2 - Using org.apache.cxf.jaxrs.client.WebClient WebClient client = WebClient.create(serviceAddy); client.path("uploadService/attachment"); client.type("multipart/form-data"); client.accept("text/html"); ContentDisposition cd = new ContentDisposition("attachment;filename=massive.mpg"); File f = new File("src/test/resources/massive.mpg"); InputStream is = new FileInputStream(f); Attachment att = new Attachment("root", is, cd); Response response = client.post(new MultipartBody(att));
I'm biased toward recommending the 2nd one :-).
With the knowledge of these details what do you think is limiting/slowing my upload and causing these problems? Finally, I have had mixed results today. The problem is still intermittent however at times I was able to upload a 550MB file without seeing the issue which is promising, but other times the issue was back. Also, I saw this in my log today - in this case the file was uploaded and written out twice and the issue was present :S 2012-01-24 19:49:21,774 INFO - I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server localhost failed to respond 2012-01-24 19:49:21,774 INFO - Retrying request
It appears the problem is that the client is not configured to wait long enough for a response to come back in case of realy big attachments being posted.
With WebClient you can do: WebClient.getConfig(client).getHttpConduit().getClient().setReceiveTimeout(timeout);
Regarding the jax-rs property "attachment-directory", this is still behaving strangely for me. For example, currently I have the value="C:\temp" set and when executing the tests from my Windows box, earlier today I did actually see some files being created in this directory (with names like cos2122005331006583203tmp) on upload. But now I'm not seeing any changes in this directory, however I am noticing similar files being created under C:\Users\<me>\AppData\Local\Temp when I run the tests. These temp files are the same size as the massive files I am trying to upload so they are a match. Also, when I run the tests on my CentOS box I use a value like "/tmp" and make sure this directory exits and has the right permissions etc. but nothing ever is created in there. Shouldn't I be able to locate these files and would you expect I would need to manually clean them up?
I believe the temp files are supposed to be removed automatically, may be it does not work predictably on Windows though.
Cheers, Sergey
Thanks for your ideas so far. -----Original Message----- From: Jeff Wang [mailto:[email protected]] Sent: Monday, 23 January 2012 7:50 PM To: [email protected] Subject: Re: Issue uploading large attachment to JAX-RS service If your service runs fine with a smaller file, and you are running into trouble with a particular file size, that means you are running into memory issues. The reason why you are experiencing a 2x expected run-time for that particular test is because of all the memory garbage collections that the VM is supposed to do. My guess is that you are passing in the file as an byte array or similar. This, unless you have a need to do so, is probably not a good idea. Pass it in as an input stream, and write it out to your particular storage device. (One of my services directly reads from the input stream and writes to a file-based output stream.) That way, your memory requirements are just the buffers for the stream and you should be able to do whatever sized file you need. CXF does not have a file limitation. It's limited by your allocated memory and the way that you map input types. Jeff On Sun, Jan 22, 2012 at 4:21 PM, noosy <[email protected]> wrote:Hi, I have a CXF JAX-RS upload service which consumes("multipart/form-data") and accepts a MultipartBody object as a param. I have an integration test which tries to see how big a file I can upload to this service. I am able to upload a file of size 321 MB successfully. This is the largest file which passes for me. The issue is, intermittently when I run this test, the test passes, but it takes ages to run (twice as long as usual) and I see these warnings in the console: 20-Jan-2012 06:47:37 WARNING: Interceptor for {http://my.company/}UploadService has thrown exception, unwinding now 20-Jan-2012 06:47:37 org.apache.cxf.interceptor.Fault: Could not send Message. 20-Jan-2012 06:47:37 at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndin gInterceptor.handleMessage(MessageSenderInterceptor.java:64) 20-Jan-2012 06:47:37 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto rChain.java:263) 20-Jan-2012 06:47:37 at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outg oingChainInterceptor.java:77) 20-Jan-2012 06:47:37 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto rChain.java:263) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitia tionObserver.java:123) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceReques t(JettyHTTPDestination.java:323) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(Jet tyHTTPDestination.java:289) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPH andler.java:72) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandle r.java:942) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler .java:878) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.ja va:117) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Conte xtHandlerCollection.java:250) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper. java:110) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.Server.handle(Server.java:345) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.j ava:441) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpCon nection.java:936) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnectio n.java:52) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEnd Point.java:586) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndP oint.java:44) 20-Jan-2012 06:47:37 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool .java:598) 20-Jan-2012 06:47:37 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool. java:533) 20-Jan-2012 06:47:37 at java.lang.Thread.run(Thread.java:662) 20-Jan-2012 06:47:37 Caused by: org.eclipse.jetty.io.EofException 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:91 9) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:812) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection.commitResponse(HttpConnection. java:572) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection$Output.close(HttpConnection.ja va:992) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStr eam.close(AbstractHTTPDestination.java:650) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56 ) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelCondu it.close(AbstractHTTPDestination.java:593) 20-Jan-2012 06:47:37 at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndin gInterceptor.handleMessage(MessageSenderInterceptor.java:62) 20-Jan-2012 06:47:37 ... 23 more 20-Jan-2012 06:47:37 Caused by: java.nio.channels.ClosedChannelException 20-Jan-2012 06:47:37 at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:13 5) 20-Jan-2012 06:47:37 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:357) 20-Jan-2012 06:47:37 at java.nio.channels.SocketChannel.write(SocketChannel.java:360) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoin t.java:346) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:28 4) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndP oint.java:300) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:84 6) 20-Jan-2012 06:47:37 ... 30 more 20-Jan-2012 06:47:37 Jan 20, 2012 6:47:37 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging 20-Jan-2012 06:47:37 WARNING: Interceptor for {http://my.company/}UploadService has thrown exception, unwinding now 20-Jan-2012 06:47:37 org.apache.cxf.interceptor.Fault: XML_WRITE_EXC 20-Jan-2012 06:47:37 at org.apache.cxf.binding.xml.interceptor.XMLFaultOutInterceptor.handleMe ssage(XMLFaultOutInterceptor.java:87) 20-Jan-2012 06:47:37 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto rChain.java:263) 20-Jan-2012 06:47:37 at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessa ge(AbstractFaultChainInitiatorObserver.java:107) 20-Jan-2012 06:47:37 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto rChain.java:323) 20-Jan-2012 06:47:37 at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outg oingChainInterceptor.java:77) 20-Jan-2012 06:47:37 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto rChain.java:263) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitia tionObserver.java:123) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceReques t(JettyHTTPDestination.java:323) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(Jet tyHTTPDestination.java:289) 20-Jan-2012 06:47:37 at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPH andler.java:72) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandle r.java:942) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler .java:878) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.ja va:117) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Conte xtHandlerCollection.java:250) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper. java:110) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.Server.handle(Server.java:345) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.j ava:441) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpCon nection.java:936) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnectio n.java:52) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEnd Point.java:586) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndP oint.java:44) 20-Jan-2012 06:47:37 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool .java:598) 20-Jan-2012 06:47:37 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool. java:533) 20-Jan-2012 06:47:37 at java.lang.Thread.run(Thread.java:662) 20-Jan-2012 06:47:37 Caused by: com.ctc.wstx.exc.WstxIOException: null 20-Jan-2012 06:47:37 at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:257) 20-Jan-2012 06:47:37 at org.apache.cxf.binding.xml.interceptor.XMLFaultOutInterceptor.handleMe ssage(XMLFaultOutInterceptor.java:85) 20-Jan-2012 06:47:37 ... 25 more 20-Jan-2012 06:47:37 Caused by: org.eclipse.jetty.io.EofException 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:91 9) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java: 452) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:94) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpConnection$Output.flush(HttpConnection.ja va:1009) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:173) 20-Jan-2012 06:47:37 at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:101) 20-Jan-2012 06:47:37 at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOut putStream.java:46) 20-Jan-2012 06:47:37 at com.ctc.wstx.sw.EncodingXmlWriter.flushBuffer(EncodingXmlWriter.java:6 97) 20-Jan-2012 06:47:37 at com.ctc.wstx.sw.EncodingXmlWriter.flush(EncodingXmlWriter.java:171) 20-Jan-2012 06:47:37 at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:255) 20-Jan-2012 06:47:37 ... 26 more 20-Jan-2012 06:47:37 Caused by: java.nio.channels.ClosedChannelException 20-Jan-2012 06:47:37 at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:13 5) 20-Jan-2012 06:47:37 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:357) 20-Jan-2012 06:47:37 at java.nio.channels.SocketChannel.write(SocketChannel.java:360) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoin t.java:346) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:28 4) 20-Jan-2012 06:47:37 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndP oint.java:300) 20-Jan-2012 06:47:37 at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:84 6) 20-Jan-2012 06:47:37 ... 35 more This concerns me and it means the build (runs unit and integration tests) takes much longer than it should to run. A couple of things. I am using CXF 2.5.1. I am using AbstractBusTestServerBase to do my testing (standing up a test server). I added the following to my spring context file but I'm not sure if it is even doing anything. I cannot see the attachment directory being created (where should I be able to find it assuming it is not being automatically cleaned up): <jaxrs:server id="upload-service" address="/upload-service"> <jaxrs:serviceBeans> <ref bean="uploadService"/> </jaxrs:serviceBeans> <jaxrs:properties> <entry key="attachment-directory" value="/temp/attachments"/> <entry key="attachment-memory-threshold" value="4000000"/> <entry key="attachment-max-size" value="409600000"/> </jaxrs:properties> </jaxrs:server> How big a file should I be able to upload before tweaking app server settings (assuming this could boost it further)? Appreciate any pointers. Thanks. -- View this message in context: http://cxf.547215.n5.nabble.com/Issue-uploading-large-attachment-to-JA X-RS-service-tp5165047p5165047.html Sent from the cxf-user mailing list archive at Nabble.com.IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.
-- Sergey Beryozkin Talend Community Coders http://coders.talend.com/ Blog: http://sberyozkin.blogspot.com
