UNCLASSIFIED
Hi Jeff,
Here are some more details.
Writing the file out: The service itself is getting a DataHandler from the
MultipartBody param and the file is written out by getting an InputStream from
the DataHandler.
Uploading: For my integration tests, I have tried two approaches. I wonder if
you could comment on these approaches in terms of good or bad.
Approach 1 - Using org.apache.commons.httpclient.*
File f = new File("src/test/resources/massive.mpg");
Part[] parts = {
...
new FilePart(f.getName(), f)
};
PostMethod filePost = new PostMethod(address); filePost.setRequestEntity(new
MultipartRequestEntity(parts, filePost.getParams())); HttpClient client = new
HttpClient(); int result = client.executeMethod(filePost);
Approach 2 - Using org.apache.cxf.jaxrs.client.WebClient
WebClient client = WebClient.create(serviceAddy);
client.path("uploadService/attachment");
client.type("multipart/form-data");
client.accept("text/html");
ContentDisposition cd = new
ContentDisposition("attachment;filename=massive.mpg");
File f = new File("src/test/resources/massive.mpg");
InputStream is = new FileInputStream(f); Attachment att = new
Attachment("root", is, cd); Response response = client.post(new
MultipartBody(att));
With the knowledge of these details what do you think is limiting/slowing my
upload and causing these problems?
Finally, I have had mixed results today. The problem is still intermittent
however at times I was able to upload a 550MB file without seeing the issue
which is promising, but other times the issue was back.
Also, I saw this in my log today - in this case the file was uploaded and
written out twice and the issue was present :S
2012-01-24 19:49:21,774 INFO - I/O exception
(org.apache.commons.httpclient.NoHttpResponseException) caught when processing
request: The server localhost failed to respond
2012-01-24 19:49:21,774 INFO - Retrying request
Regarding the jax-rs property "attachment-directory", this is still behaving
strangely for me. For example, currently I have the value="C:\temp" set and
when executing the tests from my Windows box, earlier today I did actually see
some files being created in this directory (with names like
cos2122005331006583203tmp) on upload. But now I'm not seeing any changes in
this directory, however I am noticing similar files being created under
C:\Users\<me>\AppData\Local\Temp when I run the tests. These temp files are the
same size as the massive files I am trying to upload so they are a match. Also,
when I run the tests on my CentOS box I use a value like "/tmp" and make sure
this directory exits and has the right permissions etc. but nothing ever is
created in there. Shouldn't I be able to locate these files and would you
expect I would need to manually clean them up?
Thanks for your ideas so far.
-----Original Message-----
From: Jeff Wang [mailto:[email protected]]
Sent: Monday, 23 January 2012 7:50 PM
To: [email protected]
Subject: Re: Issue uploading large attachment to JAX-RS service
If your service runs fine with a smaller file, and you are running into trouble
with a particular file size, that means you are running into memory issues. The
reason why you are experiencing a 2x expected run-time for that particular test
is because of all the memory garbage collections that the VM is supposed to do.
My guess is that you are passing in the file as an byte array or similar.
This, unless you have a need to do so, is probably not a good idea. Pass it in
as an input stream, and write it out to your particular storage device. (One
of my services directly reads from the input stream and writes to a file-based
output stream.) That way, your memory requirements are just the buffers for
the stream and you should be able to do whatever sized file you need.
CXF does not have a file limitation. It's limited by your allocated memory and
the way that you map input types.
Jeff
On Sun, Jan 22, 2012 at 4:21 PM, noosy
<[email protected]> wrote:
> Hi,
>
> I have a CXF JAX-RS upload service which
> consumes("multipart/form-data") and accepts a MultipartBody object as a param.
>
> I have an integration test which tries to see how big a file I can
> upload to this service. I am able to upload a file of size 321 MB
> successfully. This is the largest file which passes for me. The issue
> is, intermittently when I run this test, the test passes, but it takes
> ages to run (twice as long as
> usual) and I see these warnings in the console:
>
> 20-Jan-2012 06:47:37 WARNING: Interceptor for
> {http://my.company/}UploadService has thrown exception, unwinding now
> 20-Jan-2012 06:47:37 org.apache.cxf.interceptor.Fault: Could not
> send Message.
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndin
> gInterceptor.handleMessage(MessageSenderInterceptor.java:64)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto
> rChain.java:263)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outg
> oingChainInterceptor.java:77)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto
> rChain.java:263)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitia
> tionObserver.java:123)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceReques
> t(JettyHTTPDestination.java:323)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(Jet
> tyHTTPDestination.java:289)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPH
> andler.java:72)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandle
> r.java:942)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler
> .java:878)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.ja
> va:117)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Conte
> xtHandlerCollection.java:250)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.
> java:110)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.Server.handle(Server.java:345)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.j
> ava:441)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpCon
> nection.java:936)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnectio
> n.java:52)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEnd
> Point.java:586)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndP
> oint.java:44)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool
> .java:598)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.
> java:533)
> 20-Jan-2012 06:47:37 at
> java.lang.Thread.run(Thread.java:662)
> 20-Jan-2012 06:47:37 Caused by: org.eclipse.jetty.io.EofException
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:91
> 9)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:812)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection.commitResponse(HttpConnection.
> java:572)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection$Output.close(HttpConnection.ja
> va:992)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStr
> eam.close(AbstractHTTPDestination.java:650)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56
> )
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelCondu
> it.close(AbstractHTTPDestination.java:593)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndin
> gInterceptor.handleMessage(MessageSenderInterceptor.java:62)
> 20-Jan-2012 06:47:37 ... 23 more
> 20-Jan-2012 06:47:37 Caused by:
> java.nio.channels.ClosedChannelException
> 20-Jan-2012 06:47:37 at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:13
> 5)
> 20-Jan-2012 06:47:37 at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:357)
> 20-Jan-2012 06:47:37 at
> java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoin
> t.java:346)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:28
> 4)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndP
> oint.java:300)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:84
> 6)
> 20-Jan-2012 06:47:37 ... 30 more
> 20-Jan-2012 06:47:37 Jan 20, 2012 6:47:37 AM
> org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
> 20-Jan-2012 06:47:37 WARNING: Interceptor for
> {http://my.company/}UploadService has thrown exception, unwinding now
> 20-Jan-2012 06:47:37 org.apache.cxf.interceptor.Fault:
> XML_WRITE_EXC
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.binding.xml.interceptor.XMLFaultOutInterceptor.handleMe
> ssage(XMLFaultOutInterceptor.java:87)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto
> rChain.java:263)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessa
> ge(AbstractFaultChainInitiatorObserver.java:107)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto
> rChain.java:323)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(Outg
> oingChainInterceptor.java:77)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercepto
> rChain.java:263)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitia
> tionObserver.java:123)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceReques
> t(JettyHTTPDestination.java:323)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(Jet
> tyHTTPDestination.java:289)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPH
> andler.java:72)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandle
> r.java:942)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler
> .java:878)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.ja
> va:117)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Conte
> xtHandlerCollection.java:250)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.
> java:110)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.Server.handle(Server.java:345)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.j
> ava:441)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpCon
> nection.java:936)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnectio
> n.java:52)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEnd
> Point.java:586)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndP
> oint.java:44)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool
> .java:598)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.
> java:533)
> 20-Jan-2012 06:47:37 at
> java.lang.Thread.run(Thread.java:662)
> 20-Jan-2012 06:47:37 Caused by: com.ctc.wstx.exc.WstxIOException:
> null
> 20-Jan-2012 06:47:37 at
> com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:257)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.binding.xml.interceptor.XMLFaultOutInterceptor.handleMe
> ssage(XMLFaultOutInterceptor.java:85)
> 20-Jan-2012 06:47:37 ... 25 more
> 20-Jan-2012 06:47:37 Caused by: org.eclipse.jetty.io.EofException
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:91
> 9)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:
> 452)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:94)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpConnection$Output.flush(HttpConnection.ja
> va:1009)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:173)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:101)
> 20-Jan-2012 06:47:37 at
> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOut
> putStream.java:46)
> 20-Jan-2012 06:47:37 at
> com.ctc.wstx.sw.EncodingXmlWriter.flushBuffer(EncodingXmlWriter.java:6
> 97)
> 20-Jan-2012 06:47:37 at
> com.ctc.wstx.sw.EncodingXmlWriter.flush(EncodingXmlWriter.java:171)
> 20-Jan-2012 06:47:37 at
> com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:255)
> 20-Jan-2012 06:47:37 ... 26 more
> 20-Jan-2012 06:47:37 Caused by:
> java.nio.channels.ClosedChannelException
> 20-Jan-2012 06:47:37 at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:13
> 5)
> 20-Jan-2012 06:47:37 at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:357)
> 20-Jan-2012 06:47:37 at
> java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoin
> t.java:346)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:28
> 4)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndP
> oint.java:300)
> 20-Jan-2012 06:47:37 at
> org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:84
> 6)
> 20-Jan-2012 06:47:37 ... 35 more
>
> This concerns me and it means the build (runs unit and integration
> tests) takes much longer than it should to run.
>
> A couple of things.
>
> I am using CXF 2.5.1.
>
> I am using AbstractBusTestServerBase to do my testing (standing up a
> test server).
>
> I added the following to my spring context file but I'm not sure if it
> is even doing anything. I cannot see the attachment directory being
> created (where should I be able to find it assuming it is not being
> automatically cleaned up):
>
> <jaxrs:server id="upload-service" address="/upload-service">
> <jaxrs:serviceBeans>
> <ref bean="uploadService"/>
> </jaxrs:serviceBeans>
> <jaxrs:properties>
> <entry key="attachment-directory"
> value="/temp/attachments"/>
> <entry key="attachment-memory-threshold" value="4000000"/>
> <entry key="attachment-max-size" value="409600000"/>
> </jaxrs:properties>
> </jaxrs:server>
>
> How big a file should I be able to upload before tweaking app server
> settings (assuming this could boost it further)?
>
> Appreciate any pointers.
>
> Thanks.
>
> --
> View this message in context:
> http://cxf.547215.n5.nabble.com/Issue-uploading-large-attachment-to-JA
> X-RS-service-tp5165047p5165047.html
> Sent from the cxf-user mailing list archive at Nabble.com.
IMPORTANT: This email remains the property of the Department of Defence and is
subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have
received this email in error, you are requested to contact the sender and
delete the email.
