Hi Claus, Sorry, I looked into the wrong branch. Starting with Camel 2.17 the change is already there, so if you have Camel 2.17 or later and enable stream caching on your route, you should not go OOM (at least not with the zip file unmarshalling, if there is no more space for a 45 MB byte array your route may fail later).
Best regards Stephan -----Original Message----- From: Claus Ibsen [mailto:[email protected]] Sent: Donnerstag, 24. November 2016 09:48 To: [email protected] Subject: Re: Unzip huge single entry file throws OutOfMemory I think the ZipFileDataFormat was recently improved to use stream caching so it spool to disk for big streams. Ticket is: CAMEL-9274 On Thu, Nov 24, 2016 at 9:44 AM, Siano, Stephan <[email protected]> wrote: > Hi, > > The ZipFileDataFormat reads the file content into a byte array, I guess your > server goes OOM while doing that. Theoretically it might be possible to > extend the component to support streaming if a stream caching is enabled on > the route, however someone would have to contribute that to Camel. > > Short term the only way I see for you is to increase the heap size of your > server. 45 MB is not so much memory, so this should be feasible. > > Best regards > > Stephan > > -----Original Message----- > From: adalfovo [mailto:[email protected]] > Sent: Mittwoch, 23. November 2016 10:41 > To: [email protected] > Subject: Unzip huge single entry file throws OutOfMemory > > Hi, > we have a 3.5 Mb zip file that contains a 45Mb text file. When we try to > unzip it the route throws an OutOfMemory. > Camel runs on a Jboss EAP 6.4.5 instance. > This is our route : > > from("servlet://?httpMethodRestrict=POST&servletName=XXX&matchOnUriPrefix=true") > .unmarshal().mimeMultipart() > .unmarshal().zipFile().... > > The stacktrace says: > > --------------------------------------------------------------------------------------------------------------------------------------- > RouteId ProcessorId Processor > Elapsed (ms) > [route1] [post] > [servlet:/?httpMethodRestrict=POST&matchOnUriPrefix=true&servletName=X > XX] [ 1008] [route1] [unmarshal1] > [unmarshal[org.apache.camel.model.dataformat.MimeMultipartDataFormat@5bbe2866] > ] [ 0] > [route1] [unmarshal2] > [unmarshal[org.apache.camel.model.dataformat.ZipFileDataFormat@1ebcbd75] > ] [ 1006] > > Stacktrace > ---------------------------------------------------------------------- > ----------------------------------------------------------------- > org.apache.camel.CamelExecutionException: Exception occurred during execution > on the exchange: Exchange[ID-123] > at > org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1706) > at > org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:349) > at > org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:84) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:121) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) > at > org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91) > at > org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:143) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) > at > org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.OutOfMemoryError: Java heap space > > > How can we fix this problem? > > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Unzip-huge-single-entry-file-throws- > OutOfMemory-tp5790404.html Sent from the Camel - Users mailing list > archive at Nabble.com. -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
