Hello Claus, First issue got fixed after enabling the stream cache pool.
Could you please assist for 2nd one ? Thank you, Karthik On Tue, 6 May 2025, 10:35 am Claus Ibsen, <claus.ib...@gmail.com> wrote: > Hi > > Enable spool on stream caching > https://camel.apache.org/manual/stream-caching.html > > > On Tue, May 6, 2025 at 4:37 AM KARTHIK PRABHU N <karthikprabh...@gmail.com > > > wrote: > > > Hello Team, > > > > Good morning & hope you are doing fine ! > > > > I am working on a use-case / concept to generate, download and upload > very > > large file ( > 1GB file size <100GB) should support both binary and > > non-binary file type. To achieve this concept, I tried Camel’s File or > > Stream component to read / write file as stream and REST component that > > plugins to servlet component which exposes routes as REST endpoint. > > > > > > > > The download route designed to searches for given file and once found, > > streams the data to target file location ( intermediate step we can > ignore) > > as well as http response body. While testing the route using curl tool, > we > > observed multiple issues as listed below. > > > > > > > > 1. Route streams data to client perfectly unless the file size <= 2GB, > > beyond this size the route will throw below error. The exception > clearly > > indicates that JVM is trying to allocate an array with the size of > about > > 2.13 GB and java arrays are limited to max size of array index around > 2GB . > > I tried to understand how streams work internally but somehow, I am > lost. > > Could you please assist how I can handle this scenario may be read and > > publish data as chucks ? > > > > > > > > > > *java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too > > large at > > > java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) > > * > > > > *at > > > java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) > > * > > > > *at > > java.base/java.io > .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) > > * > > > > *at > > java.base/java.io > .ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)* > > > > *at > > > org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) > > * > > > > *at java.base/java.io.InputStream.transferTo(InputStream.java:783) at > > org.apache.camel.util.IOHelper.copy(IOHelper.java:165)* > > > > > > > > > > > > 2. How to increase the download rate ? As the file size is large, the > > total time taken to complete the download is very long. I tried with > higher > > buffer range for file component & max response size in the servlet > consumer > > level but didn’t work. > > > > > > > > > > > > > > > > Could you please assist. Let us know if you need more details form > myside. > > > > > > > > Thank you, > > > > Karthik Prabhu > > > > > > > > > > > > -- > Claus Ibsen > ----------------- > @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
