Hi Enable spool on stream caching https://camel.apache.org/manual/stream-caching.html
On Tue, May 6, 2025 at 4:37 AM KARTHIK PRABHU N <karthikprabh...@gmail.com> wrote: > Hello Team, > > Good morning & hope you are doing fine ! > > I am working on a use-case / concept to generate, download and upload very > large file ( > 1GB file size <100GB) should support both binary and > non-binary file type. To achieve this concept, I tried Camel’s File or > Stream component to read / write file as stream and REST component that > plugins to servlet component which exposes routes as REST endpoint. > > > > The download route designed to searches for given file and once found, > streams the data to target file location ( intermediate step we can ignore) > as well as http response body. While testing the route using curl tool, we > observed multiple issues as listed below. > > > > 1. Route streams data to client perfectly unless the file size <= 2GB, > beyond this size the route will throw below error. The exception clearly > indicates that JVM is trying to allocate an array with the size of about > 2.13 GB and java arrays are limited to max size of array index around 2GB . > I tried to understand how streams work internally but somehow, I am lost. > Could you please assist how I can handle this scenario may be read and > publish data as chucks ? > > > > > *java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too > large at > java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) > * > > *at > java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) > * > > *at > java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) > * > > *at > java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)* > > *at > org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) > * > > *at java.base/java.io.InputStream.transferTo(InputStream.java:783) at > org.apache.camel.util.IOHelper.copy(IOHelper.java:165)* > > > > > > 2. How to increase the download rate ? As the file size is large, the > total time taken to complete the download is very long. I tried with higher > buffer range for file component & max response size in the servlet consumer > level but didn’t work. > > > > > > > > Could you please assist. Let us know if you need more details form myside. > > > > Thank you, > > Karthik Prabhu > > > > > -- Claus Ibsen ----------------- @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
