Hello Team, Good morning & hope you are doing fine !
I am working on a use-case / concept to generate, download and upload very large file ( > 1GB file size <100GB) should support both binary and non-binary file type. To achieve this concept, I tried Camel’s File or Stream component to read / write file as stream and REST component that plugins to servlet component which exposes routes as REST endpoint. The download route designed to searches for given file and once found, streams the data to target file location ( intermediate step we can ignore) as well as http response body. While testing the route using curl tool, we observed multiple issues as listed below. 1. Route streams data to client perfectly unless the file size <= 2GB, beyond this size the route will throw below error. The exception clearly indicates that JVM is trying to allocate an array with the size of about 2.13 GB and java arrays are limited to max size of array index around 2GB . I tried to understand how streams work internally but somehow, I am lost. Could you please assist how I can handle this scenario may be read and publish data as chucks ? *java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) * *at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) * *at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) * *at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)* *at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) * *at java.base/java.io.InputStream.transferTo(InputStream.java:783) at org.apache.camel.util.IOHelper.copy(IOHelper.java:165)* 2. How to increase the download rate ? As the file size is large, the total time taken to complete the download is very long. I tried with higher buffer range for file component & max response size in the servlet consumer level but didn’t work. Could you please assist. Let us know if you need more details form myside. Thank you, Karthik Prabhu
