[
https://issues.apache.org/jira/browse/CAMEL-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821927#comment-16821927
]
MykhailoVlakh commented on CAMEL-13433:
---------------------------------------
Hello [~ancosen], thank you for your time spent on this issue.
The biggest issue is that the producer loads entire content into memory and
this causes OOM if content is large. I believe this is incorrect, especially if
it does this only to calculate size of the provided content that is already
known. Cannot we just process input as stream to make sure that we won't occupy
all heap space when we upload large content? I think this should be the right
fix. Do you agree?
> S3: Exchange body stream is loaded into memory to calculate content length
> which is already set via headers
> -----------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-13433
> URL: https://issues.apache.org/jira/browse/CAMEL-13433
> Project: Camel
> Issue Type: Bug
> Components: camel-aws
> Affects Versions: 2.22.1, 2.23.2, 3.0.0-M1
> Reporter: MykhailoVlakh
> Assignee: Andrea Cosentino
> Priority: Major
>
> My code uploads large files to S3 storage (500 MB and bigger). I create an
> exchange with properly set meta information headers including the
> _CamelAwsS3ContentLength_, as a body I provide a stream. Everything worked
> well in Camel 2.14.3 but when I upgraded to Camel ti version 2.22.1 I
> suddenly began getting OOM issues from time to time. I did some debugging and
> found that the issue is in *S3Producer* that loads content of the body stream
> into memory.
> {code:java}
> public void processSingleOp(final Exchange exchange) throws Exception {
> ObjectMetadata objectMetadata = determineMetadata(exchange); <== here we
> already set length
> File filePayload = null;
> InputStream is = null;
> ByteArrayOutputStream baos = null;
> Object obj = exchange.getIn().getMandatoryBody();
> PutObjectRequest putObjectRequest = null;
> // Need to check if the message body is WrappedFile
> if (obj instanceof WrappedFile) {
> obj = ((WrappedFile<?>)obj).getFile();
> }
> if (obj instanceof File) {
> filePayload = (File)obj;
> is = new FileInputStream(filePayload);
> } else {
> is = exchange.getIn().getMandatoryBody(InputStream.class);
> baos = determineLengthInputStream(is); <====== the issue is here
> objectMetadata.setContentLength(baos.size());
> is = new ByteArrayInputStream(baos.toByteArray());
> }
> ...
> {code}
> This is a defect since, as I mentioned, I already provide content length via
> headers and in the first line of the method populates that information into
> object metadata.
> I think there should be some if check that skips content length size
> calculation if it is already known.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
