Benjamin BONNET created CAMEL-20728:
---------------------------------------
Summary: camel-aws2s3 Stream Producer should support multipart
loading
Key: CAMEL-20728
URL: https://issues.apache.org/jira/browse/CAMEL-20728
Project: Camel
Issue Type: New Feature
Components: camel-aws
Affects Versions: 4.5.0
Reporter: Benjamin BONNET
AWS2S3 Producer comes in two flavours: AWS2S3Producer and
AWS2S3StreamUploadProducer.
* AWS2S3Producer supports S3 multipart upload: exchange data will be chunked
into parts, parts will be sent separately to S3 and will be aggregated into
one file.
* AWS2S3StreamUploadProducer supports message streaming: incoming messages are
aggregated into a buffer that will be sent as one file to S3, as soon as some
condition is true (number of messages / size /timeout).
Unfortunately, AWS2S3StreamUploadProducer, although it is able to break a huge
flow of messages into parts (which is great for some use cases), is not able to
manage S3 multipart upload. So if you need to aggregate lots of messages into
one big file on S3 side, that means the producer will have to send that message
batch in one request (actually, a single part multipart request if you look at
implementation). If the volume is huge, that will blow up jvm heap.
So we propose to add multi-part support to AWS2S3StreamUploadProducer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
