[
https://issues.apache.org/jira/browse/CAMEL-26662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-26662.
---------------------------------
Resolution: Fixed
> camel-aws2-s3 - consumer gets every object of a poll before routing the
> first, and ignoreBody=true still downloads them
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26662
> URL: https://issues.apache.org/jira/browse/CAMEL-26662
> Project: Camel
> Issue Type: Improvement
> Components: camel-aws2-s3
> Affects Versions: 4.18.5, 4.22.2, 4.23.0
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
> Fix For: 4.24.0
>
>
> h3. Problem
> * *Every object of a poll is got before the first one is routed.*
> {{createExchanges}} calls GetObject for each listed key
> ({{components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java}}
> line 298 on main, commit 7978db3d42dd). With the default
> {{includeBody=true}}, it also reads each body into a byte[] (line 569). Only
> then does {{poll()}} call {{processBatch}} (line 207).
> ** The heap of a poll holds every body of the batch (default
> {{maxMessagesPerPoll=10}}), and the first exchange waits for the last
> download.
> ** With {{includeBody=false}}, every stream of the poll stays open, each
> holding a pooled connection, while the earlier exchanges are routed.
> ** Since SDK 2.32.32, a {{ResponseInputStream}} that is not read within 60 s
> is aborted. So a later exchange of the poll fails once the earlier ones
> together took longer than 60 s.
> ** With {{maxMessagesPerPoll}} above 50, the pool runs out: that is
> CAMEL-26661.
> * *ignoreBody=true still downloads every object.*
> ** The consumer does a GetObject and closes the stream without reading it
> (lines 606-607). The producer {{getObject}} with ignoreBody does the same
> ({{AWS2S3Producer.java}} lines 685-686 and 732-733).
> ** With the Apache HTTP client Camel uses by default, {{close()}} reads the
> rest of the body so that the connection can be reused (the SDK javadoc of
> {{ResponseInputStream}}; httpclient5 5.6.4
> {{ResponseEntityProxy.streamClosed}}). So the whole object is transferred
> anyway.
> * The docs say otherwise:
> {{components/camel-aws/camel-aws2-s3/src/main/docs/aws2-s3-consumer-examples.adoc}}
> line 228 reads "Use {{ignoreBody: true}} to get only the metadata."
> h3. How to reproduce
> An in-process fake S3 (JDK {{HttpServer}}) is used with the client Camel
> builds (Apache5, httpclient5 5.6.4). Measured on main:
> ||Case||Observed||
> |10 objects, maxMessagesPerPoll=10, includeBody=true|all 10 GetObject
> answered completely, then the 10 exchanges routed|
> |10 objects of 8 MiB, includeBody=false|all 10 GetObject sent before the
> first exchange is routed; at that point 9 of the 10 responses are still being
> sent|
> |1 object of 64 MiB, ignoreBody=true (consumer, and producer
> getObject)|67,108,864 of 67,108,864 bytes sent, body null|
> |Control: the same object, SDK ResponseInputStream.abort()|0.6 to 1.8 MB sent
> before the connection is closed|
> |2 objects, includeBody=false, the route takes 61 s on the first|the second
> exchange fails with IOException: Attempted read on closed stream|
> h3. Proposed change
> * Move the GetObject into {{processBatch}}: list in {{poll()}}, then get each
> listed object right before its exchange is routed. At most the objects of the
> exchanges in flight are then in memory or open.
> * Keep what CAMEL-25558 added:
> ** an object that no longer exists is skipped;
> ** an object that cannot be got is skipped and given to the exception handler;
> ** the poll fails only when none of the objects could be got.
> * {{CamelBatchSize}} and {{CamelBatchComplete}} are known only after the
> gets, so it must be decided what they count. This needs care.
> * For ignoreBody, use HeadObject instead of GetObject. HeadObject returns
> every field the consumer maps to headers (ETag, VersionId, ContentType,
> ContentLength, ContentEncoding, ContentDisposition, CacheControl,
> ServerSideEncryption, Expiration, ReplicationStatus, StorageClass, Metadata,
> LastModified; S3 service model), and it needs the same {{s3:GetObject}}
> permission.
> * The alternative is {{abort()}} instead of {{close()}}. It also stops the
> transfer (measured above), but it closes the connection, so the next request
> opens a new one.
> * Fix the example at line 228 of {{aws2-s3-consumer-examples.adoc}} along
> with the change.
> * This also fixes CAMEL-26661 (includeBody=false with maxMessagesPerPoll
> above 50), and the 60 s abort of the later streams of a poll in general.
> h3. Impact / upgrade
> * No option and no default changes. Two changes are worth a line in the 4.24
> upgrade guide:
> ** objects are downloaded one by one while the batch is routed, instead of
> all before the first exchange;
> ** ignoreBody no longer downloads the objects. It sends HEAD requests, so the
> request mix changes.
> * 4.18.5, 4.22.2 and 4.23.0 get every object in {{createExchanges}} in the
> same way. This was checked in the source; it was measured on main only.
> * The {{fileName}} path (one object) and the producer {{getObject}} with
> ignoreBody take the same ignoreBody change.
> h3. Related
> * CAMEL-26661: the camel-aws2-s3 consumer with includeBody=false and
> maxMessagesPerPoll above 50 exhausts the HTTP pool. This change fixes it.
> * CAMEL-25558 (commit ed9e3b7cd277): the skip handling that must be kept.
> * CAMEL-16727 added ignoreBody "for metadata only". CAMEL-14929 added the
> stream body with includeBody=false.
> _Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
