Federico Mariani created CAMEL-26662:
----------------------------------------
Summary: camel-aws2-s3 - consumer gets every object of a poll
before routing the first, and ignoreBody=true still downloads them
Key: CAMEL-26662
URL: https://issues.apache.org/jira/browse/CAMEL-26662
Project: Camel
Issue Type: Improvement
Components: camel-aws2-s3
Affects Versions: 4.18.5, 4.23.0, 4.22.2
Reporter: Federico Mariani
h3. Problem
* *Every object of a poll is got before the first one is routed.*
{{createExchanges}} calls GetObject for each listed key
({{components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java}}
line 298 on main, commit 7978db3d42dd). With the default {{includeBody=true}},
it also reads each body into a byte[] (line 569). Only then does {{poll()}}
call {{processBatch}} (line 207).
** The heap of a poll holds every body of the batch (default
{{maxMessagesPerPoll=10}}), and the first exchange waits for the last download.
** With {{includeBody=false}}, every stream of the poll stays open, each
holding a pooled connection, while the earlier exchanges are routed.
** Since SDK 2.32.32, a {{ResponseInputStream}} that is not read within 60 s is
aborted. So a later exchange of the poll fails once the earlier ones together
took longer than 60 s.
** With {{maxMessagesPerPoll}} above 50, the pool runs out: that is CAMEL-26661.
* *ignoreBody=true still downloads every object.*
** The consumer does a GetObject and closes the stream without reading it
(lines 606-607). The producer {{getObject}} with ignoreBody does the same
({{AWS2S3Producer.java}} lines 685-686 and 732-733).
** With the Apache HTTP client Camel uses by default, {{close()}} reads the
rest of the body so that the connection can be reused (the SDK javadoc of
{{ResponseInputStream}}; httpclient5 5.6.4
{{ResponseEntityProxy.streamClosed}}). So the whole object is transferred
anyway.
* The docs say otherwise:
{{components/camel-aws/camel-aws2-s3/src/main/docs/aws2-s3-consumer-examples.adoc}}
line 228 reads "Use {{ignoreBody: true}} to get only the metadata."
h3. How to reproduce
An in-process fake S3 (JDK {{HttpServer}}) is used with the client Camel builds
(Apache5, httpclient5 5.6.4). Measured on main:
||Case||Observed||
|10 objects, maxMessagesPerPoll=10, includeBody=true|all 10 GetObject answered
completely, then the 10 exchanges routed|
|10 objects of 8 MiB, includeBody=false|all 10 GetObject sent before the first
exchange is routed; at that point 9 of the 10 responses are still being sent|
|1 object of 64 MiB, ignoreBody=true (consumer, and producer
getObject)|67,108,864 of 67,108,864 bytes sent, body null|
|Control: the same object, SDK ResponseInputStream.abort()|0.6 to 1.8 MB sent
before the connection is closed|
|2 objects, includeBody=false, the route takes 61 s on the first|the second
exchange fails with IOException: Attempted read on closed stream|
h3. Proposed change
* Move the GetObject into {{processBatch}}: list in {{poll()}}, then get each
listed object right before its exchange is routed. At most the objects of the
exchanges in flight are then in memory or open.
* Keep what CAMEL-25558 added:
** an object that no longer exists is skipped;
** an object that cannot be got is skipped and given to the exception handler;
** the poll fails only when none of the objects could be got.
* {{CamelBatchSize}} and {{CamelBatchComplete}} are known only after the gets,
so it must be decided what they count. This needs care.
* For ignoreBody, use HeadObject instead of GetObject. HeadObject returns every
field the consumer maps to headers (ETag, VersionId, ContentType,
ContentLength, ContentEncoding, ContentDisposition, CacheControl,
ServerSideEncryption, Expiration, ReplicationStatus, StorageClass, Metadata,
LastModified; S3 service model), and it needs the same {{s3:GetObject}}
permission.
* The alternative is {{abort()}} instead of {{close()}}. It also stops the
transfer (measured above), but it closes the connection, so the next request
opens a new one.
* Fix the example at line 228 of {{aws2-s3-consumer-examples.adoc}} along with
the change.
* This also fixes CAMEL-26661 (includeBody=false with maxMessagesPerPoll above
50), and the 60 s abort of the later streams of a poll in general.
h3. Impact / upgrade
* No option and no default changes. Two changes are worth a line in the 4.24
upgrade guide:
** objects are downloaded one by one while the batch is routed, instead of all
before the first exchange;
** ignoreBody no longer downloads the objects. It sends HEAD requests, so the
request mix changes.
* 4.18.5, 4.22.2 and 4.23.0 get every object in {{createExchanges}} in the same
way. This was checked in the source; it was measured on main only.
* The {{fileName}} path (one object) and the producer {{getObject}} with
ignoreBody take the same ignoreBody change.
h3. Related
* CAMEL-26661: the camel-aws2-s3 consumer with includeBody=false and
maxMessagesPerPoll above 50 exhausts the HTTP pool. This change fixes it.
* CAMEL-25558 (commit ed9e3b7cd277): the skip handling that must be kept.
* CAMEL-16727 added ignoreBody "for metadata only". CAMEL-14929 added the
stream body with includeBody=false.
_Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
