Federico Mariani created CAMEL-26653:
----------------------------------------
Summary: camel-minio - maxMessagesPerPoll only sets the listing
page size: one poll lists, stats and routes the whole bucket
Key: CAMEL-26653
URL: https://issues.apache.org/jira/browse/CAMEL-26653
Project: Camel
Issue Type: Bug
Components: camel-minio
Affects Versions: 4.14.9, 4.18.5, 4.23.0, 4.22.2
Reporter: Federico Mariani
h3. Problem
The {{maxMessagesPerPoll}} option (default 10) is documented as "the maximum
number of messages as a limit to poll at each polling ... Use 0 or a negative
number to set it as unlimited". The minio consumer uses it only as the page
size of the listing, so one poll lists every page of the bucket, gets the
metadata of every object, and routes all of them.
Code on main (commit 7978db3d42dd), in
{{components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java}}:
* Lines 156-158: {{listObjectRequest.maxKeys(maxMessagesPerPoll)}}.
* Lines 226-242: {{createExchanges}} loops with {{do ... while
(minioObjectSummaries.hasNext())}} and has no count check.
* In minio-java 9.0.3, the iterator returned by {{listObjects}} is lazy: when
a page is truncated, {{hasNext()}} and {{next()}} fetch the next one
({{MinioAsyncClient.ObjectIterator}} calls {{populate()}} again).
* So the loop reads every page.
* Line 567: each listed object gets a {{statObject}} before anything is routed
({{MinioEndpoint.getObjectStat}}, {{MinioEndpoint.java}} line 246).
* Line 346: {{processBatch}} routes every exchange and has no limit either.
With the defaults ({{includeBody=true}}, {{deleteAfterRead=true}}), each object
is downloaded and then deleted.
* Lines 191-195: the continuation check compares the number of objects listed
by the whole poll to {{maxMessagesPerPoll}}. For any bucket with more objects
than that, the next poll starts after the last object of the bucket and gets an
empty listing.
The same behaviour is in camel-4.14.9, 4.18.5, 4.22.2 and 4.23.0 (checked in
the source, which is structured slightly differently there).
>From the code, for 10k objects with the default of 10: about 1000 list
>requests and 10k stat requests happen before the first exchange is routed, and
>that one poll then routes all 10k objects. This figure is arithmetic, not
>measured at that size.
h3. How to reproduce
Measured on main with a JUnit test:
* A real {{MinioClient}} against an in-process fake S3 endpoint: the module's
{{FakeS3Server}} test helper, extended with request counters. It pages
ListObjectsV2 by max-keys and continuation-token.
* 30 objects, {{maxMessagesPerPoll=10}}, {{startScheduler=false}}, polls called
by the test.
||deleteAfterRead||poll||list requests||stat (HEAD)||get||routed||
|true|1|3|30|30|30|
|true|2|1, start-after=o030.txt, empty|0|0|0|
|false|1|3|30|30|30|
|false|2|1, empty|0|0|0|
|false|3|3|30|30|30|
With the documented limit, a poll would make 1 list request and route at most
10 objects.
h3. Proposed change
* When {{maxMessagesPerPoll}} is above 0, stop reading the listing once that
many objects were listed. That means one page per poll, as camel-aws2-s3 does:
{{AWS2S3Consumer}} makes one ListObjectsV2 call with {{maxKeys}} and keeps the
continuation token for the next poll.
* Base the continuation on what this poll listed: continue after the last
listed object when the page was full, and start over when the listing has
ended. This also removes the empty listing every other poll.
* Keep {{maxMessagesPerPoll}} 0 or below as today (all pages).
* Risk: some listed objects are not routed. These are folders (when
{{includeFolders=false}}), objects still in flight (CAMEL-25512) and objects
that cannot be got (CAMEL-25770). So a poll can route fewer objects than the
limit, and the next poll continues after them, as with aws2-s3.
h3. Impact / upgrade
* Filed as a Bug: the documented option is not applied to routing, which goes
beyond a performance issue.
* This is a behaviour change. With the default {{maxMessagesPerPoll=10}}, a
poll routes at most 10 objects instead of the whole bucket, so draining a large
bucket takes more polls. Setting {{maxMessagesPerPoll=0}} keeps the old
behaviour.
* It needs an entry in the 4.24 upgrade guide.
* An endpoint with {{objectName}} set gets one object per poll and is not
affected.
* The component has been deprecated since 4.23 (CAMEL-24750). The change is
small and the option is documented, so it still seems worth fixing.
h3. Related
* CAMEL-17100: "Minio Consumer is really slow on listing initially if you have
a lot of files". Its fix (7de327d91eb6) moved the download of each object from
the listing into {{processBatch}}. The listing over all pages and the stat per
object stayed.
* Commit 62fd4017653d introduced the continuation check that compares the count
of the whole poll.
* The loop without a count check has been there since the component was added
(67bad5a5978a, CAMEL-13934).
* CAMEL-25512 and CAMEL-25770: recent fixes in the same loop.
_Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
