gnodet-bot commented on code in PR #29656:
URL: https://github.com/apache/camel/pull/29656#discussion_r4542539237
##########
components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java:
##########
@@ -239,9 +258,27 @@ protected Deque<Exchange>
createExchanges(Iterator<Result<Item>> minioObjectSumm
}
}
lastListed = minioObjectSummary.objectName();
- } while (minioObjectSummaries.hasNext());
+ // a limit above 0 caps the objects listed by a poll, so do
not read further pages of the listing:
+ // the next poll continues after the last listed object. A
page the server capped below the limit is
+ // truncated, so the listing has not ended when the poll stops
there or reads it to its end
+ if (maxMessagesPerPoll > 0 && messageCounter >=
maxMessagesPerPoll) {
+ if (minioObjectSummaries.hasNext()) {
+ // a truncated page is not the last fetch of the
request, so the listing continues
+ exhaustionKnown = false;
+ } else {
+ exhausted = true;
+ }
+ break;
+ }
Review Comment:
⚡ **Performance note:** When `messageCounter >= maxMessagesPerPoll` and the
listing is not yet exhausted, calling `minioObjectSummaries.hasNext()` triggers
the Minio SDK to fetch the **next page** from the server (the SDK iterator
auto-pages lazily). That page is then discarded — the next poll creates a fresh
`listObjects` iterator starting from the continuation token and re-fetches it.
This means every full non-final poll issues two `ListObjectsV2` requests:
one for the page it actually routes, and one to determine whether the listing
continues. For high-throughput buckets with many small polls this doubles the
listing API calls.
The `camel-aws2-s3` consumer avoids this by reading `isTruncated` directly
from the response object rather than calling `hasNext()` on the lazy iterator.
It may be worth checking whether the Minio SDK exposes an equivalent (e.g.
`ListBucketResult.isTruncated()`) so the continuation check doesn't require a
second network round-trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
