allthingssecurity opened a new pull request, #28076: URL: https://github.com/apache/camel/pull/28076
# Description [CAMEL-25760](https://issues.apache.org/jira/browse/CAMEL-25760) The OBS consumer lists the bucket and gets every listed object. An object that no longer exists is skipped (CAMEL-25551); any other failure of a get was rethrown and failed the whole poll, so the other objects of that listing were not consumed either. The marker of a truncated listing moves on before the gets, but a listing that is not truncated sets it back to `null`. So one object whose get always fails (access denied for that object, an Archive or Deep Archive object that is not restored, a disabled SSE-KMS key) meant: 1. a bucket or prefix whose objects fit in one listing (10 with the default `maxMessagesPerPoll`) was never consumed again: every poll listed the same objects and failed on the same one; 2. in a larger bucket, the objects listed together with it were skipped at every pass over the bucket. This change does what camel-aws2-s3 does since CAMEL-25558: a failed get of one listed object is collected, the object is skipped for this poll and reported to the consumer's exception handler with its key, and the other listed objects are consumed. The poll still fails (with the first failure) when none of the listed objects could be got, as for expired credentials or no permission to get objects, so the error handling and the backoff of the scheduled consumer apply. An object deleted after the listing is still skipped at `DEBUG`, and with the `fileName` option a failed get still fails the poll. The behaviour change (a skipped object is reported at every poll that lists it) is in the 4.24 upgrade guide. Tests: - New `OBSConsumerObjectThatCannotBeGotTest` (3 tests) with a fake bucket that lists its keys in order, `maxKeys` at a time with the last key as next marker, and answers the get of a forbidden object with 403. Without the fix `objectThatCannotBeGotDoesNotBlockTheOtherObjects` (a.txt, forbidden b.txt, c.txt, one listing) receives no object, and `objectThatCannotBeGotDoesNotBlockTheObjectsOfItsPage` (`maxMessagesPerPoll=2`, forbidden a.txt listed with b.txt) never receives b.txt. `pollFailsWhenNoListedObjectCanBeGot` passes with and without the fix; it is there because the fix must not swallow that case. - camel-huaweicloud-obs unit tests: 47 run, 0 failures, 18 skipped (functional tests that need OBS credentials) (formatter and import sort run in the same build). - Not run: against a real OBS bucket. The status codes for forbidden and archived objects are from the OBS documentation. How it was found: comparing the sibling object-store consumers (camel-aws2-s3 after CAMEL-25558, camel-huaweicloud-obs, camel-alibaba-oss) on the same listing, get and commit steps. camel-alibaba-oss has the same defect and is fixed in a separate PR. Not changed here: camel-minio (a failed get in `MinioConsumer.processBatch` stops the rest of the batch and fails the poll) and camel-google-storage (`GoogleCloudStorageConsumer.addExchange` rethrows, and the poll fails) have the same kind of defect; they are follow-ups. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-alibaba-oss, including the formatter and import-sort plugins. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
