allthingssecurity opened a new pull request, #27670: URL: https://github.com/apache/camel/pull/27670
# Description [CAMEL-25519](https://issues.apache.org/jira/browse/CAMEL-25519) The google-storage consumer hands each exchange to the route with an empty callback and does not wait; the object is deleted (`deleteAfterRead`, the default) or moved by the exchange's on completion. With an asynchronous route (`seda`, which hands the on completion over, kafka or another async producer, `threads()`), the next poll lists and downloads the same object again and routes it a second time. If the first exchange deletes the object after the next poll listed it, the download of that poll fails with a 404 and the whole poll ends with an exception. The same happens with `objectName` set. This is the defect fixed for aws2-s3 in CAMEL-17110 (and for minio in CAMEL-25512, #27645, merged). The consumer now keeps the names of the objects whose exchanges are in progress: a poll skips them (listing and `objectName` mode), and a name is released when its exchange completes or fails (after the delete/move), or when the poll does not process it (download failure, consumer stopping). A listed object that no longer exists when its content is downloaded is skipped instead of failing the poll; the exchanges are created before the batch is routed, so `CamelBatchSize` only counts the routed objects. No new option, as in the minio fix: a plain set in the consumer, not cleared on route stop (in-flight names are released when their exchanges complete). With `deleteAfterRead=false` an object is now consumed again only after its previous exchange completed. 4.23 upgrade-guide note for both behaviour changes. Tests (module `LocalStorageHelper` in-memory storage, no container): - `GoogleCloudStorageConsumerInProgressTest`: the seda route holds the exchange until the consumer has completed a second poll (counted with a `pollStrategy`); listing and `objectName` mode. Without the change: `[a.txt was consumed again by a later poll while its first exchange was still in flight] expected: 1 but was: 2` (both modes). - `GoogleCloudStorageConsumerDeletedObjectTest`: the storage deletes `b.txt` right after the listing; `a.txt` and `c.txt` are routed by the same poll with `CamelBatchSize` 2. Without the change the poll fails (`[the poll failed because b.txt was deleted between the listing and the download]`). - `GoogleCloudStorageConsumerInProgressFailureTest`: a failed object is consumed again by a later poll; `notProcessedObjectIsConsumedAgain`: an object whose exchange a stopping consumer did not process is not left in progress. Both pass without the change too (there is no in-progress set on main); they guard the releases on failure and on stop. - With the change, the camel-google-storage unit tests pass (58). Found with a TLA+ model of the poll and the asynchronous completion: "with deleteAfterRead every object is routed once" is violated in 10 steps (poll, download, dispatch, poll again, download, dispatch again). I then reproduced it with the real consumer. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-google-storage, including the formatter and import-sort plugins; the build did not change any generated file. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
