allthingssecurity opened a new pull request, #27672: URL: https://github.com/apache/camel/pull/27672
# Description [CAMEL-25521](https://issues.apache.org/jira/browse/CAMEL-25521) Since CAMEL-17110 the aws2-s3 consumer adds each listed key to the endpoint's `inProgressRepository` and removes it in the exchange's on completion. Two paths miss it: - With `fileName` set, every poll gets the object without looking at the repository. With an asynchronous route (`seda`, which hands the on completion over, an async producer, `threads()`), the next poll gets the object again while its first exchange is still in flight and routes it a second time (`deleteAfterRead`, the default, only deletes it when the first exchange is done). - When the consumer stops while it processes a batch, the exchanges it does not process are dropped (their objects stay in the bucket, by design), but their keys stay in the repository. The default memory repository holds 10000 keys and is only cleared when the endpoint stops, so after a route stop and start these objects are skipped as in progress and never consumed until the CamelContext restarts. The fix: with `fileName`, the key is added before the get, the poll is skipped while it is in progress, and the key is removed when the get fails (and, as before, when the exchange is done); the get itself moved to a small private method. After the batch loop, the keys of the exchanges that were not processed are removed, and their bodies are closed when they are the object streams (`includeBody=false`; their on completion, which closes them with `autocloseBody`, never runs; the default `includeBody=true` already closes each stream when the exchange is created). This is what the listing branch and the camel-minio fix (CAMEL-25512) do; no new option. 4.23 upgrade-guide note. Tests (`AWS2S3ConsumerInProgressTest` with a small in-process fake of the S3 calls the consumer makes, JDK `HttpServer`, no container): - `objectInFlightIsNotConsumedAgainWithFileName`: the seda route holds the exchange until the consumer has completed a second poll (counted with a `pollStrategy`). Without the change: `a.txt was consumed again by a later poll while its first exchange was still in flight ==> expected: <1> but was: <2>`. - `objectWithFileNameIsConsumedOnceItExists`: the get fails (`NoSuchKey`) until the object is created; the failed get must not leave the key in progress. It passes without the change too (nothing is added to the repository on main); it fails when the removal on a failed get is left out. - `objectsOfAStoppedBatchAreConsumedAfterRestart`: the first exchange of a batch of three stops the route and waits until the consumer no longer processes the batch; after a restart the other two objects must be consumed. Without the change: `mock://result Received message count. Expected: <2> but was: <0>`. - `notProcessedExchangesReleaseTheirObjects`: `processBatch` of a consumer that is not started removes the key and closes the object stream (`includeBody=false`). Without the change: `b.txt is still in progress`; without only the close, the close assertion fails. - With the change, the camel-aws2-s3 unit tests pass (83). The ITs need LocalStack; I did not run them. Found with TLA+ models of the consumer: with `fileName` and an asynchronous route, "with deleteAfterRead every object is routed once" is violated in 5 steps; for the listing, "every object is routed by one of the polls" is violated in 12 steps (a stop drops the batch, then every later poll skips its objects). I then reproduced both with the real consumer. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-aws2-s3, including the formatter and import-sort plugins; the build did not change any generated file. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
