allthingssecurity opened a new pull request, #27673: URL: https://github.com/apache/camel/pull/27673
# Description [CAMEL-25522](https://issues.apache.org/jira/browse/CAMEL-25522) The ibm-cos consumer adds each listed key to the endpoint's `inProgressRepository` (CAMEL-24491) and removes it in the on completion that `createExchange` adds, which also closes the object's content stream (`includeBody` and `autocloseBody`, both on by default). Two paths miss it: - With `fileName` set, every poll gets the object without looking at the repository. With an asynchronous route (`seda`, which hands the on completion over, an async producer, `threads()`), the next poll gets the object again while its first exchange is still in flight and routes it a second time (`deleteAfterRead`, the default, only deletes it when the first exchange is done). - When the consumer stops while it processes a batch, the exchanges it does not process are dropped (their objects stay in the bucket, by design), but their on completions never run: their keys stay in the repository (10000 entries, only cleared when the endpoint stops), so after a route stop and start these objects are skipped as in progress until the CamelContext restarts; and the content streams, opened for every listed object before the batch is routed, are never closed, so each such stop keeps up to `maxMessagesPerPoll - 1` connections of the client's pool. The fix: with `fileName`, the key is added before the get (when a repository is configured, as in the listing), the poll is skipped while it is in progress, and the key is removed when the get fails. After the batch loop, the keys of the exchanges that were not processed are removed and their body streams closed. The same two gaps exist in camel-aws2-s3, whose consumer this one follows; that fix is a separate PR (CAMEL-25521). No new option; 4.23 upgrade-guide note. Tests (`IBMCOSConsumerInProgressTest`, with a small in-process fake of the S3 calls the consumer makes, JDK `HttpServer`, and an IBM COS client with static HMAC credentials pointing at it): - `objectInFlightIsNotConsumedAgainWithFileName`: without the change, `a.txt was consumed again by a later poll while its first exchange was still in flight ==> expected: <1> but was: <2>`. - `objectWithFileNameIsConsumedOnceItExists`: the get fails (`NoSuchKey`) until the object is created; the failed get must not leave the key in progress. It passes without the change too (nothing is added to the repository on main); it fails when the removal on a failed get is left out. - `objectsOfAStoppedBatchAreConsumedAfterRestart`: the first exchange of a batch of three stops the route and waits until the consumer no longer processes the batch; after a restart the other two objects must be consumed. Without the change: `mock://result Received message count. Expected: <2> but was: <0>`. - `notProcessedExchangesCloseTheirContent`: `processBatch` of a consumer that is not started releases the key and closes the body stream. Without the change: `b.txt is still in progress`; without only the close, the close assertion fails. - With the change, the camel-ibm-cos unit tests pass (6). The ITs need an IBM Cloud account; I did not run them. Found with TLA+ models of the consumer (shared with the aws2-s3 fix): with `fileName` and an asynchronous route, "with deleteAfterRead every object is routed once" is violated in 5 steps; for the listing, "every object is routed by one of the polls" is violated in 12 steps. I then reproduced both with the real consumer; the open streams came from reading the code around it. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-ibm-cos, including the formatter and import-sort plugins; the build did not change any generated file. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
