allthingssecurity opened a new pull request, #27794: URL: https://github.com/apache/camel/pull/27794
# Description [CAMEL-25607](https://issues.apache.org/jira/browse/CAMEL-25607) With a timestamp query strategy (`queryStrategy=TimeStamp`, the default, and `PersistentTimestamp`) the `mongodb-gridfs` consumer asks at each poll for the files whose `uploadDate` is greater than the date of the newest file it has routed. Since CAMEL-17121 (the polling loop became a repeated task) that date is lost between the polls: `run()` captures the start date once for the lambda of the task, and `processCollection(Date fromDate, ...)` only moves its own parameter. Every poll queries `uploadDate > <date the consumer started>` again, so a file uploaded after the consumer started is routed by every poll, every `delay` (500 ms by default), for as long as the route runs. `from("mongodb-gridfs:myDb?database=test")` with the default options and one new file in the bucket is enough: the route receives the file about twice per second. With `PersistentTimestamp` the same happens until the route is restarted, and the stored timestamp is written again at every poll. The strategies that use the file attribute are not affected, their query also filters on the attribute. This change keeps the date in a field of the consumer, so a poll starts from the date the previous poll reached. Two threads use the field, so it is `volatile`: `run()` sets the start value on the thread of the consumer's `executor` before it schedules the polls, and the polls read and move it on the thread of the `taskExecutor`. Nothing else in the consumer changes. A stop and start of the consumer is as before: `doStart` runs `run()` again, which sets the date to "now" for `TimeStamp` and reads the stored timestamp for `PersistentTimestamp`. Files uploaded while a `TimeStamp` consumer was stopped are still not routed. What changes for users (4.24 upgrade guide entry): a file is routed once. Not changed, although it is in the same method: a failed exchange still moves the timestamp and still sets the file attribute to `done`, an exception thrown by `process` is still swallowed, and an exception of the poll itself (MongoDB not reachable) still ends the repeated task. The first two were the same before CAMEL-17121; I would report them separately. Why the existing integration test did not notice: `GridFsConsumerIT` uploads files with identical bodies and asserts the mock endpoint as soon as the expected number of messages has arrived, so a file that is routed again a poll later is either not seen or looks like the next expected message. CAMEL-24905 (flaky `GridFsConsumerIT.testAttribute`, `Expected: <1> but was: <2>`) is possibly related, not reproduced: the test that failed there uses the `FileAttribute` strategy, which this defect does not affect, and the root causes found in that ticket were the shared mock endpoint and buckets that were not cleaned between the tests. Tests: - New `GridFsConsumerTimestampTest` (3 tests, no MongoDB needed; adds `mockito-junit-jupiter` as a test dependency of the module). The real consumer polls a fake bucket that answers the `uploadDate > date` query of each poll from a list of files, the way the server does. One file seen by several polls is routed once (`TimeStamp`); a second file uploaded later is routed once as well, after the first; with `PersistentTimestamp` the file is routed once and the timestamp is stored once. - Without the change (main's `GridFsConsumer`, same test) all three fail: `expected: <[first]> but was: <[first, first, first]>` (the file once per poll; three or four times, depending on the poll that is running when the test asserts) and `expected: <[first, second]> but was: <[first, first, first, first, second, first, second, first, second]>`. - With the change, camel-mongodb-gridfs tests pass: 3 run, 0 failures (the three new tests; the module has no other unit test). The integration tests (`*IT`) need a MongoDB container and were not run. How it was found: a small TLA+ model of the polls (upload, query, route, end of poll) showed a file routed twice after two polls for the code of main. The model is only a sketch of the code; the evidence is the test above. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-mongodb-gridfs, including the formatter and import-sort plugins. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
