shashank created CAMEL-25607:
--------------------------------
Summary: camel-mongodb-gridfs - with the default TimeStamp query
strategy (and PersistentTimestamp) the consumer routes every file uploaded
since it started again at each poll, every 500 ms
Key: CAMEL-25607
URL: https://issues.apache.org/jira/browse/CAMEL-25607
Project: Camel
Issue Type: Bug
Components: camel-mongodb-gridfs
Reporter: shashank
Assignee: shashank
{{GridFsConsumer}} with a timestamp query strategy
({{queryStrategy=TimeStamp}}, the default, and {{PersistentTimestamp}}) asks
for the files whose {{uploadDate}} is greater than the date of the newest file
it has routed. Since CAMEL-17121 (3.13, the loop was replaced by a repeated
task) that date is lost between the polls ({{GridFsConsumer}} at main
a7169d7c6ab6):
{code:java}
// run(), lines 128-133: the start date is captured once
Date finalFromDate = fromDate;
task.run(endpoint.getCamelContext(),
() -> processCollection(finalFromDate, usesTimestamp,
persistsTimestamp, usesAttribute, finalPtsCollection,
finalPersistentTimestamp));
// processCollection(Date fromDate, ...), lines 180-183: moves the parameter of
this call only
if (usesTimestamp && file.getUploadDate().compareTo(fromDate) > 0) {
fromDate = file.getUploadDate();
dateModified = true;
}
{code}
Every poll therefore queries {{uploadDate > <date the consumer started>}}
again. A file uploaded after the consumer started is routed by the first poll
that sees it and then again by every following poll, every {{delay}} (500 ms by
default), for as long as the route runs. Before CAMEL-17121 {{fromDate}} was a
local variable of the loop and the file was routed once.
Configuration: the defaults. {{from("mongodb-gridfs:myDb?database=test")}} and
one file created in the bucket are enough; the route receives that file about
twice per second.
With {{PersistentTimestamp}} the effect is the same until the route is
restarted (the stored timestamp is written again at every poll, with the same
value; a restart then reads it and starts after the file). The strategies that
use the file attribute ({{FileAttribute}}, {{TimeStampAndFileAttribute}},
{{PersistentTimestampAndFileAttribute}}) are not affected: their query also
asks for files without the attribute.
The integration test of the module ({{GridFsConsumerIT}}) does not see it: all
files have the same body, and the mock endpoint is asserted as soon as the
expected number of messages has arrived, so a file that is routed again a poll
later is either not seen or looks like the next expected message. CAMEL-24905
(flaky {{GridFsConsumerIT.testAttribute}}, "Expected: <1> but was: <2>") is
possibly related, not reproduced: the test that failed there uses the
{{FileAttribute}} strategy, which is not affected, and the root causes found in
that ticket were the shared mock endpoint and buckets that were not cleaned
between the tests.
h3. Reproduction
New {{GridFsConsumerTimestampTest}} (3 tests, no MongoDB needed): the real
{{GridFsConsumer}} polls a fake bucket (Mockito) that answers the {{uploadDate
> date}} query of each poll from a list of files, the way the server does. All
three fail with the consumer of main:
* {{timestampStrategyRoutesAFileOnce}}: {{three polls saw the file, it must be
routed by the first of them only ==> expected: <[first]> but was: <[first,
first, first, first]>}} (three or four times, depending on the poll that is
running when the test asserts)
* {{timestampStrategyRoutesALaterFileOnce}}: {{expected: <[first, second]> but
was: <[first, first, first, first, second, first, second, first, second]>}}
* {{persistentTimestampStrategyRoutesAFileOnce}}: {{expected: <[first]> but
was: <[first, first, first]>}}
h3. Proposed fix
Keep the date in a field of the consumer, so that a poll starts from the date
the previous poll reached. The field is {{volatile}}: {{run()}} sets the start
value on the thread of the consumer's {{executor}} before it schedules the
polls, and the polls read and move it on the thread of the {{taskExecutor}}.
Nothing else changes.
A stop and start of the consumer is as before: {{doStart}} runs {{run()}}
again, which sets the date to "now" for {{TimeStamp}} and reads the stored
timestamp for {{PersistentTimestamp}}.
Behaviour change, with a 4.24 upgrade-guide entry: a file is routed once.
Not changed, seen in the same method: a failed exchange still moves the
timestamp (and sets the file attribute to {{done}}), an exception thrown by
{{process}} is still swallowed, and an exception of the poll itself (MongoDB
not reachable) still ends the repeated task. These were the same before
CAMEL-17121 or are a different defect, and would be separate tickets.
camel-mongodb-gridfs tests with the fix: 3 run, 0 failures (the module has no
other unit test) (the integration tests need a MongoDB container and were not
run).
How it was found: a small TLA+ model of the polls (upload, query, route, end of
poll). For the code of main TLC reports a violation of "a file is routed at
most once" after two polls (7 states) and of "the timestamp is stored only when
it moved"; with the date kept between the polls both hold, and no file is
skipped. The model is a sketch of the code; the evidence is the test above.
Affected: main (a7169d7c6ab6, tested). Read, not run: camel-4.23.x
(197e70656a41), camel-4.22.x (8dfae2fa437c), camel-4.18.x (c3fd5317a5a0),
camel-4.14.x, camel-3.14.x and the camel-3.13.0 tag have the same lines
({{finalFromDate}} captured by the task, {{processCollection(Date fromDate,
...)}}), so every release since 3.13.0. The camel-3.12.0 tag still has the loop
with a local variable.
Duplicate check (2026-10-13): JIRA text "GridFsConsumer" (3: CAMEL-24286,
CAMEL-24272, CAMEL-17121, none about duplicates), "gridfs consumer" (3),
"gridfs duplicate" (2), "gridfs timestamp" (1), "mongodb-gridfs" (26, none
about the consumer routing a file more than once); GitHub pull requests
"gridfs": only #26787 (CAMEL-24905, test only) touches the consumer tests.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
