Claus Ibsen created CAMEL-27195:
-----------------------------------
Summary: camel-mongodb - change streams consumer: resume after
restart without a bean
Key: CAMEL-27195
URL: https://issues.apache.org/jira/browse/CAMEL-27195
Project: Camel
Issue Type: Improvement
Components: camel-mongodb
Reporter: Claus Ibsen
h3. Background
A {{changeStreams}} consumer resumes after a restart only when
{{changeStreamTokenRepository}} is set. It takes a {{StateRepository<String,
String>}} object, such as a {{FileStateRepository}}. After each exchange, the
resume token is stored under {{<routeId>/<collection>}}, and the next start
resumes after it. Without a repository, the changes made while the route was
stopped are never consumed.
The other option, {{changeStreamToken}}, is a fixed token: it is used on every
start and wins over the repository, so each restart consumes the same changes
again. It is not a way to keep the place.
h3. The problem
The repository can only be given as a bean. That works in a Java or YAML route,
e.g.:
{code}
mongodb:mongo?...&consumerType=changeStreams&changeStreamTokenRepository=#class:org.apache.camel.support.processor.state.FileStateRepository#fileStateRepository('/var/data/tokens.dat')
{code}
but it cannot be offered as a plain, optional option where only endpoint
parameters can be set. The Kamelet {{mongodb-changes-stream-source}} cannot
offer resume after restart because of this: an optional placeholder drops a
parameter only when the value starts with the placeholder, so the
{{#class:...('{{?file}}')}} form is never dropped, and a Kamelet that always
creates the bean forces every user to have a file.
h3. Suggested improvement
Add a simple built-in way to keep the tokens, set from the endpoint URI, for
example one of:
* {{changeStreamTokenFile=<path>}}: the consumer creates and owns a
{{FileStateRepository}} on that file (like {{offsetStorageFileName}} in
camel-debezium);
* {{persistentChangeStreamToken=true}} with an optional {{changeStreamTokenDb}}
/ {{changeStreamTokenCollection}}: store the token in a MongoDB collection,
like {{persistentTailTracking}} does for the tailable consumer.
{{changeStreamTokenRepository}}, when set, would still win.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)