This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new b4c66b742 chore: nats-source can use a durable JetStream consumer that
carries on after a restart (#3170) (#3236)
b4c66b742 is described below
commit b4c66b7427c1c3f32dba49d0c9011418044046b6
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:38:54 2026 +0100
chore: nats-source can use a durable JetStream consumer that carries on
after a restart (#3170) (#3236)
nats-source gets three optional JetStream options, mapped with optional
placeholders, so that with them unset the endpoint is the same as before:
- durableName: a durable JetStream consumer. After a restart the source
carries on with the first message that was not acknowledged, instead of
reading the stream from the beginning, and copies of the integration with
the same durableName share the messages. camel-nats names the consumer
on the server <durableName>-durable. The pattern allows letters, digits,
'-' and '_'; jnats refuses '.', '*', '>', '/' and '\'.
- ackPolicy (Explicit, All, None): how the messages are acknowledged, and so
whether a message that failed is delivered again.
- maxDeliver: how many times at most a message is delivered.
requestTimeout is not added to nats-sink: the sink is InOnly, so it never
sends a request and waits for a reply. camel-nats uses requestTimeout only
as the wait for the JetStream publish acknowledgement with jetstreamAsync,
where jnats cancels the publish after its own 2 second request timeout
anyway, so the option can only shorten the wait, not lengthen it.
The descriptions of the existing JetStream options are corrected after
running them: nats-sink does not use jetstreamName (the message goes to the
stream that takes in the subject, whatever its name), nats-source does not
use jetstreamAsync, a stream also stores what the sink publishes without
jetstreamEnabled, and on a server without JetStream jetstreamEnabled is
ignored by the sink. The partial pages get a durable consumer section, an
ackPolicy table, a sink JetStream section and examples.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../ROOT/partials/nats-sink-description.adoc | 32 +++++++++++++
.../ROOT/partials/nats-source-description.adoc | 56 ++++++++++++++++++++--
kamelets/nats-sink.kamelet.yaml | 8 ++--
kamelets/nats-source.kamelet.yaml | 25 ++++++++--
.../main/resources/META-INF/kamelets-index.json | 4 +-
.../main/resources/kamelets/nats-sink.kamelet.yaml | 8 ++--
.../resources/kamelets/nats-source.kamelet.yaml | 25 ++++++++--
7 files changed, 139 insertions(+), 19 deletions(-)
diff --git a/docs/modules/ROOT/partials/nats-sink-description.adoc
b/docs/modules/ROOT/partials/nats-sink-description.adoc
index b3a0b6c0d..efa6ae19c 100644
--- a/docs/modules/ROOT/partials/nats-sink-description.adoc
+++ b/docs/modules/ROOT/partials/nats-sink-description.adoc
@@ -72,3 +72,35 @@ That trust store then replaces the one of Java for every
NATS connection of the
secure: true
credentialsFilePath: "file:/etc/nats/app-user.creds"
----
+
+=== JetStream
+
+A plain NATS subject stores nothing: only the receivers subscribed at that
moment get the message. A JetStream stream
+stores the messages of the subjects it takes in. With `jetstreamEnabled:
true`, the Kamelet waits until that stream
+has stored each message, and sending fails with `503 No Responders Available
For Request` when no stream takes in the
+subject in `topic`. The stream must exist already: create it with
`nats-source` and `jetstreamEnabled`, or with the
+`nats` command line tool:
+
+[source,bash]
+----
+nats stream add ORDERS --subjects "orders.created" --defaults
+----
+
+The sink does not use `jetstreamName`: the message goes to the stream that
takes in the subject, whatever its name.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "hello"
+ period: 5000
+ steps:
+ - to:
+ uri: "kamelet:nats-sink"
+ parameters:
+ servers: "nats://localhost:4222"
+ topic: "orders.created"
+ jetstreamEnabled: true
+----
diff --git a/docs/modules/ROOT/partials/nats-source-description.adoc
b/docs/modules/ROOT/partials/nats-source-description.adoc
index 45680c6bd..905d3f834 100644
--- a/docs/modules/ROOT/partials/nats-source-description.adoc
+++ b/docs/modules/ROOT/partials/nats-source-description.adoc
@@ -32,8 +32,8 @@ An address that starts with `tls://` also connects with TLS,
but it uses that tr
Each copy of the integration receives every message of the subject. To share
the messages instead, so that each
message is processed by only one copy, give every copy the same `queueName`.
NATS then delivers each message to one
-member of the queue group. This works only without JetStream: with
`jetstreamEnabled`, each copy still receives every
-message of the stream.
+member of the queue group. This works only without JetStream: with
`jetstreamEnabled`, give every copy the same
+`durableName` instead (see below).
=== Output format
@@ -43,7 +43,38 @@ become headers too.
=== JetStream
With `jetstreamEnabled` set to true, the Kamelet reads from the JetStream
stream `jetstreamName`, and creates it for
-the subject in `topic` when it does not exist. Each time it starts, it reads
the stream from the beginning.
+the subject in `topic` when it does not exist. Without `durableName`, each
time it starts it reads the stream from the
+beginning, so the messages it already read are received again.
+
+=== Durable consumer: carry on after a restart
+
+With `durableName`, the server keeps a durable consumer for the Kamelet, and
remembers which messages it acknowledged.
+After a restart, the Kamelet carries on with the first message that was not
acknowledged: no message is lost and none
+is received again. A new durable consumer starts at the beginning of the
stream.
+
+* A message is acknowledged once the integration has processed it. A message
that fails is delivered again 5 seconds
+ later. Set `maxDeliver` to stop after that many deliveries; the message then
stays in the stream.
+* When the integration stops abruptly, the messages it had not finished come
back after 30 seconds, so the message in
+ progress can be received twice.
+* Copies of the integration with the same `durableName` share the messages:
each message goes to only one of them.
+* `ackPolicy` says how the messages are acknowledged:
++
+[width="100%",cols="1,3",options="header"]
+|===
+| `ackPolicy` | A message that fails
+| `Explicit` (when not set) | is delivered again.
+| `All` | is delivered again, until a later message succeeds: acknowledging a
message also acknowledges the earlier ones.
+| `None` | is not delivered again: JetStream counts a message as done when it
sends it.
+|===
+
+On the server, the consumer is named after `durableName` with `-durable`
added. It stays there when the integration is
+removed, and its `ackPolicy` and `maxDeliver` cannot be changed afterwards:
the integration then logs
+`Existing consumer cannot be modified` and receives nothing. Delete it with
the `nats` command line tool first:
+
+[source,bash]
+----
+nats consumer rm ORDERS order-processor-durable
+----
=== Usage example
@@ -77,3 +108,22 @@ the subject in `topic` when it does not exist. Each time it
starts, it reads the
- to:
uri: "kamelet:log-sink"
----
+
+=== Example with a Durable JetStream Consumer
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:nats-source"
+ parameters:
+ servers: "nats://localhost:4222"
+ topic: "orders.created"
+ jetstreamEnabled: true
+ jetstreamName: "ORDERS"
+ durableName: "order-processor"
+ maxDeliver: 5
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
diff --git a/kamelets/nats-sink.kamelet.yaml b/kamelets/nats-sink.kamelet.yaml
index e7684b3e0..c56c5c6ec 100644
--- a/kamelets/nats-sink.kamelet.yaml
+++ b/kamelets/nats-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the message is unchanged.
- Good to know: a plain NATS subject stores nothing: only the receivers
subscribed at that moment get the message. With `jetstreamEnabled`, the message
is stored in the stream `jetstreamName`, which must exist already, for example
created by `nats-source`; otherwise sending fails. It connects with TLS when
`secure` is true. The certificate of the server is checked against the
certificate authorities that Java trusts. To trust a private one with Camel
JBang or Camel Main, set the appl [...]
+ Good to know: a plain NATS subject stores nothing: only the receivers
subscribed at that moment get the message, unless a JetStream stream takes in
the subject. With `jetstreamEnabled`, the sink waits until the stream that
takes in the subject has stored the message, and sending fails with `503 No
Responders Available For Request` when no stream takes it in. The stream must
exist already, for example created by `nats-source`. On a server without
JetStream, `jetstreamEnabled` is ign [...]
required:
- topic
- servers
@@ -70,17 +70,17 @@ spec:
pattern: "^(file|classpath|http|https):.+"
jetstreamEnabled:
title: "JetStream Enabled"
- description: "If true, each message is published to the JetStream
stream `jetstreamName`, which stores it, instead of to plain NATS. The stream
must exist and take in the subject in `topic`."
+ description: "If true, each message is published to JetStream, and the
sink waits until the stream that takes in the subject in `topic` has stored it.
Sending fails when no stream takes in the subject, or when the server does not
answer within a few seconds."
type: boolean
default: false
jetstreamName:
title: "JetStream Stream Name"
- description: "The name of the JetStream stream that stores the
messages. Only used when `jetstreamEnabled` is true."
+ description: "Not used by the sink: the message is stored by the
stream that takes in the subject in `topic`, whatever its name."
example: "ORDERS"
type: string
jetstreamAsync:
title: "JetStream Async"
- description: "If true, JetStream calls are made without blocking a
thread while waiting for the server to answer. Only used when
`jetstreamEnabled` is true."
+ description: "How the sink waits for JetStream to confirm that it
stored a message. Either way it waits before it sends the next message. When
the server does not answer, sending fails after about 2 seconds when false, and
after 2 to 8 seconds when true. Only used when `jetstreamEnabled` is true."
type: boolean
default: true
dependencies:
diff --git a/kamelets/nats-source.kamelet.yaml
b/kamelets/nats-source.kamelet.yaml
index ba21f90b1..d58427c23 100644
--- a/kamelets/nats-source.kamelet.yaml
+++ b/kamelets/nats-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the payload, as bytes, unchanged. The
`CamelNatsSubject` header is the subject, and the NATS headers of the message
become headers too.
- Good to know: without JetStream, it receives only the messages published
while it is connected. With `jetstreamEnabled`, it creates the stream
`jetstreamName` for the subject when it does not exist, and each time it starts
it reads the stream from the beginning, so messages already read are received
again. When several copies of the integration run, each copy receives every
message, unless they share a `queueName`: then each message goes to only one of
them. This works only without [...]
+ Good to know: without JetStream, it receives only the messages published
while it is connected. With `jetstreamEnabled`, it creates the stream
`jetstreamName` for the subject when it does not exist. Each time it starts it
reads the stream from the beginning, so messages already read are received
again, unless `durableName` is set: then it carries on after the last message
it acknowledged. A message is acknowledged once it has been processed. A
message that fails is delivered again [...]
required:
- topic
- servers
@@ -58,7 +58,7 @@ spec:
- urn:camel:group:credentials
queueName:
title: "Queue Group"
- description: "The name of a NATS queue group to join. The subscribers
of `topic` that use the same queue group share its messages: each message goes
to only one of them. Set the same name on every copy of the integration when
you run several of them, so that each message is processed once; without it,
each copy receives every message. Only used without JetStream: with
`jetstreamEnabled`, each copy still receives every message of the stream."
+ description: "The name of a NATS queue group to join. The subscribers
of `topic` that use the same queue group share its messages: each message goes
to only one of them. Set the same name on every copy of the integration when
you run several of them, so that each message is processed once; without it,
each copy receives every message. Only used without JetStream: with
`jetstreamEnabled`, each copy still receives every message of the stream,
unless the copies share a `durableName`."
example: "order-workers"
type: string
pattern: "^\\S+$"
@@ -84,9 +84,25 @@ spec:
type: string
jetstreamAsync:
title: "JetStream Async"
- description: "If true, JetStream calls are made without blocking a
thread while waiting for the server to answer. Only used when
`jetstreamEnabled` is true."
+ description: "Not used by the source, which receives the messages the
same way whether it is true or false."
type: boolean
default: true
+ durableName:
+ title: "Durable Consumer Name"
+ description: "The name of a durable JetStream consumer: the server
keeps it, and the position of the integration in the stream, when the
integration stops. After a restart, it carries on with the first message that
was not acknowledged, instead of reading the stream from the beginning. Copies
of the integration with the same `durableName` share the messages: each message
goes to only one of them. A new durable consumer starts at the beginning of the
stream. On the server, the con [...]
+ example: "order-processor"
+ type: string
+ pattern: "^[A-Za-z0-9_-]+$"
+ ackPolicy:
+ title: "Acknowledge Policy"
+ description: "How the messages are acknowledged to JetStream, so that
it does not deliver them again. `Explicit`: each message on its own, once it
has been processed; a message that fails is delivered again. `All`: the same,
but acknowledging a message also acknowledges all the earlier ones, so a
message that failed is not delivered again once a later message succeeds.
`None`: JetStream counts a message as done when it sends it, so a message that
fails is not delivered again. Whe [...]
+ type: string
+ enum: ["Explicit", "All", "None"]
+ maxDeliver:
+ title: "Maximum Deliveries"
+ description: "How many times at most a message is delivered, the first
time included. A message that fails is delivered again 5 seconds later, until
it has been delivered this many times; it then stays in the stream but is not
delivered again. When not set, a message that fails is delivered again with no
end. Only used when `jetstreamEnabled` is true."
+ example: 5
+ type: integer
types:
out:
mediaType: application/octet-stream
@@ -117,6 +133,9 @@ spec:
secure: "{{?secure}}"
credentialsFilePath: "{{?credentialsFilePath}}"
queueName: "{{?queueName}}"
+ durableName: "{{?durableName}}"
+ ackPolicy: "{{?ackPolicy}}"
+ maxDeliver: "{{?maxDeliver}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 6600e8c3c..dfda5211d 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1665,7 +1665,7 @@
"supportLevel" : "Stable",
"title" : "NATS Sink",
"summary" : "Publishes each message to a subject on a NATS server, or to a
JetStream stream that stores the messages.",
- "description" : "Publishes each message to a subject on a NATS server, or
to a JetStream stream that stores the messages.\n\nUse it to send events or
commands to services that listen on NATS, such as microservices or edge
devices. Use `nats-source` to receive them.\n\nInput: the body is the payload,
as text or bytes. A Map is sent as its Java text form, such as `{a=1, b=x}`, so
turn it into JSON text first, for example with `json-serialize-action`. Headers
of the message, such as `Co [...]
+ "description" : "Publishes each message to a subject on a NATS server, or
to a JetStream stream that stores the messages.\n\nUse it to send events or
commands to services that listen on NATS, such as microservices or edge
devices. Use `nats-source` to receive them.\n\nInput: the body is the payload,
as text or bytes. A Map is sent as its Java text form, such as `{a=1, b=x}`, so
turn it into JSON text first, for example with `json-serialize-action`. Headers
of the message, such as `Co [...]
"group" : "NATS",
"namespace" : "Messaging",
"provider" : "Apache Software Foundation"
@@ -1675,7 +1675,7 @@
"supportLevel" : "Stable",
"title" : "NATS Source",
"summary" : "Receives each message published to a subject on a NATS
server, or stored in a JetStream stream.",
- "description" : "Receives each message published to a subject on a NATS
server, or stored in a JetStream stream.\n\nUse it to react to events or
commands that services publish on NATS. Use `nats-sink` to publish
them.\n\nOutput: the body is the payload, as bytes, unchanged. The
`CamelNatsSubject` header is the subject, and the NATS headers of the message
become headers too.\n\nGood to know: without JetStream, it receives only the
messages published while it is connected. With `jetstr [...]
+ "description" : "Receives each message published to a subject on a NATS
server, or stored in a JetStream stream.\n\nUse it to react to events or
commands that services publish on NATS. Use `nats-sink` to publish
them.\n\nOutput: the body is the payload, as bytes, unchanged. The
`CamelNatsSubject` header is the subject, and the NATS headers of the message
become headers too.\n\nGood to know: without JetStream, it receives only the
messages published while it is connected. With `jetstr [...]
"group" : "NATS",
"namespace" : "Messaging",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/nats-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/nats-sink.kamelet.yaml
index e7684b3e0..c56c5c6ec 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/nats-sink.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/nats-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the message is unchanged.
- Good to know: a plain NATS subject stores nothing: only the receivers
subscribed at that moment get the message. With `jetstreamEnabled`, the message
is stored in the stream `jetstreamName`, which must exist already, for example
created by `nats-source`; otherwise sending fails. It connects with TLS when
`secure` is true. The certificate of the server is checked against the
certificate authorities that Java trusts. To trust a private one with Camel
JBang or Camel Main, set the appl [...]
+ Good to know: a plain NATS subject stores nothing: only the receivers
subscribed at that moment get the message, unless a JetStream stream takes in
the subject. With `jetstreamEnabled`, the sink waits until the stream that
takes in the subject has stored the message, and sending fails with `503 No
Responders Available For Request` when no stream takes it in. The stream must
exist already, for example created by `nats-source`. On a server without
JetStream, `jetstreamEnabled` is ign [...]
required:
- topic
- servers
@@ -70,17 +70,17 @@ spec:
pattern: "^(file|classpath|http|https):.+"
jetstreamEnabled:
title: "JetStream Enabled"
- description: "If true, each message is published to the JetStream
stream `jetstreamName`, which stores it, instead of to plain NATS. The stream
must exist and take in the subject in `topic`."
+ description: "If true, each message is published to JetStream, and the
sink waits until the stream that takes in the subject in `topic` has stored it.
Sending fails when no stream takes in the subject, or when the server does not
answer within a few seconds."
type: boolean
default: false
jetstreamName:
title: "JetStream Stream Name"
- description: "The name of the JetStream stream that stores the
messages. Only used when `jetstreamEnabled` is true."
+ description: "Not used by the sink: the message is stored by the
stream that takes in the subject in `topic`, whatever its name."
example: "ORDERS"
type: string
jetstreamAsync:
title: "JetStream Async"
- description: "If true, JetStream calls are made without blocking a
thread while waiting for the server to answer. Only used when
`jetstreamEnabled` is true."
+ description: "How the sink waits for JetStream to confirm that it
stored a message. Either way it waits before it sends the next message. When
the server does not answer, sending fails after about 2 seconds when false, and
after 2 to 8 seconds when true. Only used when `jetstreamEnabled` is true."
type: boolean
default: true
dependencies:
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/nats-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/nats-source.kamelet.yaml
index ba21f90b1..d58427c23 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/nats-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/nats-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the payload, as bytes, unchanged. The
`CamelNatsSubject` header is the subject, and the NATS headers of the message
become headers too.
- Good to know: without JetStream, it receives only the messages published
while it is connected. With `jetstreamEnabled`, it creates the stream
`jetstreamName` for the subject when it does not exist, and each time it starts
it reads the stream from the beginning, so messages already read are received
again. When several copies of the integration run, each copy receives every
message, unless they share a `queueName`: then each message goes to only one of
them. This works only without [...]
+ Good to know: without JetStream, it receives only the messages published
while it is connected. With `jetstreamEnabled`, it creates the stream
`jetstreamName` for the subject when it does not exist. Each time it starts it
reads the stream from the beginning, so messages already read are received
again, unless `durableName` is set: then it carries on after the last message
it acknowledged. A message is acknowledged once it has been processed. A
message that fails is delivered again [...]
required:
- topic
- servers
@@ -58,7 +58,7 @@ spec:
- urn:camel:group:credentials
queueName:
title: "Queue Group"
- description: "The name of a NATS queue group to join. The subscribers
of `topic` that use the same queue group share its messages: each message goes
to only one of them. Set the same name on every copy of the integration when
you run several of them, so that each message is processed once; without it,
each copy receives every message. Only used without JetStream: with
`jetstreamEnabled`, each copy still receives every message of the stream."
+ description: "The name of a NATS queue group to join. The subscribers
of `topic` that use the same queue group share its messages: each message goes
to only one of them. Set the same name on every copy of the integration when
you run several of them, so that each message is processed once; without it,
each copy receives every message. Only used without JetStream: with
`jetstreamEnabled`, each copy still receives every message of the stream,
unless the copies share a `durableName`."
example: "order-workers"
type: string
pattern: "^\\S+$"
@@ -84,9 +84,25 @@ spec:
type: string
jetstreamAsync:
title: "JetStream Async"
- description: "If true, JetStream calls are made without blocking a
thread while waiting for the server to answer. Only used when
`jetstreamEnabled` is true."
+ description: "Not used by the source, which receives the messages the
same way whether it is true or false."
type: boolean
default: true
+ durableName:
+ title: "Durable Consumer Name"
+ description: "The name of a durable JetStream consumer: the server
keeps it, and the position of the integration in the stream, when the
integration stops. After a restart, it carries on with the first message that
was not acknowledged, instead of reading the stream from the beginning. Copies
of the integration with the same `durableName` share the messages: each message
goes to only one of them. A new durable consumer starts at the beginning of the
stream. On the server, the con [...]
+ example: "order-processor"
+ type: string
+ pattern: "^[A-Za-z0-9_-]+$"
+ ackPolicy:
+ title: "Acknowledge Policy"
+ description: "How the messages are acknowledged to JetStream, so that
it does not deliver them again. `Explicit`: each message on its own, once it
has been processed; a message that fails is delivered again. `All`: the same,
but acknowledging a message also acknowledges all the earlier ones, so a
message that failed is not delivered again once a later message succeeds.
`None`: JetStream counts a message as done when it sends it, so a message that
fails is not delivered again. Whe [...]
+ type: string
+ enum: ["Explicit", "All", "None"]
+ maxDeliver:
+ title: "Maximum Deliveries"
+ description: "How many times at most a message is delivered, the first
time included. A message that fails is delivered again 5 seconds later, until
it has been delivered this many times; it then stays in the stream but is not
delivered again. When not set, a message that fails is delivered again with no
end. Only used when `jetstreamEnabled` is true."
+ example: 5
+ type: integer
types:
out:
mediaType: application/octet-stream
@@ -117,6 +133,9 @@ spec:
secure: "{{?secure}}"
credentialsFilePath: "{{?credentialsFilePath}}"
queueName: "{{?queueName}}"
+ durableName: "{{?durableName}}"
+ ackPolicy: "{{?ackPolicy}}"
+ maxDeliver: "{{?maxDeliver}}"
steps:
- to:
uri: kamelet:sink