davsclaus opened a new pull request, #3236:
URL: https://github.com/apache/camel-kamelets/pull/3236
Part of #3170: `nats-source` can use a durable JetStream consumer, which
carries on after a restart where it left off instead of reading the stream
again from the beginning, and which several copies of an integration can share.
It can also choose how messages are acknowledged and how many times a failing
message is delivered. The descriptions of the existing JetStream options of
both Kamelets are corrected after running them.
### The options
| Option | Kamelet | Template mapping |
|---|---|---|
| `durableName` (`pattern: ^[A-Za-z0-9_-]+$`) | `nats-source` |
`durableName: "{{?durableName}}"` |
| `ackPolicy` (`enum: Explicit, All, None`) | `nats-source` | `ackPolicy:
"{{?ackPolicy}}"` |
| `maxDeliver` (integer) | `nats-source` | `maxDeliver: "{{?maxDeliver}}"` |
- **`durableName`.** camel-nats creates a durable pull consumer. The server
keeps it and remembers which messages were acknowledged, so after a restart the
source carries on with the first unacknowledged message. Copies of the
integration with the same `durableName` share the messages, which `queueName`
cannot do with JetStream (#3205). camel-nats names the consumer
`<durableName>-durable` on the server, and the description says so, because
that is the name `nats consumer rm` needs. jnats refuses `.`, `*`, `>`, `/` and
`\` in the name, so the `pattern` allows only letters, digits, `-` and `_`.
- **`ackPolicy` and `maxDeliver`** are plain endpoint options of camel-nats,
so they need no `ConsumerConfiguration` bean. They apply to ephemeral consumers
too. A durable consumer keeps the values it was created with: if they change
later, jnats fails with `Existing consumer cannot be modified`, logs it, and
the source receives nothing until the consumer is deleted. The description says
this.
- **`requestTimeout` is not added to `nats-sink`.** The sink is `InOnly`
since #3156, so it never makes a request and waits for a reply. The only other
place camel-nats uses `requestTimeout` is the wait for the JetStream publish
acknowledgement with `jetstreamAsync=true`. jnats already cancels that publish
after its own request timeout, which is 2 seconds (the `connectionTimeout`) and
is applied at the next cleanup run, every 5 seconds. So the option could only
shorten the wait, never lengthen it. With the server paused, `requestTimeout:
500` failed after 0.5 s, the default (20000) after 6 to 7.5 s, `60000` after 7
s, and with `jetstreamAsync=false` the option is ignored (2 s). For a longer
wait, camel-nats would need to pass `requestTimeout` to the JetStream options
of jnats. That would be a Camel change.
- **Corrected descriptions of the existing JetStream options**, all checked
by running them:
- `nats-sink` does not use `jetstreamName`. The message is stored by
whichever stream takes in the subject, whatever its name. The `jetstreamName`
value is used only in dead code (`jsm.jetStream()` is never null).
- With `jetstreamEnabled`, the sink waits until the stream confirms that
it stored the message. It fails with `503 No Responders Available For Request`
when no stream takes in the subject.
- A stream also stores what the sink publishes without `jetstreamEnabled`.
- On a server without JetStream, the sink ignores `jetstreamEnabled` and
publishes as plain NATS.
- `nats-source` does not use `jetstreamAsync`.
- The sink's `jetstreamAsync` text said "without blocking a thread". In
fact it waits for the acknowledgement either way.
- In the "Good to know" paragraph of the source: what is acknowledged and
when, the redelivery of a failed message 5 seconds later, the 30 seconds after
an abrupt stop, and sharing with `durableName`. The source partial page has a
durable consumer section, an `ackPolicy` table and an example. The sink partial
page has a JetStream section and an example. Both examples were run against the
test server.
- None of the options has a `default`, so with them unset the endpoint URI
is exactly as on `main`.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT (`camel run
--local-kamelet-dir=kamelets`) against `nats:2` (nats-server 2.15.3) started
with `-js`, in Docker. `natsio/nats-box` (`nats` CLI) published, read streams
and inspected consumers. The routes logged each body and the
`CamelNatsDeliveryCounter` header. A route failed on purpose with
`throwException` for bodies that start with `bad`.
| Case | Result |
|---|---|
| Options unset, endpoint URIs from `main` and from this PR (`camel get
endpoint --json`): plain source, plain sink, JetStream source and sink | Same,
e.g.
`nats://ep.js?jetstreamAsync=true&jetstreamEnabled=true&jetstreamName=EPJS&servers=...`
|
| All three set |
`nats://dur.x?ackPolicy=All&durableName=allopts&...&maxDeliver=4&...`; `nats
consumer info` shows `ack_policy: all`, `max_deliver: 4` |
| `durableName: proc`: 5 messages, stop, 5 more published while it is
stopped, start again | First run m1–m5; second run n1–n5 only: nothing lost,
nothing repeated. Consumer `proc-durable` on the server |
| The same stream without `durableName` (as on `main`) | Receives all 10
again at start |
| `durableName`, slow route, `kill -9` while 10 messages are in flight | Ack
floor at k4. After restart k5–k10 arrive after the 30 s ack wait: k5, which was
in progress, twice; nothing lost |
| Two copies with `durableName: workers`, 40 messages | 19 and 21, no
duplicate, all 40 |
| Failing message, defaults (`Explicit`, no `maxDeliver`) | Delivered again
about every 5 s, 7 times in 40 s, with no end |
| Failing message, `maxDeliver: 3` | Delivered 3 times, then not again |
| Failing message, `ackPolicy: None` | Delivered once |
| Failing message then a good one, `ackPolicy: All` | Bad delivered once:
the good one's ack acknowledged it |
| Failing message alone, `ackPolicy: All` | Delivered again (3 times in 16
s) |
| `maxDeliver: 2` without `durableName` (ephemeral consumer) | Delivered 2
times |
| Existing durable consumer started again with `maxDeliver: 5` instead of 3
| Integration starts; WARN `[SUB-90016] Existing consumer cannot be modified.
Changed fields: [maxDeliver]`; a new message is not received |
| The same after `nats consumer rm FAILmd fmd-durable` | New consumer reads
the stream from the beginning; bad message delivered 5 times |
| `durableName: a.b` | WARN `Durable must be in the printable ASCII range
and cannot include '*', '.', '>' ...`: hence the `pattern` |
| `durableName`, `ackPolicy`, `maxDeliver` without `jetstreamEnabled` |
Ignored: plain NATS, message received, no stream or consumer created |
| Source `jetstreamAsync: false` | Receives as with `true` |
| Sink `jetstreamEnabled`, stream `SINK` takes in the subject; with
`jetstreamName: SINK`, with `jetstreamName: NOPE`, and without
`jetstreamEnabled` | All 3×5 messages stored in `SINK` (read back with `nats
stream get`, with the `Content-Type` header); no stream `NOPE` created |
| Sink `jetstreamEnabled`, no stream for the subject | Each send fails: `503
No Responders Available For Request` |
| Sink `jetstreamEnabled` on a server without `-js` | Sent with no error; a
`nats sub` subscriber received all 5 |
| Server paused (`docker pause`): `requestTimeout` 500 / unset / 60000 / 500
with `jetstreamAsync=false` (tested, then left out) | Fails after 0.5 s / 6–7.5
s / 7 s / 2 s |
| Partial-page examples (durable source, JetStream sink), address replaced |
Source created `ORDERS` and received the sink's messages; `nats consumer rm
ORDERS order-processor-durable` worked |
`./mvnw clean verify -DskipTests` and `./mvnw clean verify` from the root
both pass. The generated `kamelets-index.json` and the `library/camel-kamelets`
copies of the Kamelets were regenerated and are committed. The copies were made
with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. No Citrus test was added.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]