This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-jbang-examples.git
The following commit(s) were added to refs/heads/main by this push:
new 2dad588 fail-well/chaos - chaos rules break the shop on purpose, and
a scenario gates the build (#134)
2dad588 is described below
commit 2dad588d3d8a0a3677cb8337226b0dc1c405eadf
Author: Claus Ibsen <[email protected]>
AuthorDate: Tue Nov 10 16:34:34 2026 +0100
fail-well/chaos - chaos rules break the shop on purpose, and a scenario
gates the build (#134)
* fail-well/chaos - chaos rules break the shop on purpose, and a scenario
gates the build
The shop charges an order every second at a payment service over HTTP and
hands
it to shipping over seda. Chaos rules added while it runs make the payment
service answer 503, go down for 10 seconds, fail shipping and fail the dead
letter channel too, and the README says what each one shows in the log, in
camel get chaos and in the Camel TUI. chaos/payment-outage.chaos replays a
bad
half minute with a seed and three expectations; with --max-seconds its
verdict
is the exit code.
Needs Camel 4.24 (camel-chaos is new there), so the CI entry comes with the
move
of the examples to Camel 4.24.0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Claus Ibsen <[email protected]>
* fail-well/chaos - the scenario is in test/chaos, which camel run leaves
alone (CAMEL-26761)
The catalog generator takes the scenarios of test/chaos as files of the
example, as camel run --chaos=test/chaos/payment-outage.chaos needs them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Claus Ibsen <[email protected]>
* fail-well/chaos - a refunds route from Kafka (stubbed) with a gap on
purpose, for an AI to find: let there be chaos (CAMEL-26760)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Claus Ibsen <[email protected]>
---------
Signed-off-by: Claus Ibsen <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.gitignore | 2 +-
README.md | 3 +-
camel-jbang-example-catalog.json | 51 +++++
fail-well/README.md | 3 +-
fail-well/chaos/README.md | 245 ++++++++++++++++++++++++
fail-well/chaos/application.properties | 29 +++
fail-well/chaos/chaos.camel.yaml | 63 ++++++
fail-well/chaos/metadata.json | 40 ++++
fail-well/chaos/parked.camel.yaml | 18 ++
fail-well/chaos/refunds.camel.yaml | 25 +++
fail-well/chaos/test/chaos.citrus.it.yaml | 42 ++++
fail-well/chaos/test/chaos/payment-outage.chaos | 11 ++
generate-catalog.sh | 10 +-
13 files changed, 538 insertions(+), 4 deletions(-)
diff --git a/.gitignore b/.gitignore
index db62dd3..c16bf84 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,7 +8,7 @@ camel-runner.jar
.citrus-jbang/
# written by the csv-to-json example at runtime
outbox/
-# written by the error-handling example at runtime
+# written by the error-handling and chaos examples at runtime
parked/
# written by the file-processing example at runtime
inbox/
diff --git a/README.md b/README.md
index 9822da3..2061360 100644
--- a/README.md
+++ b/README.md
@@ -80,12 +80,13 @@ The routing patterns: content-based router, splitter,
aggregator, filter and mul
### [Fail well](fail-well/)
-Retries, a dead letter channel, and a circuit breaker in front of a flaky
service.
+Retries, a dead letter channel, and a circuit breaker in front of a flaky
service, then chaos rules that break the routes on purpose to prove it.
| Example | What you will see | Needs |
|---|---|---|
| [Error handling](fail-well/error-handling/) | The three orders go to a
payment provider: the first is charged at once, the second gets no answer twice
and is charged on the third attempt after two retries logged as warnings, and
the third is declined, logged as such and parked as a file for manual review. |
nothing |
| [Circuit breaker](fail-well/circuit-breaker/) | A stock check calls the
supplier every second; the supplier goes down for nine calls, the breaker opens
after two failures in its window of four, answers from the fallback without
calling the supplier while open, tries it again after five seconds and closes
once a call succeeds; each line logs the breaker state and whether the supplier
failed or was not called. | nothing |
+| [Chaos](fail-well/chaos/) | The shop charges an order every second at a
payment service over HTTP and hands it to shipping; chaos rules added while it
runs make the payment service answer 503, go down for 10 seconds and fail
shipping, the retries recover most orders and the dead letter channel parks the
rest, and a scenario file replays a bad half minute with a seed and ends with
the verdict PASS as its exit code; a refunds route from Kafka (stubbed) has a
gap on purpose, for an AI to [...]
### [Connect without a service](connect/)
diff --git a/camel-jbang-example-catalog.json b/camel-jbang-example-catalog.json
index 261b291..ce96df4 100644
--- a/camel-jbang-example-catalog.json
+++ b/camel-jbang-example-catalog.json
@@ -913,6 +913,57 @@
],
"order": 1
},
+ {
+ "name": "fail-well/chaos",
+ "title": "Chaos",
+ "description": "The shop charges an order every second at a payment
service over HTTP and hands it to shipping; chaos rules added while it runs
make the payment service answer 503, go down for 10 seconds and fail shipping,
the retries recover most orders and the dead letter channel parks the rest, and
a scenario file replays a bad half minute with a seed and ends with the verdict
PASS as its exit code; a refunds route from Kafka (stubbed) has a gap on
purpose, for an AI to find w [...]
+ "level": "fail-well",
+ "teaches": {
+ "components": [
+ "timer",
+ "kafka",
+ "http",
+ "platform-http",
+ "seda",
+ "direct",
+ "file",
+ "log"
+ ],
+ "eips": [
+ "errorHandler",
+ "deadLetterChannel",
+ "redeliveryPolicy",
+ "setVariable",
+ "setBody",
+ "removeHeaders"
+ ],
+ "languages": [
+ "simple"
+ ]
+ },
+ "tags": [
+ "shop",
+ "chaos",
+ "resilience",
+ "ai",
+ "error-handling",
+ "dead-letter",
+ "tui"
+ ],
+ "bundled": true,
+ "requiresDocker": false,
+ "hasCitrusTests": true,
+ "files": [
+ "README.md",
+ "application.properties",
+ "chaos.camel.yaml",
+ "parked.camel.yaml",
+ "refunds.camel.yaml",
+ "test/chaos/payment-outage.chaos"
+ ],
+ "needs": "Camel 4.24 or newer",
+ "order": 3
+ },
{
"name": "fail-well/circuit-breaker",
"title": "Circuit breaker",
diff --git a/fail-well/README.md b/fail-well/README.md
index 8e50e0f..8e34dff 100644
--- a/fail-well/README.md
+++ b/fail-well/README.md
@@ -1,12 +1,13 @@
# Fail well
<!-- group:start -->
-Retries, a dead letter channel, and a circuit breaker in front of a flaky
service.
+Retries, a dead letter channel, and a circuit breaker in front of a flaky
service, then chaos rules that break the routes on purpose to prove it.
| Example | What you will see | Needs |
|---|---|---|
| [Error handling](error-handling/) | The three orders go to a payment
provider: the first is charged at once, the second gets no answer twice and is
charged on the third attempt after two retries logged as warnings, and the
third is declined, logged as such and parked as a file for manual review. |
nothing |
| [Circuit breaker](circuit-breaker/) | A stock check calls the supplier every
second; the supplier goes down for nine calls, the breaker opens after two
failures in its window of four, answers from the fallback without calling the
supplier while open, tries it again after five seconds and closes once a call
succeeds; each line logs the breaker state and whether the supplier failed or
was not called. | nothing |
+| [Chaos](chaos/) | The shop charges an order every second at a payment
service over HTTP and hands it to shipping; chaos rules added while it runs
make the payment service answer 503, go down for 10 seconds and fail shipping,
the retries recover most orders and the dead letter channel parks the rest, and
a scenario file replays a bad half minute with a seed and ends with the verdict
PASS as its exit code; a refunds route from Kafka (stubbed) has a gap on
purpose, for an AI to find with [...]
Start with [Error handling](error-handling/); the examples read best in the
order above, each one building on what the one before it set up.
diff --git a/fail-well/chaos/README.md b/fail-well/chaos/README.md
new file mode 100644
index 0000000..ac1d20c
--- /dev/null
+++ b/fail-well/chaos/README.md
@@ -0,0 +1,245 @@
+# Chaos
+
+The shop takes an order every second, charges it at the payment service over
HTTP and hands it to shipping. Nothing
+in the routes fails on its own: you break things on purpose while it runs,
with chaos rules, and watch whether the
+retries, the dead letter channel and the hand-off to shipping do what they
should. A scenario file then replays a bad
+half minute at the payment service, and its verdict is the exit code, so it
can gate a build.
+
+Chaos is new in Camel 4.24
([camel-chaos](https://camel.apache.org/components/next/others/chaos.html)),
and it is for
+development only: it is on with profile `dev`, the default of `camel run`, and
never with profile `prod`.
+
+## What you will see
+
+Without rules, every order is charged and shipped:
+
+```text
+INFO ... chaos.camel.yaml:35 : Order ORD-1 charged: {"payment": "PAY-30456"}
+INFO ... chaos.camel.yaml:45 : Order ORD-1 shipped
+```
+
+With a rule that makes the payment service answer 503 to one call in four, the
error handler retries, and the order
+is charged on the second or third attempt; the few that fail three times in a
row are parked:
+
+```text
+WARN ... DeadLetterChannel : Failed delivery for (...) at orders[to1]
chaos.camel.yaml:33. On delivery attempt: 0 caught:
org.apache.camel.http.base.HttpOperationFailedException: HTTP operation failed
invoking http://localhost:8080/payment with statusCode: 503
+INFO ... chaos.camel.yaml:35 : Order ORD-45 charged: {"payment": "PAY-27475"}
+...
+WARN ... DeadLetterChannel : Failed delivery for (...) at orders[to1]
chaos.camel.yaml:33. On delivery attempt: 2 caught:
org.apache.camel.http.base.HttpOperationFailedException: HTTP operation failed
invoking http://localhost:8080/payment with statusCode: 503
+INFO ... parked.camel.yaml:12 : Order ORD-47 parked for manual review: HTTP
operation failed invoking http://localhost:8080/payment with statusCode: 503
+```
+
+and the scenario ends with its verdict:
+
+```text
+Chaos verdict: PASS (3 expectations met)
+```
+
+## Install Camel CLI
+
+Install [JBang](https://www.jbang.dev/download/) and the Camel CLI as
described in the
+[root README](../../README.md#install-the-camel-cli); `camel --version`
confirms the install. This example needs
+Camel 4.24 or newer.
+
+## Run it
+
+```shell
+camel run * --chaos
+```
+
+`--chaos` adds camel-chaos to the integration. Leave it running, and give the
commands below in another terminal.
+Stop it with `ctrl` + `c`: when rules were added, the chaos report is logged
as the application stops.
+
+## How it works
+
+- `chaos.camel.yaml` is the shop. Route `orders` creates an order every second
(`order.period`), charges it with a
+ `POST` to `http://localhost:8080/payment` and sends it to `seda:shipping`, a
hand-off to route `shipping` that does
+ not wait for it. The `errorHandler` of the file, a dead letter channel to
`direct:parked` with two retries half a
+ second apart, applies to both routes. Route `payment` is the payment
service, a `platform-http` route of the same
+ application, so the example needs nothing else to run; it has
`noErrorHandler`, so a failure there is the caller's
+ to deal with.
+- `parked.camel.yaml` writes an order that could not be charged or shipped to
`parked/<order id>.json`. It is in a
+ file of its own, with an error handler of its own, as in [Error
handling](../error-handling/).
+- `refunds.camel.yaml` pays refunds back at the payment service. They arrive
on the Kafka topic `refunds`; the shop
+ needs no broker, as `camel.jbang.stub=kafka` in `application.properties`
stubs Kafka, and a refund is sent with
+ `camel cmd send --endpoint=kafka:refunds --body='{"refundId": "R-1"}'`. The
route has only the default error
+ handler, on purpose: see [Let there be chaos](#let-there-be-chaos).
+- `test/chaos/payment-outage.chaos` is a scenario: chaos rules in a file, one
per line, with a seed, so the same
+ experiment runs again.
+- A chaos rule is one line: what it targets (an `endpoint=`, a `route=`, a
`component=` or a `step=`), the fault
+ (`error`, `delay`, `drop`, `duplicate`, `corrupt`, `stall`), how often
(`rate=`) and for how long (`until=`), and
+ optionally what must still hold (`expect=`). A rule ends by itself, at the
latest after 10 minutes.
+
+## Break it step by step
+
+1. Ask the integration what is worth trying. Chaos reads the routes and
suggests rules, the most worth trying first:
+
+ ```shell
+ camel cmd chaos-suggest
+ ```
+
+ ```text
+ Chaos rules to try, the most worth trying first:
+
+ 1. route orders: Returns an error (HTTP status) (errors)
+ an HTTP call to http://localhost:8080/payment without a circuit breaker:
what does the route do when the service answers 503?
+ camel cmd chaos-add --rule='endpoint=http://localhost:8080/payment error
status=503 rate=0.25 until=10m'
+
+ 2. route orders: The route after a hand-off fails (lost)
+ route orders hands the message to route shipping over seda and does not
wait: when shipping fails, is the message lost?
+ camel cmd chaos-add --rule='route=shipping error rate=0.2 until=2m
expect=orders->shipping:lost<=0 within=3m'
+ ...
+ ```
+
+2. Make the payment service answer 503 to one call in four, the first
suggestion:
+
+ ```shell
+ camel cmd chaos-add --rule='endpoint=http://localhost:8080/payment error
status=503 rate=0.25 until=10m'
+ ```
+
+ ```text
+ Added chaos rule r1: fail 25% of the sends to http://localhost:8080/payment
with HTTP status 503, for 10 minutes
+ ```
+
+ The log shows the retries, and once in a while an order parked in
`parked/`. `camel get chaos` shows how the rule
+ is doing: its observed rate, and the effect of its faults (`60 recovered, 4
handled`: recovered by a retry, or
+ handled by the dead letter channel). Before you add a rule, `chaos-add
--dry-run` says what it would hit, without
+ adding it.
+
+3. Take the payment service down for 10 seconds. A suspended `platform-http`
route answers 503, so the HTTP client
+ retries once by itself, then the error handler twice, and the orders of
those seconds are parked:
+
+ ```shell
+ camel cmd chaos-add --rule='route=payment stall duration=10s'
+ ```
+
+4. Fail route `shipping` one message in five, with an expectation that no
order charged is lost on the way:
+
+ ```shell
+ camel cmd chaos-add --rule='route=shipping error rate=0.2 until=1m
expect=orders->shipping:lost<=0 within=90s'
+ ```
+
+ The dead letter channel of the file covers `shipping` too: a retry ships
the order, or else it is parked, so
+ nothing is lost; when the expectation is decided, `camel get chaos
--report` says `met, 0 lost`.
+
+5. Now the last safety net: fail the dead letter channel too, while the
payment service is down:
+
+ ```shell
+ camel cmd chaos-add --rule='endpoint=http://localhost:8080/payment error
status=503 until=10s' --rule='endpoint=direct:parked error at=dlq until=10s'
+ ```
+
+ The orders of those seconds are neither charged nor parked. The log has
only a WARN for each
+ (`New exception occurred during processing by the
DeadLetterChannel[direct:parked]`), and the report counts them
+ as lost, with the verdict FAIL:
+
+ ```text
+ chaos (pid 58473): FAIL
+ lost: 9
+ FAIL: rule r5 failed or dropped 9 sends of the dead letter channel, which
handled them without delivering them: 9 lost
+ ```
+
+6. Read the report: per rule its matches, faults and effect, per route the
exchanges completed, failed, handled by
+ the error handler and redelivered, the recent faults, and per route which
kinds of fault have been tried and which
+ not yet:
+
+ ```shell
+ camel get chaos --report
+ ```
+
+ `camel cmd chaos-clear --reset` removes the rules and starts the report
again, for the next experiment.
+
+## Watch it in the Camel TUI
+
+```shell
+camel tui
+```
+
+- The chaos screen (`Chaos` under `0 More`) lists the rules with their live
rate, faults per second and effect, and
+ the report of the selected rule. `a` adds a rule from a form of known rules,
`w` lists what to try (the
+ suggestions), `e` changes, `x` removes and `c` clears a rule, `R` starts a
new experiment, and `s` saves the rules
+ as a scenario file.
+- `4 Timeline` shows per second the faults of each rule, and what each route
did: completed, recovered, handled,
+ failed, and the seconds route `payment` was down.
+- `t` on a rule asks the AI panel to make a test of it, with the seed, so it
fails the same messages again.
+
+## A scenario as a gate
+
+`test/chaos/payment-outage.chaos` is a bad half minute at the payment service,
with an expectation on each rule:
+
+```text
+seed=42
+
+# one call in four fails for 20 seconds: the redeliveries recover them, and
the shop still takes 20 orders
+endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=20s
expect=orders:20 within=25s
+# shipping fails one message in five for 15 seconds: its redeliveries still
ship every order charged
+route=shipping error rate=0.2 until=15s expect=orders->shipping:lost<=0
within=20s
+# then the payment service is down for 10 seconds: every order is charged or
parked, none fails
+route=payment stall duration=10s from=+20s expect=orders:failed<=0 within=15s
+```
+
+Run the shop with it for 45 seconds:
+
+```shell
+camel run * --chaos=test/chaos/payment-outage.chaos --max-seconds=45
+```
+
+The report is logged when it stops, and the verdict is the exit code: 0 for
PASS, 1 for FAIL, so a build can run it.
+`seed=42` makes the rules fail the same calls each time. `camel cmd chaos-load
test/chaos/payment-outage.chaos` loads the
+scenario into a shop that runs already.
+
+## Let there be chaos
+
+Instead of choosing the rules yourself, let an AI explore for a time budget.
It reads the suggestions, says what it
+expects (a hypothesis), runs an experiment (rules with expectations, and
messages for the route it is about), reads the
+verdict and chooses the next one. It needs an LLM; a local model with Ollama
is enough:
+
+```shell
+camel cmd chaos-explore --budget=10m
+```
+
+Route `refunds` has a gap on purpose: with only the default error handler, a
refund the payment service answers with
+503 fails, and is neither retried nor parked, while route `orders` recovers
from the same fault. An exploration finds
+it as a FAIL, and often the loss of step 5 as well: the orders lost when the
dead letter channel fails too.
+
+Each experiment ends with a verdict: PASS, FAIL, or INCONCLUSIVE when it
cannot tell (the faults never fired, or the
+route had no messages). The exploration saves its report, and a scenario of
each experiment that failed, in
+`test/chaos`, so `camel cmd chaos-load` replays it. In the Camel TUI, `l` on
the chaos screen shows the exploration:
+the budget as a strip of time with the experiments, and the selected one in
detail.
+
+## Try changing
+
+- Add `endpoint=direct:parked error at=dlq until=10s from=+20s` to the
scenario: the parked orders are lost, and the
+ verdict is FAIL, exit code 1.
+- Give route `refunds` an `errorHandler` with redeliveries, and let the AI
explore again: its refunds experiments
+ pass now.
+- Set `maximumRedeliveries` to 0 in `chaos.camel.yaml` and run the scenario:
the first rule parks the orders instead
+ of recovering them.
+- Wrap the payment call in a `circuitBreaker` (see [Circuit
breaker](../circuit-breaker/)) and ask
+ `camel cmd chaos-suggest` again: it suggests an outage, to see the breaker
open and the fallback answer.
+- Slow the payment service down, with an expectation on the time of an order:
+ `endpoint=http://localhost:8080/payment delay duration=100ms..2s rate=0.5
until=1m expect=orders:meanTime<=1s within=90s`.
+- Make the payment service charge twice:
`endpoint=http://localhost:8080/payment duplicate rate=0.1` sends one call
+ in ten again. Is your payment service idempotent?
+- Fail only some orders: `when="${variable.orderId} ends with '7'"` on any
rule fires only for the orders whose
+ number ends with 7.
+
+## Integration testing
+
+The example comes with a test in the [Citrus](https://citrusframework.org/)
YAML DSL,
+`test/chaos.citrus.it.yaml`, which the Camel CLI runs:
+
+```shell
+camel test run test/chaos.citrus.it.yaml
+```
+
+The test starts the shop with the scenario and verifies that the payment
service goes down and comes back, and that
+orders are still shipped.
+
+## Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+[let us know](https://camel.apache.org/community/support/).
+
+We also love contributors, so
+[get involved](https://camel.apache.org/community/contributing/) :-)
+
+The Camel riders!
diff --git a/fail-well/chaos/application.properties
b/fail-well/chaos/application.properties
new file mode 100644
index 0000000..3c21428
--- /dev/null
+++ b/fail-well/chaos/application.properties
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# a new order every second
+order.period=1000
+# the port of the payment service, which the shop calls on localhost
+camel.server.port=8080
+
+# the folder the parked route writes holds data, not sources: with camel run
--source-dir in dev mode,
+# a change in it does not reload the routes
+camel.jbang.dataDir=parked
+
+# route refunds reads Kafka: stub it, so the example needs no broker; a refund
is sent with camel cmd send
+camel.jbang.stub=kafka
diff --git a/fail-well/chaos/chaos.camel.yaml b/fail-well/chaos/chaos.camel.yaml
new file mode 100644
index 0000000..2d2ce5e
--- /dev/null
+++ b/fail-well/chaos/chaos.camel.yaml
@@ -0,0 +1,63 @@
+# The web shop takes an order every second, charges it at the payment service
over HTTP and hands it to
+# shipping. Nothing here fails on its own: the faults come from the chaos
rules you add while it runs.
+- errorHandler:
+ deadLetterChannel:
+ deadLetterUri: direct:parked
+ redeliveryPolicy:
+ maximumRedeliveries: 2
+ redeliveryDelay: 500
+ retryAttemptedLogLevel: WARN
+
+- route:
+ id: orders
+ from:
+ uri: timer
+ parameters:
+ timerName: orders
+ period: "{{order.period}}"
+ includeMetadata: true
+ steps:
+ - setVariable:
+ name: orderId
+ expression:
+ simple:
+ expression: "ORD-${header.CamelTimerCounter}"
+ - removeHeaders:
+ pattern: "Camel*"
+ - setBody:
+ expression:
+ simple:
+ expression: |-
+ {"orderId": "${variable.orderId}", "customer":
"C-${random(100,999)}", "country": "DK",
+ "lines": [{"sku": "CAMEL-TSHIRT", "qty": ${random(1,3)},
"price": 19.95}], "status": "paid"}
+ - to:
+ uri: "http://localhost:{{camel.server.port}}/payment";
+ - log:
+ message: "Order ${variable.orderId} charged: ${body}"
+ - to:
+ uri: seda:shipping
+
+- route:
+ id: shipping
+ from:
+ uri: seda:shipping
+ steps:
+ - log:
+ message: "Order ${variable.orderId} shipped"
+
+# The payment service the shop calls. In a real shop it runs elsewhere; here
it is a route of
+# the same application, so the example needs nothing else to run.
+- route:
+ id: payment
+ errorHandler:
+ noErrorHandler: {}
+ from:
+ uri: platform-http
+ parameters:
+ path: /payment
+ httpMethodRestrict: POST
+ steps:
+ - setBody:
+ expression:
+ simple:
+ expression: "{\"payment\": \"PAY-${random(10000,99999)}\"}"
diff --git a/fail-well/chaos/metadata.json b/fail-well/chaos/metadata.json
new file mode 100644
index 0000000..b707a84
--- /dev/null
+++ b/fail-well/chaos/metadata.json
@@ -0,0 +1,40 @@
+{
+ "title": "Chaos",
+ "description": "The shop charges an order every second at a payment
service over HTTP and hands it to shipping; chaos rules added while it runs
make the payment service answer 503, go down for 10 seconds and fail shipping,
the retries recover most orders and the dead letter channel parks the rest, and
a scenario file replays a bad half minute with a seed and ends with the verdict
PASS as its exit code; a refunds route from Kafka (stubbed) has a gap on
purpose, for an AI to find with [...]
+ "level": "fail-well",
+ "order": 3,
+ "teaches": {
+ "components": [
+ "timer",
+ "kafka",
+ "http",
+ "platform-http",
+ "seda",
+ "direct",
+ "file",
+ "log"
+ ],
+ "eips": [
+ "errorHandler",
+ "deadLetterChannel",
+ "redeliveryPolicy",
+ "setVariable",
+ "setBody",
+ "removeHeaders"
+ ],
+ "languages": [
+ "simple"
+ ]
+ },
+ "tags": [
+ "shop",
+ "chaos",
+ "resilience",
+ "ai",
+ "error-handling",
+ "dead-letter",
+ "tui"
+ ],
+ "needs": "Camel 4.24 or newer",
+ "bundled": true
+}
diff --git a/fail-well/chaos/parked.camel.yaml
b/fail-well/chaos/parked.camel.yaml
new file mode 100644
index 0000000..f1574b2
--- /dev/null
+++ b/fail-well/chaos/parked.camel.yaml
@@ -0,0 +1,18 @@
+# The orders the shop could not charge or ship are written as files for
someone to look at.
+# The route has an error handler of its own, so a failure in parking an order
is logged
+# instead of being sent back here.
+- route:
+ id: parked
+ errorHandler:
+ deadLetterChannel:
+ deadLetterUri: "log:parking-failed?level=ERROR"
+ from:
+ uri: direct:parked
+ steps:
+ - log:
+ message: "Order ${variable.orderId} parked for manual review:
${exception.message}"
+ - to:
+ uri: file
+ parameters:
+ directoryName: parked
+ fileName: "${variable.orderId}.json"
diff --git a/fail-well/chaos/refunds.camel.yaml
b/fail-well/chaos/refunds.camel.yaml
new file mode 100644
index 0000000..351da9a
--- /dev/null
+++ b/fail-well/chaos/refunds.camel.yaml
@@ -0,0 +1,25 @@
+# Refunds arrive on Kafka from the returns desk, and the shop pays them back
at the payment service.
+# It is a route of its own: a failed refund is logged, without the dead letter
channel of the shop.
+- route:
+ id: refunds
+ errorHandler:
+ defaultErrorHandler: {}
+ from:
+ uri: kafka:refunds
+ parameters:
+ brokers: localhost:9092
+ groupId: shop
+ steps:
+ - setVariable:
+ name: refundId
+ expression:
+ jsonpath:
+ expression: "$.refundId"
+ - log:
+ message: "Refund ${variable.refundId} received"
+ - removeHeaders:
+ pattern: "*"
+ - to:
+ uri: "http://localhost:{{camel.server.port}}/payment";
+ - log:
+ message: "Refund ${variable.refundId} paid back: ${body}"
diff --git a/fail-well/chaos/test/chaos.citrus.it.yaml
b/fail-well/chaos/test/chaos.citrus.it.yaml
new file mode 100644
index 0000000..627510b
--- /dev/null
+++ b/fail-well/chaos/test/chaos.citrus.it.yaml
@@ -0,0 +1,42 @@
+name: chaos-test
+description: The chaos example runs its scenario, takes the payment service
down and brings it back, and still ships orders
+actions:
+ - camel:
+ cli:
+ run:
+ integration:
+ name: "chaos"
+ file: "../chaos.camel.yaml"
+ systemProperties:
+ file: "../application.properties"
+ resources:
+ - "parked.camel.yaml"
+ - "test/chaos/payment-outage.chaos"
+ args:
+ - "--chaos=test/chaos/payment-outage.chaos"
+ - camel:
+ cli:
+ verify:
+ integration: "chaos"
+ logMessage: "Order ORD-1 shipped"
+ - camel:
+ cli:
+ verify:
+ integration: "chaos"
+ logMessage: "Chaos rule r3 suspended route payment for 10s"
+ maxAttempts: 30
+ delayBetweenAttempts: 1000
+ - camel:
+ cli:
+ verify:
+ integration: "chaos"
+ logMessage: "Chaos rule r3 resumed route payment"
+ maxAttempts: 20
+ delayBetweenAttempts: 1000
+ - camel:
+ cli:
+ verify:
+ integration: "chaos"
+ logMessage: "Order ORD-35 shipped"
+ maxAttempts: 20
+ delayBetweenAttempts: 1000
diff --git a/fail-well/chaos/test/chaos/payment-outage.chaos
b/fail-well/chaos/test/chaos/payment-outage.chaos
new file mode 100644
index 0000000..28ba946
--- /dev/null
+++ b/fail-well/chaos/test/chaos/payment-outage.chaos
@@ -0,0 +1,11 @@
+# A bad half minute at the payment service: for 20 seconds it answers 503 to
one call in four,
+# then it is down for 10 seconds. The shop must keep taking orders, ship every
order it charged,
+# and park the orders it could not charge instead of losing them.
+seed=42
+
+# one call in four fails for 20 seconds: the redeliveries recover them, and
the shop still takes 20 orders
+endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=20s
expect=orders:20 within=25s
+# shipping fails one message in five for 15 seconds: its redeliveries still
ship every order charged
+route=shipping error rate=0.2 until=15s expect=orders->shipping:lost<=0
within=20s
+# then the payment service is down for 10 seconds: every order is charged or
parked, none fails
+route=payment stall duration=10s from=+20s expect=orders:failed<=0 within=15s
diff --git a/generate-catalog.sh b/generate-catalog.sh
index 9fa9660..840921f 100755
--- a/generate-catalog.sh
+++ b/generate-catalog.sh
@@ -93,6 +93,14 @@ for dirpath, dirnames, filenames in
sorted(os.walk(repo_root)):
rel = os.path.relpath(os.path.join(sub_dirpath, f), dirpath)
if os.path.relpath(os.path.join(sub_dirpath, f), repo_root)
not in ignored:
files.append(rel)
+ # the chaos scenarios of test/chaos are files the example runs with (camel
run --chaos=test/chaos/name.chaos)
+ chaos_dir = os.path.join(dirpath, "test", "chaos")
+ for sub_dirpath, sub_dirnames, sub_filenames in os.walk(chaos_dir):
+ sub_dirnames[:] = [d for d in sub_dirnames if not d.startswith(".")]
+ for f in sub_filenames:
+ rel = os.path.relpath(os.path.join(sub_dirpath, f), dirpath)
+ if not f.startswith(".") and
os.path.relpath(os.path.join(sub_dirpath, f), repo_root) not in ignored:
+ files.append(rel)
files.sort()
requires_docker = ("compose.yaml" in filenames or "docker-compose.yaml" in
filenames
@@ -146,7 +154,7 @@ GROUPS = [
("run", "Run", "Running Camel: timers and cron schedules, a bean in a
route, properties and profiles."),
("transform", "Transform and map", "JSON, XML and CSV in and out,
field-by-field mapping, Groovy and XSLT."),
("route", "Route", "The routing patterns: content-based router, splitter,
aggregator, filter and multicast."),
- ("fail-well", "Fail well", "Retries, a dead letter channel, and a circuit
breaker in front of a flaky service."),
+ ("fail-well", "Fail well", "Retries, a dead letter channel, and a circuit
breaker in front of a flaky service, then chaos rules that break the routes on
purpose to prove it."),
("connect", "Connect without a service", "Files, an HTTP client and a REST
server; everything runs inside the example."),
("connect-service", "Connect to one service", "SQL, JMS, MQTT, Kafka and
FTP against a service the Camel CLI starts for you with `camel infra run`."),
("contracts", "Contracts and security", "An OpenAPI contract served and
called, and an API protected by Keycloak."),
