gnodet-bot commented on code in PR #28539:
URL: https://github.com/apache/camel/pull/28539#discussion_r4385916704
##########
design/chaos.adoc:
##########
@@ -0,0 +1,408 @@
+---
+title: Chaos
+authors:
+ - "@allthingssecurity"
+reviewers: ["@davsclaus"]
+approvers: []
+creation-date: 2026-10-26
+last-updated: 2026-10-27
+status: proposed
+see-also:
+ - "CAMEL-26330"
+ - "CAMEL-26331"
+ - "CAMEL-26338"
+ - "CAMEL-25911"
+ - "CAMEL-25684"
+ - "CAMEL-25682"
+ - "CAMEL-26286"
+replaces: []
+superseded-by: []
+---
+
+== Summary
+
+`camel-chaos` lets a developer, or an AI tool, turn on faults in a running
Camel application and see how its routes
+cope. A rule such as "fail 30% of the sends to `kafka:orders` for one minute"
or "stall route `poller` for 20 s" is
+added while the application runs, for example in dev mode (`camel run --dev`).
A report then says per route how many
+exchanges completed, failed, were handled by the error handler or were
redelivered, whether the consumer keeps a
+failed message, and, for a rule with an expectation, how many messages were
lost, with a verdict.
+
+With the `camel-chaos` JAR and profile `dev` (the default of `camel run`), or
`camel.chaos.enabled=true`, its hooks
+are installed and stay idle until a rule is added. Without the JAR, as in
production, nothing of it is there. Faults
+run in real time, and the same rules can run inside a `camel-test-dst` world
in virtual time.
+
+Chaos needs two small changes in camel-core: a hook for every send to an
endpoint (CAMEL-26331) and a hook for every
+step in the channel. The other levels use SPIs that exist today.
+
+== Motivation
+
+Routes are written for the happy path, and their error handling is rarely
exercised before production. Questions such
+as "what happens to the messages in flight when the broker is slow for 30 s",
"does this `onException` really match the
+error the HTTP endpoint returns" or "does the consumer resume after its route
was suspended" are answered today by
+hand-written tests, one route and one fault at a time.
+
+A prototype (CAMEL-26330 links it), which hooks only `to`, reproduced
CAMEL-25684 (camel-slack, a route suspend in the
+middle of a poll) and CAMEL-25682 (camel-resilience4j, a timeout while the
task writes its result) on the code from
+before their fixes. The recent "remove a route" bugs (CAMEL-26286,
CAMEL-26287, CAMEL-26288, CAMEL-26302 to
+CAMEL-26305: one route goes, and another route loses the cache, channel or
session it shares) have a shape a chaos
+rule can produce on demand.
+
+AI tools that write and change routes need the same: a way to try a route
against failures and read a short result.
+
+== Goals
+
+- Out of the box chaos for any Camel application in development: faults on
sends, on the messages that enter a route,
+ on consumer polls, on processing steps and on the route lifecycle.
+- Switched on by the classpath with profile `dev` or
`camel.chaos.enabled=true`, idle until a rule is added; rules
+ added, changed and removed live.
+- Configuration through `camel.chaos.*` options of camel-main, as for
`camel.debug.*` and `camel.trace.*`.
+- A report per route with counts and a verdict, readable by a person and by a
tool.
+- Control from the CLI (`camel cmd chaos`) and the camel-jbang MCP server; a
dev console shows the rules and the report.
+- Real time by default; the same rules in virtual time inside a DST world.
+- A seed makes a run repeatable.
+
+== Non-goals
+
+- Production use. Chaos fails closed, and stays off with profile `prod` even
when enabled.
+- Custom fault types written by users. The first version has a fixed set; a
plugin SPI (a JAR with its own faults) can
+ come later.
+- Faults outside Camel: network partitions, killing processes, disk or clock
faults of the host. Tools outside Camel
+ do that; this is about what Camel does with the failures it sees.
+- The video, the models (TLA+, Lean, Z3) and the demo scripts of the
prototype. They stay on the prototype branch.
+
+== Proposal
+
+=== Switched on by the classpath
+
+`camel-chaos` registers a `ContextServicePlugin`
(`META-INF/services/org.apache.camel.spi.ContextServicePlugin`).
+The `CamelContext` loads the plugins found on the classpath while it is built,
before the routes are created.
+
+In camel-main the profile and the `camel.chaos.*` options are set after the
plugins are loaded, so the plugin reads
+them when the context initializes (`LifecycleStrategy.onContextInitializing`),
which is still before the routes are
+created. The profile comes from `ExtendedCamelContext.getProfile()`.
+
+The plugin installs the hooks below only with profile `dev`, the default of
`camel run`, or with
+`camel.chaos.enabled=true` (a plain `Main`, tests, DST worlds), and never with
`prod`. Otherwise it fails closed. That
+includes no profile, the usual case for a plain `Main`, Spring Boot and
Quarkus: the profile is then null, not `prod`
+(`ProfileConfigurer.configureMain` returns early), and a `camel.chaos.rules`
left in `application.properties` does
+nothing. When it installs the hooks, it logs a WARN that chaos is active. Each
hook asks the `ChaosManager` whether a
+rule matches; with no rules that is one volatile read, and the call goes on
unchanged.
+
+=== Getting it
+
+- `camel run --chaos` adds the `camel:chaos` dependency, as `--observe` adds
`camel:observability-services` in
+ `Run.java`; `--dep camel:chaos` works too.
+- A plain `Main` or a test adds the `camel-chaos` dependency and sets
`camel.chaos.enabled=true`.
+- Spring Boot and Quarkus: the plugin works with any `CamelContext` and
installs the hooks with Camel profile `dev`.
+ The options are camel-main only, so rules come from the CLI connector; a
starter or an extension can bind
+ `camel.chaos.*` later.
+
+=== Options
+
+[cols="1,1,3"]
+|===
+|Option |Default |Description
+
+|`camel.chaos.enabled` |`false`, `true` with profile `dev` |Install the hooks:
`true` for a plain `Main`, tests and
+DST worlds, ignored with profile `prod`. They do nothing until a rule is added.
+|`camel.chaos.seed` | |Seed of the random choices; without it a seed is picked
and shown in the report.
+|`camel.chaos.rules` | |Rules to add at startup, in the text form of Rule
syntax, separated by `;` (marked
+`insecure:dev`, see Risks).
+|`camel.chaos.maxDuration` |`10m` |Upper bound for every rule: a rule without
`until` ends after it, and a
+`duration` of `stall` or `lifecycle` may not be longer (such a rule is
rejected when it is added).
+|`camel.chaos.reportOnShutdown` |`true` |Log the report when the context stops.
+|===
+
+They are a `ChaosConfigurationProperties` group in camel-main, as
`DebuggerConfigurationProperties` is for
+`camel.debug.*`, so the catalog and tooling know them.
+
+=== Rules
+
+A rule has a target, a fault, how often it fires, and when it ends:
+
+[source]
+----
+target: route=<id or pattern> | endpoint=<uri pattern> | component=<name> |
step=<processor id pattern>
+fault: error | delay | drop | corrupt | lifecycle (stall) (parameters: see
Rule syntax)
+rate: a probability (0.3), or every Nth match
+until: a number of matches, a duration, or removed by hand
+----
+
+Rules are added, changed, listed and removed live. A rule ends by itself after
its count or duration (at most
+`camel.chaos.maxDuration`), so a forgotten rule cannot keep breaking a dev
instance. Rules stay when the routes are
+reloaded in dev mode.
+
+==== Rule syntax
+
+A rule is one line: the target, the fault and its parameters as `name=value`,
then `rate` and `until`. `rate=0.3` is a
+probability and `rate=every:5` fires on every 5th match; without `rate` every
match fires. `until=100` is a number of
+matches and `until=2m` a duration; without `until` the rule ends after
`camel.chaos.maxDuration`. A value with spaces
+is quoted. Which hooks a fault can use is in the table of the next section.
+
+[cols="1,2,3"]
+|===
+|Fault |Parameters |Effect
+
+|`error`
+|`exception=<class>`, `message=<text>`; for the HTTP components
`status=<code>`, `header.<name>=<value>` and
+`body=<text>`
+|Throws that exception. With `status` it is the exception the HTTP component
throws for that status; when the endpoint
+has `throwExceptionOnFailure=false`, chaos returns the response instead, as
the real producer does: the status in the
+`CamelHttpResponseCode` header, the `header.*` values and the `body` (empty
without it). All parameters are optional:
+without `exception` (and without `status`) the fault throws a
`ChaosException`, a `RuntimeCamelException` of
+camel-chaos, with the `message` or a default one, on any endpoint. A rule
whose exception class is not found, is not
+a `Throwable`, or has neither a constructor taking a `String` message nor a
no-argument one is rejected when it is
+added, with an error that names the class.
+
+|`delay`
+|`duration=<time>`, such as `500ms` or `2s`
+|Waits that long before the send, the step or the poll.
+
+|`drop`
+|none
+|A send that reports success without calling the producer, or a skipped poll.
+
+|`corrupt`
+|`mode=headers\|charset\|truncate`
+|Changes the message: header values with random data, a body in an unusual
charset (characters outside US-ASCII), or
+an `InputStream` body that ends too soon. `charset` applies to a body that is
text or converts to text, and `truncate`
+to an `InputStream` or stream cache body; on another body the message is left
as it is and the report counts the rule
+as skipped for that message, so a fault that did nothing is visible.
+
+|`lifecycle`
+|`action=suspend\|stop\|remove`, `duration=<time>`
+|Suspends, stops or removes the target route, then resumes it, starts it or
adds its definition again after the
+duration. `stall duration=<time>` is short for `lifecycle action=suspend
duration=<time>`. Both take only a `route=`
+target; a rule with another target is rejected when it is added.
+|===
+
+`stall` and `lifecycle` act once on the route when the rule is added; `rate`
and `until` do not apply to them, and
+their `duration` is bounded by `camel.chaos.maxDuration`.
+
+In `application.properties`:
+
+[source,properties]
+----
+camel.chaos.seed = 42
+camel.chaos.rules = \
+ endpoint=kafka:orders* error exception=java.net.ConnectException
message="broker down" rate=0.3 until=1m;\
+ endpoint=https:inventory* error status=503 header.Retry-After=5 rate=every:4
until=50;\
+ route=poller stall duration=20s
+----
+
+With the CLI, against a running application:
+
+[source,bash]
+----
+camel cmd chaos add --rule='endpoint=kafka:orders* delay duration=2s rate=0.2
until=5m'
+camel cmd chaos add --rule='route=orders corrupt mode=truncate rate=0.1
until=20'
+camel cmd chaos add --rule='route=orders lifecycle action=remove duration=10s'
+----
+
+=== Hooks at all levels
+
+[cols="1,2,2"]
+|===
+|Level |Hook |Faults
+
+|Sends to any endpoint
+|The send hook of camel-core (CAMEL-26331): one helper that the six places
that fire `ExchangeSendingEvent` share
+(`SendProcessor`, `DefaultProducerCache` twice, `EventNotifierCallback`,
`MulticastProcessor`,
+`EventNotifierProducer`), with a hook before and after the producer.
+|`error` (the producer is not called), `delay`, `drop` (a send that reports
success), `corrupt` (the message sent).
+
+|Messages that enter a route
+|A `RoutePolicy`, added to every route by a `RoutePolicyFactory`. Its
`onExchangeBegin` sees every exchange that
+enters a route and can change the message.
+|`corrupt`: header values with random data, a body in an unusual charset, an
`InputStream` that ends too soon. Not
+`drop`: the exchange already belongs to its consumer, and dropping it there
would make the consumer commit a message
+the route never processed (a file consumer deletes the file), a loss caused by
chaos rather than by the route. `drop`
+is at the sends and the polls instead.
+
+|Consumer polls
+|The `PollingConsumerPollStrategy` of a `ScheduledPollConsumer`, which can be
changed at runtime with
+`setPollStrategy`. Its `begin`, `commit` and `rollback` run around every poll.
Chaos wraps the consumer's own strategy
+while a rule is active and puts it back when the rule ends.
+|A failed poll (`error`), a slow poll (`delay`), a skipped poll (`drop`).
+
+|Every step
+|The channel (`CamelInternalProcessor`), next to the check that skips a
processor disabled with
+`camel cmd disable-processor`. That check runs before the error handler of the
step, so a failure is raised where the
+error handler calls the step, as for a send. A small change in camel-core,
CAMEL-26338.
+|`delay` or `error` at a step.
+
+|Route lifecycle
+|The `RouteController` (suspend, resume, stop, start), and
`CamelContext.removeRoute` with the route definition added
+again.
+|`stall` (suspend, then resume); `lifecycle` (suspend, stop or remove the
route, then bring it back).
+|===
+
+The send hook covers every send whatever started it: `to`, `toD`, `wireTap`,
`enrich`, `recipientList`, `multicast`
+and a `ProducerTemplate`. The components stay the real ones (a
`KafkaComponent` stays a `KafkaComponent`), and the
+routes get no extra processors. For a send or a step of a route the fault is
raised inside the error handler, so it is
+a real failure for `onException`, redelivery and the dead letter channel.
+
+A fault has a real type where it matters, so that the `onException` of the
route is what gets tested: for the HTTP
+components, the exception the component throws for a status code
(`HttpOperationFailedException` of camel-http-base,
+`NettyHttpOperationFailedException` for camel-netty-http) with the status code
and headers given in the rule (or the
+response, with `throwExceptionOnFailure=false`); for any other endpoint, the
exception class and message given in the
+rule, or a `ChaosException` when the rule gives none.
+
+=== Time
+
+The `delay` fault sleeps through `TimeSource` (camel-util, CAMEL-25911), and
the report reads its times through it.
+The timed actions (the resume or restart after `stall` or `lifecycle`, the end
of a rule) are tasks on a scheduled
+executor of the `ExecutorServiceManager`, not threads of their own or a
`java.util.Timer`. By default that is the
+system clock, so a person sees a 20 s stall last 20 s.
+
+In a `camel-test-dst` world the harness runs those executors and installs its
virtual clock in `TimeSource`: a 10 minute
+stall takes milliseconds, and a seed gives the same report again, times
included, as they come from `TimeSource`. Event
+timestamps stay on `System` in a DST run (`test-dst.adoc`), so the report does
not use them. A `stall` or `lifecycle`
+rule meets the one-stack model of `test-dst.adoc`: a suspend or stop cannot
see an exchange that is blocked in a
+synchronous call below it on the stack finish, so such a world calls its
downstream with `DstCamel.asyncStep`. Chaos
+does not need DST.
+
+The Javadoc of `TimeSource` says it is internal and not an API for components.
`camel-chaos` may use it: it reads the
+clock and sleeps through it, and never installs a provider (only a test
harness such as `camel-test-dst` does that).
+The Javadoc is updated to say so with the first chaos pull request.
+
+=== The report
+
+The plugin counts per route with an event notifier, as the prototype does:
completed, failed, handled by the error
+handler, redelivered, and the faults that fired. Attempts are not tied to one
message: each exchange is counted once,
+so a file that fails and completes on the next poll is one failed and one
completed exchange. Tying them by a key (the
+file name, `JMSMessageID`, the Kafka offset) is not in the first version.
+
+A failed exchange shows as "failed, kept by the consumer" when Camel knows
that the consumer keeps the message and
+reads it again, and as "failed" otherwise. (An exchange the error handler
handled is not failed; its consumer commits
+it.) The plugin reads the consumer's endpoint options through its generated
configurer (`PropertyConfigurerGetter`),
+so options set on the component count too, without a dependency on the
components:
+
+[cols="1,3,2"]
+|===
+|Consumer |Kept |Not kept
+
+|`file`, `ftp`, `sftp`, `ftps`
+|Without `moveFailed`: the rollback leaves the file (with `noop=true` it
removes it from the idempotent repository),
+and the next poll reads it again.
+|With `moveFailed`.
+
+|`jms`, `activemq`, `amqp`
+|`transacted=true`, or `acknowledgementModeName=CLIENT_ACKNOWLEDGE` with
`asyncConsumer=false` (the default): the
+listener throws, and the session rolls back or recovers.
+|Otherwise.
+
+|`kafka`
+|`breakOnFirstError=true`: the failed offset is not committed, and the
consumer reads the record again.
+|`breakOnFirstError=false` (the default).
+
+|Other consumers
+|
+|Always.
+|===
+
+Without an expectation the report never says "lost", and the verdict is PASS:
chaos makes exchanges fail on purpose.
+A rule can carry an expectation ("route `orders` completes 100 more messages
within 2m"), which also finds messages
Review Comment:
💡 **Expectation syntax not defined:** The rule syntax section defines
`target`, `fault`, `rate`, and `until` — but `expectation` is introduced here
without a formal definition. The example ("route `orders` completes 100 more
messages within 2m") shows the *concept* but a developer reading the Rule
Syntax section cannot build the parser. Consider adding an `expectation=`
parameter line to the rule syntax block, or at minimum a forward reference like
"(expectation syntax: see implementation PR)".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
