This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new a1b6899f0c7c CAMEL-26724: camel-chaos - assertions for chaos in a
test: the verdict, an expectation, and the faults of a rule (#29394)
a1b6899f0c7c is described below
commit a1b6899f0c7c5d09d72234e8cf0d09bdd7c31ae7
Author: Claus Ibsen <[email protected]>
AuthorDate: Sun Nov 8 09:06:12 2026 +0100
CAMEL-26724: camel-chaos - assertions for chaos in a test: the verdict, an
expectation, and the faults of a rule (#29394)
ChaosAssertions turns a finding of a chaos session into a regression test:
awaitFinalReport, assertVerdict, assertExpectationMet and assertFired throw an
AssertionError with the report, so they work with any test framework. The
camel-chaos page has a new section, Chaos in a test.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/docs/chaos.adoc | 47 +++++
components/camel-chaos/src/main/docs/chaos.adoc | 47 +++++
.../camel/component/chaos/ChaosAssertions.java | 194 +++++++++++++++++++++
.../camel/component/chaos/ChaosAssertionsTest.java | 111 ++++++++++++
design/chaos.adoc | 5 +
5 files changed, 404 insertions(+)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
index b98acd6eed9d..c9d0d3684c5a 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
@@ -869,6 +869,53 @@ exchange gets which draw: with several threads (concurrent
consumers, parallel p
so two runs with the same seed can give different reports. With one thread and
the same exchanges in the same order,
the same seed gives the same faults.
+== Chaos in a test
+
+A finding of a chaos session becomes a regression test: the same rules, the
same seed, and assertions on the report.
+`ChaosAssertions` throws an `AssertionError` with the report when it fails, so
it works with any test framework. Add
+`camel-chaos` as a test dependency, and switch chaos on in the test with
`camel.chaos.enabled=true`:
+
+[source,java]
+----
+class OrdersChaosTest extends CamelTestSupport {
+
+ @Override
+ public void configureContext(CamelContextConfiguration config) {
+ super.configureContext(config);
+ Properties props = new Properties();
+ props.put("camel.chaos.enabled", "true");
+ props.put("camel.chaos.seed", "42");
+ // every second call to the stock service fails: the redelivery must
recover it, and every order ships
+ props.put("camel.chaos.rules", "step=callStock error rate=every:2
expect=ship:4 within=10s");
+ config.withUseOverridePropertiesWithPropertiesComponent(props);
+ }
+
+ @Test
+ void theOrdersShipWhenTheStockServiceFails() {
+ for (int i = 0; i < 4; i++) {
+ template.sendBody("direct:orders", "order " + i);
+ }
+ ChaosAssertions.assertFired(context, "r1", 2);
+ ChaosAssertions.assertExpectationMet(context, "r1",
Duration.ofSeconds(10));
+ ChaosAssertions.assertVerdict(context, ChaosReport.PASS,
Duration.ofSeconds(10));
+ }
+}
+----
+
+* `assertVerdict(context, verdict, timeout)` waits until the report is final
(no expectation pending), and checks the
+verdict; `assertVerdict(context, verdict)` checks it now, and fails when the
report is not final yet.
+* `assertExpectationMet(context, ruleId, timeout)` waits until the expectation
of a rule is decided. An expectation of
+at least so many exchanges is decided as soon as they completed; one of at
most so many (`expect=orders:handled<=0`)
+only when its `within` time is up, so keep that time short in a test.
+* `assertFired(context, ruleId, atLeast)` checks that a rule fired at least so
many faults: a rule that fires 0 times
+is often a typo in its target.
+* `awaitFinalReport(context, timeout)` returns the final report, for other
checks (`getLost()`, `toJson()`).
+
+The rules can also come from a scenario
(`camel.chaos.scenario=classpath:chaos/outage.chaos`), or be added in the test
+with `ChaosManager.get(context).addRule(...)`. With `camel-test-main-junit6`,
set the same properties in the test
+configuration. In a deterministic simulation test (see
<<_time_and_the_seed,Time and the seed>>), the times of the
+rules run in virtual time, and the report is read with `ChaosManager.report()`
when the simulation is done.
+
== The model of the rules
The rules are also described as data, for the tools that help to write them (a
form in a terminal UI, an AI that turns
diff --git a/components/camel-chaos/src/main/docs/chaos.adoc
b/components/camel-chaos/src/main/docs/chaos.adoc
index b98acd6eed9d..c9d0d3684c5a 100644
--- a/components/camel-chaos/src/main/docs/chaos.adoc
+++ b/components/camel-chaos/src/main/docs/chaos.adoc
@@ -869,6 +869,53 @@ exchange gets which draw: with several threads (concurrent
consumers, parallel p
so two runs with the same seed can give different reports. With one thread and
the same exchanges in the same order,
the same seed gives the same faults.
+== Chaos in a test
+
+A finding of a chaos session becomes a regression test: the same rules, the
same seed, and assertions on the report.
+`ChaosAssertions` throws an `AssertionError` with the report when it fails, so
it works with any test framework. Add
+`camel-chaos` as a test dependency, and switch chaos on in the test with
`camel.chaos.enabled=true`:
+
+[source,java]
+----
+class OrdersChaosTest extends CamelTestSupport {
+
+ @Override
+ public void configureContext(CamelContextConfiguration config) {
+ super.configureContext(config);
+ Properties props = new Properties();
+ props.put("camel.chaos.enabled", "true");
+ props.put("camel.chaos.seed", "42");
+ // every second call to the stock service fails: the redelivery must
recover it, and every order ships
+ props.put("camel.chaos.rules", "step=callStock error rate=every:2
expect=ship:4 within=10s");
+ config.withUseOverridePropertiesWithPropertiesComponent(props);
+ }
+
+ @Test
+ void theOrdersShipWhenTheStockServiceFails() {
+ for (int i = 0; i < 4; i++) {
+ template.sendBody("direct:orders", "order " + i);
+ }
+ ChaosAssertions.assertFired(context, "r1", 2);
+ ChaosAssertions.assertExpectationMet(context, "r1",
Duration.ofSeconds(10));
+ ChaosAssertions.assertVerdict(context, ChaosReport.PASS,
Duration.ofSeconds(10));
+ }
+}
+----
+
+* `assertVerdict(context, verdict, timeout)` waits until the report is final
(no expectation pending), and checks the
+verdict; `assertVerdict(context, verdict)` checks it now, and fails when the
report is not final yet.
+* `assertExpectationMet(context, ruleId, timeout)` waits until the expectation
of a rule is decided. An expectation of
+at least so many exchanges is decided as soon as they completed; one of at
most so many (`expect=orders:handled<=0`)
+only when its `within` time is up, so keep that time short in a test.
+* `assertFired(context, ruleId, atLeast)` checks that a rule fired at least so
many faults: a rule that fires 0 times
+is often a typo in its target.
+* `awaitFinalReport(context, timeout)` returns the final report, for other
checks (`getLost()`, `toJson()`).
+
+The rules can also come from a scenario
(`camel.chaos.scenario=classpath:chaos/outage.chaos`), or be added in the test
+with `ChaosManager.get(context).addRule(...)`. With `camel-test-main-junit6`,
set the same properties in the test
+configuration. In a deterministic simulation test (see
<<_time_and_the_seed,Time and the seed>>), the times of the
+rules run in virtual time, and the report is read with `ChaosManager.report()`
when the simulation is done.
+
== The model of the rules
The rules are also described as data, for the tools that help to write them (a
form in a terminal UI, an AI that turns
diff --git
a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAssertions.java
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAssertions.java
new file mode 100644
index 000000000000..d050233a8d50
--- /dev/null
+++
b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAssertions.java
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.chaos;
+
+import java.time.Duration;
+import java.util.Objects;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.util.json.JsonArray;
+import org.apache.camel.util.json.JsonObject;
+
+/**
+ * Assertions for chaos in a test (CAMEL-26724): a finding of a chaos session
becomes a regression test. They throw an
+ * {@link AssertionError} with the report, so they work with any test
framework.
+ * <p/>
+ * Chaos is switched on in a test with {@code camel.chaos.enabled=true} (and
{@code camel.chaos.seed} for the same
+ * faults each time), with the rules of {@code camel.chaos.rules} or {@code
camel.chaos.scenario}, or added with
+ * {@link ChaosManager#addRule(String)}.
+ *
+ * <pre>
+ * ChaosAssertions.assertVerdict(context, ChaosReport.PASS,
Duration.ofSeconds(10));
+ * ChaosAssertions.assertExpectationMet(context, "r1", Duration.ofSeconds(10));
+ * ChaosAssertions.assertFired(context, "r1", 3);
+ * </pre>
+ *
+ * @since 4.24
+ */
+public final class ChaosAssertions {
+
+ // how often a report is read while waiting for it to be final
+ private static final long POLL_MILLIS = 50;
+
+ private ChaosAssertions() {
+ }
+
+ /**
+ * Waits until the report is final, which is when no expectation is
pending any more (an expectation of at least so
+ * many exchanges is decided when they completed; one of at most so many,
when its time is up).
+ *
+ * @param camelContext the context, with chaos switched on
+ * @param timeout how long to wait at most
+ * @return the final report
+ * @throws AssertionError when chaos is off, or the report is still
pending after the timeout
+ */
+ public static ChaosReport awaitFinalReport(CamelContext camelContext,
Duration timeout) {
+ ChaosManager chaos = manager(camelContext);
+ long deadline = System.nanoTime() + timeout.toNanos();
+ ChaosReport report = chaos.report();
+ while (!report.isFinal()) {
+ if (System.nanoTime() - deadline >= 0) {
+ throw new AssertionError(
+ "The chaos report is still pending after " + timeout +
": an expectation is not decided yet\n"
+ + report.toText());
+ }
+ try {
+ Thread.sleep(POLL_MILLIS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new AssertionError("Interrupted while waiting for the
chaos report to be final", e);
+ }
+ report = chaos.report();
+ }
+ return report;
+ }
+
+ /**
+ * Asserts the verdict of the report as it is now, which must be final.
+ *
+ * @param camelContext the context, with chaos switched on
+ * @param verdict {@link ChaosReport#PASS} or {@link ChaosReport#FAIL}
+ */
+ public static void assertVerdict(CamelContext camelContext, String
verdict) {
+ ChaosReport report = manager(camelContext).report();
+ if (!report.isFinal()) {
+ throw new AssertionError("The chaos report is not final: an
expectation is pending\n" + report.toText());
+ }
+ checkVerdict(report, verdict);
+ }
+
+ /**
+ * Waits until the report is final, and asserts its verdict.
+ *
+ * @param camelContext the context, with chaos switched on
+ * @param verdict {@link ChaosReport#PASS} or {@link ChaosReport#FAIL}
+ * @param timeout how long to wait at most for the report to be final
+ */
+ public static void assertVerdict(CamelContext camelContext, String
verdict, Duration timeout) {
+ checkVerdict(awaitFinalReport(camelContext, timeout), verdict);
+ }
+
+ /**
+ * Waits until the expectation of a rule is decided, and asserts that it
is met.
+ *
+ * @param camelContext the context, with chaos switched on
+ * @param ruleId the id of the rule with the expectation, such as
{@code r1}
+ * @param timeout how long to wait at most for the expectation to be
decided
+ */
+ public static void assertExpectationMet(CamelContext camelContext, String
ruleId, Duration timeout) {
+ ChaosManager chaos = manager(camelContext);
+ long deadline = System.nanoTime() + timeout.toNanos();
+ while (true) {
+ ChaosReport report = chaos.report();
+ JsonObject rule = rule(report, ruleId);
+ if (!(rule.get("expectation") instanceof JsonObject expectation)) {
+ throw new AssertionError("Chaos rule " + ruleId + " has no
expectation (expect=)\n" + report.toText());
+ }
+ String outcome = expectation.getString("outcome");
+ if ("met".equals(outcome)) {
+ return;
+ }
+ if (!"pending".equals(outcome) || System.nanoTime() - deadline >=
0) {
+ throw new AssertionError(
+ "The expectation of chaos rule " + ruleId + " is " +
outcome
+ + ("pending".equals(outcome) ? "
after " + timeout : "") + ", not met\n"
+ + report.toText());
+ }
+ try {
+ Thread.sleep(POLL_MILLIS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new AssertionError("Interrupted while waiting for the
expectation of chaos rule " + ruleId, e);
+ }
+ }
+ }
+
+ /**
+ * Asserts that a rule fired at least so many faults, as the report says
now: it does not wait, so send the messages
+ * first (a fault fires as a message is sent).
+ *
+ * @param camelContext the context, with chaos switched on
+ * @param ruleId the id of the rule, such as {@code r1}
+ * @param atLeast the least number of faults
+ */
+ public static void assertFired(CamelContext camelContext, String ruleId,
long atLeast) {
+ ChaosReport report = manager(camelContext).report();
+ JsonObject rule = rule(report, ruleId);
+ long fired = 0;
+ if (rule.get("fired") instanceof JsonObject perHook) {
+ for (Object count : perHook.values()) {
+ fired += count instanceof Number n ? n.longValue() : 0;
+ }
+ }
+ if (fired < atLeast) {
+ throw new AssertionError(
+ "Chaos rule " + ruleId + " fired " + fired + (fired == 1 ?
" fault" : " faults") + ", expected at least "
+ + atLeast + "\n" + report.toText());
+ }
+ }
+
+ private static void checkVerdict(ChaosReport report, String verdict) {
+ if (!Objects.equals(verdict, report.getVerdict())) {
+ throw new AssertionError(
+ "The chaos verdict is " + report.getVerdict() + ",
expected " + verdict
+ + (report.getReasons().isEmpty() ? "" :
": " + String.join("; ", report.getReasons()))
+ + "\n" + report.toText());
+ }
+ }
+
+ /** The latest version of a rule in the report. */
+ private static JsonObject rule(ChaosReport report, String ruleId) {
+ // the report lists the versions of an id from the latest
+ if (report.toJson().get("rules") instanceof JsonArray rules) {
+ for (Object o : rules) {
+ if (o instanceof JsonObject rule &&
ruleId.equals(rule.getString("id"))) {
+ return rule;
+ }
+ }
+ }
+ throw new AssertionError("There is no chaos rule " + ruleId + "\n" +
report.toText());
+ }
+
+ private static ChaosManager manager(CamelContext camelContext) {
+ ChaosManager chaos = ChaosManager.get(camelContext);
+ if (chaos == null) {
+ throw new AssertionError(
+ "Chaos is off in this CamelContext: switch it on with
camel.chaos.enabled=true (and add camel-chaos)");
+ }
+ return chaos;
+ }
+}
diff --git
a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAssertionsTest.java
b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAssertionsTest.java
new file mode 100644
index 000000000000..4544a997c206
--- /dev/null
+++
b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAssertionsTest.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.chaos;
+
+import java.time.Duration;
+
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * The assertions for chaos in a test: the verdict, an expectation and the
faults of a rule, with the report when they
+ * fail.
+ */
+class ChaosAssertionsTest extends ChaosTestSupport {
+
+ private static final Duration TIMEOUT = Duration.ofSeconds(10);
+
+ @Override
+ protected RouteBuilder createRouteBuilder() {
+ return new RouteBuilder() {
+ @Override
+ public void configure() {
+ // the stock call is retried, so a fault there is recovered
and the order still ships
+
errorHandler(defaultErrorHandler().maximumRedeliveries(3).redeliveryDelay(0));
+ from("direct:orders").routeId("orders")
+ .to("mock:stock").id("callStock")
+ .to("direct:ship");
+ from("direct:ship").routeId("ship").to("mock:ship");
+ }
+ };
+ }
+
+ private void orders(int count) {
+ for (int i = 0; i < count; i++) {
+ template.sendBody("direct:orders", "order " + i);
+ }
+ }
+
+ @Test
+ void aRetriedFaultPassesAndTheExpectationIsMet() {
+ String id = chaos().addRule("step=callStock error rate=every:2
expect=ship:4 within=10s");
+ orders(4);
+
+ ChaosAssertions.assertFired(context, id, 2);
+ ChaosAssertions.assertExpectationMet(context, id, TIMEOUT);
+ ChaosAssertions.assertVerdict(context, ChaosReport.PASS, TIMEOUT);
+ ChaosAssertions.assertVerdict(context, ChaosReport.PASS);
+ }
+
+ @Test
+ void aLostMessageFailsWithTheReasonAndTheReport() {
+ String id = chaos().addRule("endpoint=direct:ship drop expect=ship:2
within=300ms");
+ orders(2);
+
+ assertThatThrownBy(() -> ChaosAssertions.assertExpectationMet(context,
id, TIMEOUT))
+ .isInstanceOf(AssertionError.class)
+ .hasMessageStartingWith("The expectation of chaos rule " + id
+ " is unmet, not met")
+ .hasMessageContaining("Chaos report: FAIL");
+ ChaosAssertions.assertVerdict(context, ChaosReport.FAIL, TIMEOUT);
+ assertThatThrownBy(() -> ChaosAssertions.assertVerdict(context,
ChaosReport.PASS))
+ .hasMessageStartingWith("The chaos verdict is FAIL, expected
PASS: route ship completed 0 of 2 exchanges");
+ }
+
+ @Test
+ void aPendingReportIsNotFinalInTime() {
+ String id = chaos().addRule("endpoint=direct:ship drop expect=ship:2
within=1m");
+
+ assertThatThrownBy(() -> ChaosAssertions.awaitFinalReport(context,
Duration.ofMillis(100)))
+ .hasMessageStartingWith("The chaos report is still pending
after PT0.1S");
+ assertThatThrownBy(() -> ChaosAssertions.assertVerdict(context,
ChaosReport.PASS))
+ .hasMessageStartingWith("The chaos report is not final");
+ assertThatThrownBy(() -> ChaosAssertions.assertExpectationMet(context,
id, Duration.ofMillis(100)))
+ .hasMessageStartingWith("The expectation of chaos rule " + id
+ " is pending after PT0.1S, not met");
+ }
+
+ @Test
+ void whatTheAssertionsCannotFind() throws Exception {
+ String id = chaos().addRule("step=callStock delay duration=1ms");
+ orders(1);
+
+ assertThatThrownBy(() -> ChaosAssertions.assertFired(context, id, 2))
+ .hasMessageStartingWith("Chaos rule " + id + " fired 1 fault,
expected at least 2");
+ assertThatThrownBy(() -> ChaosAssertions.assertExpectationMet(context,
id, TIMEOUT))
+ .hasMessageStartingWith("Chaos rule " + id + " has no
expectation (expect=)");
+ assertThatThrownBy(() -> ChaosAssertions.assertFired(context, "r99",
1))
+ .hasMessageStartingWith("There is no chaos rule r99");
+ try (DefaultCamelContext off = new DefaultCamelContext()) {
+ assertThatThrownBy(() -> ChaosAssertions.assertVerdict(off,
ChaosReport.PASS))
+ .hasMessageStartingWith("Chaos is off in this
CamelContext");
+ }
+ assertThat(ChaosAssertions.awaitFinalReport(context,
TIMEOUT).isFinal()).isTrue();
+ }
+}
diff --git a/design/chaos.adoc b/design/chaos.adoc
index 781f0796f15f..73a115855cd5 100644
--- a/design/chaos.adoc
+++ b/design/chaos.adoc
@@ -383,6 +383,11 @@ when the context stops first (the report of
`reportOnShutdown`) or the rule is r
arrive, so they are not called lost. A tool must treat a pending expectation
as not yet decided, so a PASS with one
pending is not final.
+In a test (stage 2, CAMEL-26724), `ChaosAssertions` of camel-chaos waits for
the report to be final and asserts the
+verdict, an expectation and the faults of a rule, throwing an `AssertionError`
with the report. It needs no test
+framework, so camel-test does not depend on camel-chaos; the rules come from
the properties (`camel.chaos.rules`,
+`camel.chaos.scenario`) or `ChaosManager`.
+
=== Control
- A `chaos` dev console lists the rules and shows the report, so it is
`@DevConsole(readOnly = true)`. It does not add,
