This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch CAMEL-26724-chaos-assertions in repository https://gitbox.apache.org/repos/asf/camel.git
commit 78efe375f05cc7aff4eaa1e824ce08a3eee20736 Author: Claus Ibsen <[email protected]> AuthorDate: Sat Nov 7 22:58:51 2026 +0100 CAMEL-26724: camel-chaos - assertions for chaos in a test: the verdict, an expectation, and how often a rule fired ChaosAssertions waits until the report is final and asserts the verdict (assertVerdict), the expectation of a rule (assertExpectationMet) and its faults (assertFired); awaitFinalReport returns the final report. They throw an AssertionError with the report, so they need no test framework, and camel-test does not depend on camel-chaos. The chaos page has a test with CamelTestSupport. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../org/apache/camel/catalog/docs/chaos.adoc | 47 +++++ components/camel-chaos/src/main/docs/chaos.adoc | 47 +++++ .../camel/component/chaos/ChaosAssertions.java | 192 +++++++++++++++++++++ .../camel/component/chaos/ChaosAssertionsTest.java | 111 ++++++++++++ design/chaos.adoc | 5 + 5 files changed, 402 insertions(+) diff --git a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc index ba00e70049de..7279edd946a3 100644 --- a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc +++ b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc @@ -770,6 +770,53 @@ exchange gets which draw: with several threads (concurrent consumers, parallel p so two runs with the same seed can give different reports. With one thread and the same exchanges in the same order, the same seed gives the same faults. +== Chaos in a test + +A finding of a chaos session becomes a regression test: the same rules, the same seed, and assertions on the report. +`ChaosAssertions` throws an `AssertionError` with the report when it fails, so it works with any test framework. Add +`camel-chaos` as a test dependency, and switch chaos on in the test with `camel.chaos.enabled=true`: + +[source,java] +---- +class OrdersChaosTest extends CamelTestSupport { + + @Override + public void configureContext(CamelContextConfiguration config) { + super.configureContext(config); + Properties props = new Properties(); + props.put("camel.chaos.enabled", "true"); + props.put("camel.chaos.seed", "42"); + // every second call to the stock service fails: the redelivery must recover it, and every order ships + props.put("camel.chaos.rules", "step=callStock error rate=every:2 expect=ship:4 within=10s"); + config.withUseOverridePropertiesWithPropertiesComponent(props); + } + + @Test + void theOrdersShipWhenTheStockServiceFails() { + for (int i = 0; i < 4; i++) { + template.sendBody("direct:orders", "order " + i); + } + ChaosAssertions.assertFired(context, "r1", 2); + ChaosAssertions.assertExpectationMet(context, "r1", Duration.ofSeconds(10)); + ChaosAssertions.assertVerdict(context, ChaosReport.PASS, Duration.ofSeconds(10)); + } +} +---- + +* `assertVerdict(context, verdict, timeout)` waits until the report is final (no expectation pending), and checks the +verdict; `assertVerdict(context, verdict)` checks it now, and fails when the report is not final yet. +* `assertExpectationMet(context, ruleId, timeout)` waits until the expectation of a rule is decided. An expectation of +at least so many exchanges is decided as soon as they completed; one of at most so many (`expect=orders:handled<=0`) +only when its `within` time is up, so keep that time short in a test. +* `assertFired(context, ruleId, atLeast)` checks that a rule fired at least so many faults: a rule that fires 0 times +is often a typo in its target. +* `awaitFinalReport(context, timeout)` returns the final report, for other checks (`getLost()`, `toJson()`). + +The rules can also come from a scenario (`camel.chaos.scenario=classpath:chaos/outage.chaos`), or be added in the test +with `ChaosManager.get(context).addRule(...)`. With `camel-test-main-junit6`, set the same properties in the test +configuration. In a deterministic simulation test (see <<_time_and_the_seed,Time and the seed>>), the times of the +rules run in virtual time, and the report is read with `ChaosManager.report()` when the simulation is done. + == The model of the rules The rules are also described as data, for the tools that help to write them (a form in a terminal UI, an AI that turns diff --git a/components/camel-chaos/src/main/docs/chaos.adoc b/components/camel-chaos/src/main/docs/chaos.adoc index ba00e70049de..7279edd946a3 100644 --- a/components/camel-chaos/src/main/docs/chaos.adoc +++ b/components/camel-chaos/src/main/docs/chaos.adoc @@ -770,6 +770,53 @@ exchange gets which draw: with several threads (concurrent consumers, parallel p so two runs with the same seed can give different reports. With one thread and the same exchanges in the same order, the same seed gives the same faults. +== Chaos in a test + +A finding of a chaos session becomes a regression test: the same rules, the same seed, and assertions on the report. +`ChaosAssertions` throws an `AssertionError` with the report when it fails, so it works with any test framework. Add +`camel-chaos` as a test dependency, and switch chaos on in the test with `camel.chaos.enabled=true`: + +[source,java] +---- +class OrdersChaosTest extends CamelTestSupport { + + @Override + public void configureContext(CamelContextConfiguration config) { + super.configureContext(config); + Properties props = new Properties(); + props.put("camel.chaos.enabled", "true"); + props.put("camel.chaos.seed", "42"); + // every second call to the stock service fails: the redelivery must recover it, and every order ships + props.put("camel.chaos.rules", "step=callStock error rate=every:2 expect=ship:4 within=10s"); + config.withUseOverridePropertiesWithPropertiesComponent(props); + } + + @Test + void theOrdersShipWhenTheStockServiceFails() { + for (int i = 0; i < 4; i++) { + template.sendBody("direct:orders", "order " + i); + } + ChaosAssertions.assertFired(context, "r1", 2); + ChaosAssertions.assertExpectationMet(context, "r1", Duration.ofSeconds(10)); + ChaosAssertions.assertVerdict(context, ChaosReport.PASS, Duration.ofSeconds(10)); + } +} +---- + +* `assertVerdict(context, verdict, timeout)` waits until the report is final (no expectation pending), and checks the +verdict; `assertVerdict(context, verdict)` checks it now, and fails when the report is not final yet. +* `assertExpectationMet(context, ruleId, timeout)` waits until the expectation of a rule is decided. An expectation of +at least so many exchanges is decided as soon as they completed; one of at most so many (`expect=orders:handled<=0`) +only when its `within` time is up, so keep that time short in a test. +* `assertFired(context, ruleId, atLeast)` checks that a rule fired at least so many faults: a rule that fires 0 times +is often a typo in its target. +* `awaitFinalReport(context, timeout)` returns the final report, for other checks (`getLost()`, `toJson()`). + +The rules can also come from a scenario (`camel.chaos.scenario=classpath:chaos/outage.chaos`), or be added in the test +with `ChaosManager.get(context).addRule(...)`. With `camel-test-main-junit6`, set the same properties in the test +configuration. In a deterministic simulation test (see <<_time_and_the_seed,Time and the seed>>), the times of the +rules run in virtual time, and the report is read with `ChaosManager.report()` when the simulation is done. + == The model of the rules The rules are also described as data, for the tools that help to write them (a form in a terminal UI, an AI that turns diff --git a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAssertions.java b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAssertions.java new file mode 100644 index 000000000000..35e507b186fa --- /dev/null +++ b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAssertions.java @@ -0,0 +1,192 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.component.chaos; + +import java.time.Duration; + +import org.apache.camel.CamelContext; +import org.apache.camel.util.json.JsonArray; +import org.apache.camel.util.json.JsonObject; + +/** + * Assertions for chaos in a test (CAMEL-26724): a finding of a chaos session becomes a regression test. They throw an + * {@link AssertionError} with the report, so they work with any test framework. + * <p/> + * Chaos is switched on in a test with {@code camel.chaos.enabled=true} (and {@code camel.chaos.seed} for the same + * faults each time), with the rules of {@code camel.chaos.rules} or {@code camel.chaos.scenario}, or added with + * {@link ChaosManager#addRule(String)}. + * + * <pre> + * ChaosAssertions.assertVerdict(context, ChaosReport.PASS, Duration.ofSeconds(10)); + * ChaosAssertions.assertExpectationMet(context, "r1", Duration.ofSeconds(10)); + * ChaosAssertions.assertFired(context, "r1", 3); + * </pre> + * + * @since 4.24 + */ +public final class ChaosAssertions { + + // how often a report is read while waiting for it to be final + private static final long POLL_MILLIS = 50; + + private ChaosAssertions() { + } + + /** + * Waits until the report is final, which is when no expectation is pending any more (an expectation of at least so + * many exchanges is decided when they completed; one of at most so many, when its time is up). + * + * @param camelContext the context, with chaos switched on + * @param timeout how long to wait at most + * @return the final report + * @throws AssertionError when chaos is off, or the report is still pending after the timeout + */ + public static ChaosReport awaitFinalReport(CamelContext camelContext, Duration timeout) { + ChaosManager chaos = manager(camelContext); + long deadline = System.nanoTime() + timeout.toNanos(); + ChaosReport report = chaos.report(); + while (!report.isFinal()) { + if (System.nanoTime() - deadline >= 0) { + throw new AssertionError( + "The chaos report is still pending after " + timeout + ": an expectation is not decided yet\n" + + report.toText()); + } + try { + Thread.sleep(POLL_MILLIS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("Interrupted while waiting for the chaos report to be final", e); + } + report = chaos.report(); + } + return report; + } + + /** + * Asserts the verdict of the report as it is now, which must be final. + * + * @param camelContext the context, with chaos switched on + * @param verdict {@link ChaosReport#PASS} or {@link ChaosReport#FAIL} + */ + public static void assertVerdict(CamelContext camelContext, String verdict) { + ChaosReport report = manager(camelContext).report(); + if (!report.isFinal()) { + throw new AssertionError("The chaos report is not final: an expectation is pending\n" + report.toText()); + } + checkVerdict(report, verdict); + } + + /** + * Waits until the report is final, and asserts its verdict. + * + * @param camelContext the context, with chaos switched on + * @param verdict {@link ChaosReport#PASS} or {@link ChaosReport#FAIL} + * @param timeout how long to wait at most for the report to be final + */ + public static void assertVerdict(CamelContext camelContext, String verdict, Duration timeout) { + checkVerdict(awaitFinalReport(camelContext, timeout), verdict); + } + + /** + * Waits until the expectation of a rule is decided, and asserts that it is met. + * + * @param camelContext the context, with chaos switched on + * @param ruleId the id of the rule with the expectation, such as {@code r1} + * @param timeout how long to wait at most for the expectation to be decided + */ + public static void assertExpectationMet(CamelContext camelContext, String ruleId, Duration timeout) { + ChaosManager chaos = manager(camelContext); + long deadline = System.nanoTime() + timeout.toNanos(); + while (true) { + ChaosReport report = chaos.report(); + JsonObject rule = rule(report, ruleId); + JsonObject expectation = (JsonObject) rule.get("expectation"); + if (expectation == null) { + throw new AssertionError("Chaos rule " + ruleId + " has no expectation (expect=)\n" + report.toText()); + } + String outcome = expectation.getString("outcome"); + if ("met".equals(outcome)) { + return; + } + if (!"pending".equals(outcome) || System.nanoTime() - deadline >= 0) { + throw new AssertionError( + "The expectation of chaos rule " + ruleId + " is " + outcome + + ("pending".equals(outcome) ? " after " + timeout : "") + ", not met\n" + + report.toText()); + } + try { + Thread.sleep(POLL_MILLIS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("Interrupted while waiting for the expectation of chaos rule " + ruleId, e); + } + } + } + + /** + * Asserts that a rule fired at least so many faults. + * + * @param camelContext the context, with chaos switched on + * @param ruleId the id of the rule, such as {@code r1} + * @param atLeast the least number of faults + */ + public static void assertFired(CamelContext camelContext, String ruleId, long atLeast) { + ChaosReport report = manager(camelContext).report(); + JsonObject rule = rule(report, ruleId); + long fired = 0; + if (rule.get("fired") instanceof JsonObject perHook) { + for (Object count : perHook.values()) { + fired += count instanceof Number n ? n.longValue() : 0; + } + } + if (fired < atLeast) { + throw new AssertionError( + "Chaos rule " + ruleId + " fired " + fired + (fired == 1 ? " fault" : " faults") + ", expected at least " + + atLeast + "\n" + report.toText()); + } + } + + private static void checkVerdict(ChaosReport report, String verdict) { + if (!report.getVerdict().equals(verdict)) { + throw new AssertionError( + "The chaos verdict is " + report.getVerdict() + ", expected " + verdict + + (report.getReasons().isEmpty() ? "" : ": " + String.join("; ", report.getReasons())) + + "\n" + report.toText()); + } + } + + /** The latest version of a rule in the report. */ + private static JsonObject rule(ChaosReport report, String ruleId) { + // the report lists the versions of an id from the latest + for (Object o : (JsonArray) report.toJson().get("rules")) { + JsonObject rule = (JsonObject) o; + if (ruleId.equals(rule.getString("id"))) { + return rule; + } + } + throw new AssertionError("There is no chaos rule " + ruleId + "\n" + report.toText()); + } + + private static ChaosManager manager(CamelContext camelContext) { + ChaosManager chaos = ChaosManager.get(camelContext); + if (chaos == null) { + throw new AssertionError( + "Chaos is off in this CamelContext: switch it on with camel.chaos.enabled=true (and add camel-chaos)"); + } + return chaos; + } +} diff --git a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAssertionsTest.java b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAssertionsTest.java new file mode 100644 index 000000000000..4544a997c206 --- /dev/null +++ b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAssertionsTest.java @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.component.chaos; + +import java.time.Duration; + +import org.apache.camel.builder.RouteBuilder; +import org.apache.camel.impl.DefaultCamelContext; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** + * The assertions for chaos in a test: the verdict, an expectation and the faults of a rule, with the report when they + * fail. + */ +class ChaosAssertionsTest extends ChaosTestSupport { + + private static final Duration TIMEOUT = Duration.ofSeconds(10); + + @Override + protected RouteBuilder createRouteBuilder() { + return new RouteBuilder() { + @Override + public void configure() { + // the stock call is retried, so a fault there is recovered and the order still ships + errorHandler(defaultErrorHandler().maximumRedeliveries(3).redeliveryDelay(0)); + from("direct:orders").routeId("orders") + .to("mock:stock").id("callStock") + .to("direct:ship"); + from("direct:ship").routeId("ship").to("mock:ship"); + } + }; + } + + private void orders(int count) { + for (int i = 0; i < count; i++) { + template.sendBody("direct:orders", "order " + i); + } + } + + @Test + void aRetriedFaultPassesAndTheExpectationIsMet() { + String id = chaos().addRule("step=callStock error rate=every:2 expect=ship:4 within=10s"); + orders(4); + + ChaosAssertions.assertFired(context, id, 2); + ChaosAssertions.assertExpectationMet(context, id, TIMEOUT); + ChaosAssertions.assertVerdict(context, ChaosReport.PASS, TIMEOUT); + ChaosAssertions.assertVerdict(context, ChaosReport.PASS); + } + + @Test + void aLostMessageFailsWithTheReasonAndTheReport() { + String id = chaos().addRule("endpoint=direct:ship drop expect=ship:2 within=300ms"); + orders(2); + + assertThatThrownBy(() -> ChaosAssertions.assertExpectationMet(context, id, TIMEOUT)) + .isInstanceOf(AssertionError.class) + .hasMessageStartingWith("The expectation of chaos rule " + id + " is unmet, not met") + .hasMessageContaining("Chaos report: FAIL"); + ChaosAssertions.assertVerdict(context, ChaosReport.FAIL, TIMEOUT); + assertThatThrownBy(() -> ChaosAssertions.assertVerdict(context, ChaosReport.PASS)) + .hasMessageStartingWith("The chaos verdict is FAIL, expected PASS: route ship completed 0 of 2 exchanges"); + } + + @Test + void aPendingReportIsNotFinalInTime() { + String id = chaos().addRule("endpoint=direct:ship drop expect=ship:2 within=1m"); + + assertThatThrownBy(() -> ChaosAssertions.awaitFinalReport(context, Duration.ofMillis(100))) + .hasMessageStartingWith("The chaos report is still pending after PT0.1S"); + assertThatThrownBy(() -> ChaosAssertions.assertVerdict(context, ChaosReport.PASS)) + .hasMessageStartingWith("The chaos report is not final"); + assertThatThrownBy(() -> ChaosAssertions.assertExpectationMet(context, id, Duration.ofMillis(100))) + .hasMessageStartingWith("The expectation of chaos rule " + id + " is pending after PT0.1S, not met"); + } + + @Test + void whatTheAssertionsCannotFind() throws Exception { + String id = chaos().addRule("step=callStock delay duration=1ms"); + orders(1); + + assertThatThrownBy(() -> ChaosAssertions.assertFired(context, id, 2)) + .hasMessageStartingWith("Chaos rule " + id + " fired 1 fault, expected at least 2"); + assertThatThrownBy(() -> ChaosAssertions.assertExpectationMet(context, id, TIMEOUT)) + .hasMessageStartingWith("Chaos rule " + id + " has no expectation (expect=)"); + assertThatThrownBy(() -> ChaosAssertions.assertFired(context, "r99", 1)) + .hasMessageStartingWith("There is no chaos rule r99"); + try (DefaultCamelContext off = new DefaultCamelContext()) { + assertThatThrownBy(() -> ChaosAssertions.assertVerdict(off, ChaosReport.PASS)) + .hasMessageStartingWith("Chaos is off in this CamelContext"); + } + assertThat(ChaosAssertions.awaitFinalReport(context, TIMEOUT).isFinal()).isTrue(); + } +} diff --git a/design/chaos.adoc b/design/chaos.adoc index 7f8d504d67a0..6ddea71ba9e3 100644 --- a/design/chaos.adoc +++ b/design/chaos.adoc @@ -383,6 +383,11 @@ when the context stops first (the report of `reportOnShutdown`) or the rule is r arrive, so they are not called lost. A tool must treat a pending expectation as not yet decided, so a PASS with one pending is not final. +In a test (stage 2, CAMEL-26724), `ChaosAssertions` of camel-chaos waits for the report to be final and asserts the +verdict, an expectation and the faults of a rule, throwing an `AssertionError` with the report. It needs no test +framework, so camel-test does not depend on camel-chaos; the rules come from the properties (`camel.chaos.rules`, +`camel.chaos.scenario`) or `ChaosManager`. + === Control - A `chaos` dev console lists the rules and shows the report, so it is `@DevConsole(readOnly = true)`. It does not add,
