This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch CAMEL-26817-suggest-expectation in repository https://gitbox.apache.org/repos/asf/camel.git
commit 1d0a3c976c1bf35aef77ce5844f8788e1ffe3ffe Author: Claus Ibsen <[email protected]> AuthorDate: Tue Nov 10 17:02:56 2026 +0100 CAMEL-26817: camel-chaos - every suggested rule says what should hold: an expectation by default A suggestion without an expectation gets expect=<route>:failed<=0: every exchange its faults hit in the route is recovered by redelivery or handled by the error handler. An experiment needs an expectation, so every suggestion can now be run as one, which the start dialog of an exploration in the TUI will do (let there be chaos, CAMEL-26760). Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../org/apache/camel/catalog/docs/chaos.adoc | 17 +++++++++--- components/camel-chaos/src/main/docs/chaos.adoc | 17 +++++++++--- .../apache/camel/component/chaos/ChaosAdvisor.java | 16 +++++++++++ .../chaos/ChaosAdvisorDeadLetterTest.java | 2 +- .../camel/component/chaos/ChaosAdvisorTest.java | 31 +++++++++++++++++----- .../camel/component/chaos/ChaosPairsTest.java | 10 ++++--- .../camel/component/chaos/ChaosStubsTest.java | 5 ++-- .../modules/ROOT/pages/camel-jbang-managing.adoc | 2 +- 8 files changed, 79 insertions(+), 21 deletions(-) diff --git a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc index 3557980aed9a..3e67f0e62b80 100644 --- a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc +++ b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc @@ -211,7 +211,7 @@ Chaos rules to try, the most worth trying first: 1. route orders: Returns an error (HTTP status) (errors) an HTTP call to http://stock/check without a circuit breaker: what does the route do when the service answers 503? - camel cmd chaos-add --rule='endpoint=http://stock/check error status=503 rate=0.25 until=10m' + camel cmd chaos-add --rule='endpoint=http://stock/check error status=503 rate=0.25 until=10m expect=orders:failed<=0' 2. route orders: The route after a hand-off fails (lost) route orders hands the message to route shipping over seda and does not wait: when shipping fails, is the message lost? @@ -261,6 +261,16 @@ channel fails too. Are messages lost? The report counts what the dead letter cha does an exchange take? |=== +Every suggestion says what should hold despite its faults, as an expectation (see +<<_expectations_on_the_error_handler,Expectations on the error handler>>), so it can be run as an experiment of an +exploration as it is, and a rule added from it has a verdict that tells something. A hand-off expects that no message is +lost between the two routes, and a duplicate that the route receiving the call does no work twice. Every other +suggestion expects that no exchange fails in its route (`expect=orders:failed<=0`): whatever the fault does, each +exchange it hits is recovered by redelivery or handled by the error handler. A route without error handling for the +fault then fails the verdict, which is the finding, and a route with a dead letter channel, redelivery or a circuit +breaker with a fallback passes. Change the expectation when the route is meant to fail, such as a sender that should see +the error. + Two faults at once (kind `combined`) come after the single faults, as most bugs that lose messages need two failures that meet. Their rules are added together: @@ -268,10 +278,11 @@ that meet. Their rules are added together: ---- 5. route orders: An outage while the dead letter channel fails (combined) route orders sends what fails to the dead letter channel direct:parked: when http://payment/charge is down for a minute and the dead letter channel fails too, are messages lost? The report counts what the dead letter channel did not take as lost - camel cmd chaos-add --rule='endpoint=http://payment/charge error status=503 until=1m' --rule='endpoint=direct:parked error at=dlq until=1m' + camel cmd chaos-add --rule='endpoint=http://payment/charge error status=503 until=1m expect=orders:failed<=0' --rule='endpoint=direct:parked error at=dlq until=1m' ---- -Each of their rules counts as its own kind in what has been tried, so a pair is never "not tried yet". +Each of their rules counts as its own kind in what has been tried, so a pair is never "not tried yet". The expectation +is on the first rule, and the messages the dead letter channel did not take fail the verdict too. The rules leave the query of the endpoint out: its options do not tell which endpoint it is, and may hold a secret. `ChaosManager.suggest()` gives the same as JSON: `suggestions`, each with `route`, `kind`, `recipe`, `title`, `rule`, diff --git a/components/camel-chaos/src/main/docs/chaos.adoc b/components/camel-chaos/src/main/docs/chaos.adoc index 3557980aed9a..3e67f0e62b80 100644 --- a/components/camel-chaos/src/main/docs/chaos.adoc +++ b/components/camel-chaos/src/main/docs/chaos.adoc @@ -211,7 +211,7 @@ Chaos rules to try, the most worth trying first: 1. route orders: Returns an error (HTTP status) (errors) an HTTP call to http://stock/check without a circuit breaker: what does the route do when the service answers 503? - camel cmd chaos-add --rule='endpoint=http://stock/check error status=503 rate=0.25 until=10m' + camel cmd chaos-add --rule='endpoint=http://stock/check error status=503 rate=0.25 until=10m expect=orders:failed<=0' 2. route orders: The route after a hand-off fails (lost) route orders hands the message to route shipping over seda and does not wait: when shipping fails, is the message lost? @@ -261,6 +261,16 @@ channel fails too. Are messages lost? The report counts what the dead letter cha does an exchange take? |=== +Every suggestion says what should hold despite its faults, as an expectation (see +<<_expectations_on_the_error_handler,Expectations on the error handler>>), so it can be run as an experiment of an +exploration as it is, and a rule added from it has a verdict that tells something. A hand-off expects that no message is +lost between the two routes, and a duplicate that the route receiving the call does no work twice. Every other +suggestion expects that no exchange fails in its route (`expect=orders:failed<=0`): whatever the fault does, each +exchange it hits is recovered by redelivery or handled by the error handler. A route without error handling for the +fault then fails the verdict, which is the finding, and a route with a dead letter channel, redelivery or a circuit +breaker with a fallback passes. Change the expectation when the route is meant to fail, such as a sender that should see +the error. + Two faults at once (kind `combined`) come after the single faults, as most bugs that lose messages need two failures that meet. Their rules are added together: @@ -268,10 +278,11 @@ that meet. Their rules are added together: ---- 5. route orders: An outage while the dead letter channel fails (combined) route orders sends what fails to the dead letter channel direct:parked: when http://payment/charge is down for a minute and the dead letter channel fails too, are messages lost? The report counts what the dead letter channel did not take as lost - camel cmd chaos-add --rule='endpoint=http://payment/charge error status=503 until=1m' --rule='endpoint=direct:parked error at=dlq until=1m' + camel cmd chaos-add --rule='endpoint=http://payment/charge error status=503 until=1m expect=orders:failed<=0' --rule='endpoint=direct:parked error at=dlq until=1m' ---- -Each of their rules counts as its own kind in what has been tried, so a pair is never "not tried yet". +Each of their rules counts as its own kind in what has been tried, so a pair is never "not tried yet". The expectation +is on the first rule, and the messages the dead letter channel did not take fail the verdict too. The rules leave the query of the endpoint out: its options do not tell which endpoint it is, and may hold a secret. `ChaosManager.suggest()` gives the same as JSON: `suggestions`, each with `route`, `kind`, `recipe`, `title`, `rule`, diff --git a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAdvisor.java b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAdvisor.java index e93f2da975f1..39b1d66ec7a2 100644 --- a/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAdvisor.java +++ b/components/camel-chaos/src/main/java/org/apache/camel/component/chaos/ChaosAdvisor.java @@ -436,10 +436,26 @@ final class ChaosAdvisor { LOG.debug("No chaos suggestions for route {}: {}", route.getRouteId(), e.getMessage(), e); } } + answer.replaceAll(ChaosAdvisor::withExpectation); answer.sort(Comparator.comparingInt(Suggestion::rank).reversed()); return answer.size() > MAX ? List.copyOf(answer.subList(0, MAX)) : answer; } + /** + * The suggestion with what should hold despite its faults, when none of its rules says (CAMEL-26817): every + * exchange the faults hit in its route is recovered by redelivery or handled by the error handler, and none fails + * ({@code expect=orders:failed<=0}). An experiment needs an expectation, and a rule added without one passes + * whatever happens. For two faults at once the first rule has it. + */ + static Suggestion withExpectation(Suggestion s) { + if (s.rules().stream().anyMatch(r -> r.contains(" expect="))) { + return s; + } + return new Suggestion( + s.route(), s.kind(), s.recipe(), s.title(), s.rule() + " expect=" + s.route() + ":failed<=0", s.why(), + s.rank(), s.routes(), s.with()); + } + private static void suggest( Route route, Map<String, ChaosModel.Recipe> recipes, Map<String, String> consumers, Map<String, List<String>> deadLetters, List<Suggestion> answer) { diff --git a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorDeadLetterTest.java b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorDeadLetterTest.java index 10269380c7c2..a83ed8644689 100644 --- a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorDeadLetterTest.java +++ b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorDeadLetterTest.java @@ -59,7 +59,7 @@ class ChaosAdvisorDeadLetterTest extends ChaosTestSupport { assertThat(s.route()).isEqualTo("orders"); // the route that consumes the dead letter channel is not one of its routes assertThat(s.routes()).containsExactly("orders", "billing", "audit"); - assertThat(s.rule()).isEqualTo("endpoint=seda:dlq error at=dlq"); + assertThat(s.rule()).isEqualTo("endpoint=seda:dlq error at=dlq expect=orders:failed<=0"); assertThat(s.why()).startsWith("routes orders, billing, audit send what fails to the dead letter channel seda:dlq"); } diff --git a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorTest.java b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorTest.java index 14a6fd12526c..4df6e1cf6776 100644 --- a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorTest.java +++ b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosAdvisorTest.java @@ -74,17 +74,22 @@ class ChaosAdvisorTest extends ChaosTestSupport { Map<String, String> rules = rules(suggestions); assertThat(rules) - .containsEntry("orders http-error errors", "endpoint=http://stock/check error status=503 rate=0.25 until=10m") + .containsEntry("orders http-error errors", + "endpoint=http://stock/check error status=503 rate=0.25 until=10m expect=orders:failed<=0") .containsEntry("orders timeout slow", - "endpoint=http://stock/check error after=5s exception=java.net.SocketTimeoutException rate=0.1 until=10m") + "endpoint=http://stock/check error after=5s exception=java.net.SocketTimeoutException rate=0.1 until=10m" + + " expect=orders:failed<=0") // a hand-off over seda: the next route fails, and nothing may be lost between the two .containsEntry("orders route-error lost", "route=shipping error rate=0.2 until=2m expect=orders->shipping:lost<=0 within=3m") - .containsEntry("shipping dlq-unreachable dead letter channel", "endpoint=seda:dlq error at=dlq") + .containsEntry("shipping dlq-unreachable dead letter channel", + "endpoint=seda:dlq error at=dlq expect=shipping:failed<=0") // a circuit breaker: the service is down for a while - .containsEntry("pay outage errors", "endpoint=http://pay/charge error rate=burst:10s/1m until=10m") - .containsEntry("inbox poll-error consumer", "endpoint=testpoll://inbox error at=poll rate=0.5 until=10m") - .containsEntry("inbox stall lifecycle", "route=inbox stall duration=20s"); + .containsEntry("pay outage errors", + "endpoint=http://pay/charge error rate=burst:10s/1m until=10m expect=pay:failed<=0") + .containsEntry("inbox poll-error consumer", + "endpoint=testpoll://inbox error at=poll rate=0.5 until=10m expect=inbox:failed<=0") + .containsEntry("inbox stall lifecycle", "route=inbox stall duration=20s expect=inbox:failed<=0"); // no HTTP error without a breaker for the call inside the breaker, and the query is left out assertThat(rules).doesNotContainKey("pay http-error errors"); assertThat(rules.values()).noneMatch(r -> r.contains("throwExceptionOnFailure")); @@ -93,7 +98,8 @@ class ChaosAdvisorTest extends ChaosTestSupport { // a local file has no connection to refuse, but the write can fail assertThat(rules) .containsEntry("archive write-error errors", - "endpoint=file://target/chaos-archive error exception=java.io.IOException rate=0.25 until=10m") + "endpoint=file://target/chaos-archive error exception=java.io.IOException rate=0.25 until=10m" + + " expect=archive:failed<=0") .doesNotContainKeys("archive connection-refused errors", "archive slow slow"); // the most worth trying first @@ -112,6 +118,17 @@ class ChaosAdvisorTest extends ChaosTestSupport { } } + @Test + void everySuggestionSaysWhatShouldHold() { + for (ChaosAdvisor.Suggestion s : ChaosAdvisor.suggest(context)) { + // one expectation, so the suggestion can be run as an experiment + assertThat(s.rules().stream().filter(r -> r.contains(" expect=")).count()).as(s.title()).isEqualTo(1); + } + // a suggestion with an expectation of its own keeps it + assertThat(rules(ChaosAdvisor.suggest(context)).get("orders route-error lost")) + .endsWith("expect=orders->shipping:lost<=0 within=3m"); + } + @Test void theManagerGivesThemAsJson() { JsonArray list = (JsonArray) chaos().suggest().get("suggestions"); diff --git a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosPairsTest.java b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosPairsTest.java index 537276989892..a81bef525b2b 100644 --- a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosPairsTest.java +++ b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosPairsTest.java @@ -60,8 +60,9 @@ class ChaosPairsTest extends ChaosTestSupport { ChaosAdvisor.Suggestion s = pair("An outage while the dead letter channel fails"); assertThat(s.route()).isEqualTo("orders"); assertThat(s.kind()).isEqualTo(ChaosAdvisor.Kind.COMBINED); + // what should hold is on the first rule assertThat(s.rules()).containsExactly( - "endpoint=http://payment/charge error status=503 until=1m", + "endpoint=http://payment/charge error status=503 until=1m expect=orders:failed<=0", "endpoint=seda:parked error at=dlq until=1m"); assertThat(s.why()).contains("dead letter channel seda:parked", "are messages lost?"); } @@ -70,7 +71,7 @@ class ChaosPairsTest extends ChaosTestSupport { void aSlowServiceThatAlsoFails() { ChaosAdvisor.Suggestion s = pair("A slow service that also fails"); assertThat(s.rules()).containsExactly( - "endpoint=http://payment/charge delay duration=1s..3s rate=0.5 until=2m", + "endpoint=http://payment/charge delay duration=1s..3s rate=0.5 until=2m expect=orders:failed<=0", "endpoint=http://payment/charge error status=503 rate=0.2 until=2m"); assertThat(s.why()).contains("expect=orders:maxTime<=30s"); } @@ -96,10 +97,11 @@ class ChaosPairsTest extends ChaosTestSupport { .filter(s -> "An outage while the dead letter channel fails".equals(s.getString("title"))) .findFirst().orElseThrow(); assertThat(pair.getString("kind")).isEqualTo("combined"); - assertThat(pair.getString("rule")).isEqualTo("endpoint=http://payment/charge error status=503 until=1m"); + assertThat(pair.getString("rule")) + .isEqualTo("endpoint=http://payment/charge error status=503 until=1m expect=orders:failed<=0"); List<Object> rules = pair.getCollection("rules"); assertThat(rules).containsExactly( - "endpoint=http://payment/charge error status=503 until=1m", + "endpoint=http://payment/charge error status=503 until=1m expect=orders:failed<=0", "endpoint=seda:parked error at=dlq until=1m"); assertThat(pair.containsKey("tried")).isFalse(); // a single suggestion has no rules list diff --git a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosStubsTest.java b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosStubsTest.java index 57d8353dbeb7..ff8ac835e6ad 100644 --- a/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosStubsTest.java +++ b/components/camel-chaos/src/test/java/org/apache/camel/component/chaos/ChaosStubsTest.java @@ -88,9 +88,10 @@ class ChaosStubsTest extends ChaosTestSupport { void theSuggestionsSeeTheStubbedRemoteEndpoints() { Map<String, String> rules = rules(ChaosAdvisor.suggest(context)); assertThat(rules) - .containsEntry("mq route-error consumer", "route=mq error rate=0.05 until=10m") + .containsEntry("mq route-error consumer", "route=mq error rate=0.05 until=10m expect=mq:failed<=0") .containsEntry("mq connection-refused errors", - "endpoint=remotemq://out error exception=java.net.ConnectException rate=0.25 until=10m"); + "endpoint=remotemq://out error exception=java.net.ConnectException rate=0.25 until=10m" + + " expect=mq:failed<=0"); // the stub of audit stands in for nothing, so there is nothing remote to try in route plain assertThat(rules.keySet()).noneMatch(k -> k.startsWith("plain ")); } diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc b/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc index 625b11255da9..0b120ab261b6 100644 --- a/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc +++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc @@ -907,7 +907,7 @@ and with `--dry-run` it adds nothing, and prints what the rule would hit now: th their route. `camel cmd chaos-suggest` suggests rules to try per route from what the routes do (an HTTP call without a circuit breaker, a dead letter channel, a hand-off over seda, an input from outside), the most worth trying first, each with -why and the `chaos-add` command that adds it. +why and the `chaos-add` command that adds it, with an expectation of what should hold despite the faults. `camel get chaos` lists the current rules with their id, version, state, matches and the faults that fired, and `--report` shows the report: the verdict, the seed, every version of every rule, the counts per route, and what has been tried per route (the kinds of fault, how it went, and what the suggestions offer that no rule tried yet). Chaos is for
