This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new a12f29b0fa99 CAMEL-26788: camel-jbang - camel cmd chaos-explore saves
its report, and a replayable scenario per failed experiment, in test/chaos
(#29514)
a12f29b0fa99 is described below
commit a12f29b0fa99068fb4f14ebbf927e921c2d4c875
Author: Claus Ibsen <[email protected]>
AuthorDate: Tue Nov 10 00:55:28 2026 +0100
CAMEL-26788: camel-jbang - camel cmd chaos-explore saves its report, and a
replayable scenario per failed experiment, in test/chaos (#29514)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/docs/chaos.adoc | 3 +-
components/camel-chaos/src/main/docs/chaos.adoc | 3 +-
.../modules/ROOT/pages/camel-jbang-managing.adoc | 5 +-
.../camel-jbang-cmd-chaos-experiment.adoc | 2 +-
.../camel-jbang-cmd-chaos-explore.adoc | 1 +
.../apache/camel/cli/connector/ChaosAction.java | 2 +
.../LocalCliConnectorChaosActionTest.java | 8 +
.../META-INF/camel-jbang-commands-metadata.json | 2 +-
.../apache/camel/dsl/jbang/core/commands/Ask.java | 6 +
.../dsl/jbang/core/commands/ChaosExplore.java | 185 ++++++++++++++++++++-
.../action/CamelChaosExperimentAction.java | 11 +-
.../core/commands/ai/ChaosExplorePlaybook.java | 28 +++-
.../dsl/jbang/core/commands/ai/ToolRegistry.java | 47 +++---
.../dsl/jbang/core/common/ChaosExperimentRun.java | 73 ++++++++
.../dsl/jbang/core/commands/ChaosExploreTest.java | 81 ++++++++-
.../core/commands/action/CamelChaosActionTest.java | 7 +
.../jbang/core/common/ChaosExperimentRunTest.java | 70 +++++++-
.../src/test/resources/golden/cli-help.txt | 14 +-
.../dsl/jbang/core/commands/mcp/RuntimeTools.java | 7 +-
19 files changed, 499 insertions(+), 56 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
index 0c8b96325a9c..3a4f00f07c39 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
@@ -439,7 +439,8 @@ The Camel CLI runs an experiment in one call: `camel cmd
chaos-experiment` begin
`--body`, `--repeat` and `--interval`, as `camel cmd send`), waits until its
expectations are decided (at most
`--wait`), and ends it. `camel cmd chaos-explore-start` and
`chaos-explore-finish` start and finish the exploration.
`camel cmd chaos-explore` hands a whole exploration to an AI: it runs the
experiments by itself for the budget, and
-reports what held and what broke (the MCP prompt `camel_chaos_explore` is the
same playbook).
+reports what held and what broke (the MCP prompt `camel_chaos_explore` is the
same playbook). It saves the report in
+`test/chaos/reports`, and each experiment that failed as a scenario in
`test/chaos`, to replay it.
== Scenarios
diff --git a/components/camel-chaos/src/main/docs/chaos.adoc
b/components/camel-chaos/src/main/docs/chaos.adoc
index 0c8b96325a9c..3a4f00f07c39 100644
--- a/components/camel-chaos/src/main/docs/chaos.adoc
+++ b/components/camel-chaos/src/main/docs/chaos.adoc
@@ -439,7 +439,8 @@ The Camel CLI runs an experiment in one call: `camel cmd
chaos-experiment` begin
`--body`, `--repeat` and `--interval`, as `camel cmd send`), waits until its
expectations are decided (at most
`--wait`), and ends it. `camel cmd chaos-explore-start` and
`chaos-explore-finish` start and finish the exploration.
`camel cmd chaos-explore` hands a whole exploration to an AI: it runs the
experiments by itself for the budget, and
-reports what held and what broke (the MCP prompt `camel_chaos_explore` is the
same playbook).
+reports what held and what broke (the MCP prompt `camel_chaos_explore` is the
same playbook). It saves the report in
+`test/chaos/reports`, and each experiment that failed as a scenario in
`test/chaos`, to replay it.
== Scenarios
diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
b/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
index 4720ad63b47e..e744bbfe3775 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
@@ -954,7 +954,10 @@ camel cmd chaos-explore --budget=10m
camel cmd chaos-explore myApp --budget=1h --focus=refunds --api-type=anthropic
----
-Run the integration with `--stub=remote`, so the experiments do not reach real
systems. A route fed over HTTP or from
+It saves what it found in the project: the report and the journal in
`test/chaos/reports/explore-<time>.md` (and
+`.json`), and each experiment that failed as a scenario,
`test/chaos/explore-<time>-<n>.chaos`, which
+`camel run * --chaos=<file>` or `camel cmd chaos-load` replays (`--no-save`
leaves them out). Run the integration with
+`--stub=remote`, so the experiments do not reach real systems. A route fed
over HTTP or from
files cannot be fed by an experiment yet. MCP clients have the same playbook
as the prompt `camel_chaos_explore`. See
xref:next@components:others:chaos.adoc#_an_exploration[An exploration] in the
camel-chaos documentation.
diff --git
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
index f058c4f411d6..a8793619685d 100644
---
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
+++
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
@@ -29,7 +29,7 @@ camel cmd chaos-experiment [options]
| `--repeat` | How many messages to send in all (at most 10000) | | int
| `--rule` | A rule in its one-line text form, as an entry of
camel.chaos.rules; can be given more than once, and one of them needs an
expectation (expect=) | | List
| `--timeout` | How long to wait for the integration to answer, in millis or
as a duration such as 30s | 10000 | long
-| `--wait` | The longest wait for the verdict after the messages were sent (at
most 10m) | 2m | String
+| `--wait` | The longest wait for the verdict after the messages were sent,
and at least the within= of the expectations (at most 10m) | 2m | String
| `-h,--help` | Display the help and sub-commands | | boolean
|===
diff --git
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore.adoc
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore.adoc
index 8ddbf5b6a012..eb684864ce1a 100644
---
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore.adoc
+++
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore.adoc
@@ -25,6 +25,7 @@ camel cmd chaos-explore [options]
| `--focus` | What to explore, such as a route id (default: all routes) | |
String
| `--max-iterations` | Maximum number of tool-calling rounds | 100 | int
| `--model` | Model to use. Default: the provider's default; with Ollama the
installed model best at tool calling | | String
+| `--save` | Save the report in test/chaos/reports (as markdown, and the
journal as JSON), and each experiment that failed as a scenario in test/chaos,
to replay it | true | boolean
| `--show-tools` | Show the tool calls and results as they happen | | boolean
| `--timeout` | Timeout in seconds for one answer of the LLM (an experiment
can take minutes) | 300 | int
| `--tools` | The tools the model gets: core (a small set, for a local model),
full, or auto | auto | ToolSet
diff --git
a/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
b/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
index e313d421dca3..1d3cbfceac55 100644
---
a/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
+++
b/dsl/camel-cli-connector/src/main/java/org/apache/camel/cli/connector/ChaosAction.java
@@ -161,6 +161,8 @@ final class ChaosAction {
Map<String, Object> begun = (Map<String, Object>) call(type,
chaos, "beginExperiment",
root.getString("hypothesis"), rule);
answer.putAll(begun);
+ // a rule that matches nothing makes the experiment say
nothing: as with add, the reason goes along
+ warnings(type, chaos, (List<?>) begun.get("rules"), answer);
} else if ("experiment-end".equals(command)) {
// what came of it, and its rules are removed
@SuppressWarnings("unchecked")
diff --git
a/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
b/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
index eae9ee24a76d..1f41b807f294 100644
---
a/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
+++
b/dsl/camel-cli-connector/src/test/java/org/apache/camel/cli/connector/LocalCliConnectorChaosActionTest.java
@@ -278,6 +278,14 @@ class LocalCliConnectorChaosActionTest {
assertThat(ended.result().getString("verdict")).isEqualTo("FAIL");
assertThat(ended.result().getString("note")).isEqualTo("no error
handler");
assertThat(chaos("experiment-end", null, null).error()).isEqualTo("No
experiment is open");
+
+ // a rule that matches nothing (a typo) comes with its warning, as
with add
+ action.put("rule", "endpoint=file:/nowhere error
expect=start:failed<=0 within=1m");
+ CliActionOutcome typo = CliActionOutcome.dispatch(connector,
action).get(10, TimeUnit.SECONDS);
+ assertThat(typo.error()).isNull();
+ assertThat(typo.result().<List<Object>>
getCollection("warnings")).singleElement().asString()
+ .startsWith("r2: endpoint=file:/nowhere matches no");
+ assertThat(chaos("experiment-end", null, null).error()).isNull();
assertThat(chaos("experiment-begin", null,
null).error()).isEqualTo("The rules of the experiment are required");
}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
b/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
index b7bba9ca7fc9..f05a290e35fb 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
+++
b/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
@@ -3,7 +3,7 @@
{ "name": "ask", "fullName": "ask", "description": "Ask a question about a
running Camel application using AI", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.Ask", "options": [ { "names":
"--api-key", "description": "API key. Also reads ANTHROPIC_API_KEY,
OPENAI_API_KEY, WATSONX_APIKEY, or LLM_API_KEY env vars", "javaType":
"java.lang.String", "type": "string" }, { "names": "--api-type", "description":
"API type: 'ollama', 'openai', 'anthropic', or 'watsonx'", "javaType": [...]
{ "name": "bind", "fullName": "bind", "description": "DEPRECATED: Bind
source and sink Kamelets as a new Camel integration", "deprecated": true,
"sourceClass": "org.apache.camel.dsl.jbang.core.commands.bind.Bind", "options":
[ { "names": "--error-handler", "description": "Add error handler
(none|log|sink:<endpoint>). Sink endpoints are expected in the format
\"[[apigroup\/]version:]kind:[namespace\/]name\", plain Camel URIs or Kamelet
name.", "javaType": "java.lang.String", "type": " [...]
{ "name": "catalog", "fullName": "catalog", "description": "List artifacts
from Camel Catalog", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.catalog.CatalogCommand", "options": [
{ "names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name":
"component", "fullName": "catalog component", "description": "List components
from the Camel Catalog", "sourceClass": "org.apache.camel.dsl.jbang.co [...]
- { "name": "cmd", "fullName": "cmd", "description": "Performs commands in
the running Camel integrations, such as start\/stop route, or change logging
levels.", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.action.CamelAction", "options": [ {
"names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name":
"browse", "fullName": "cmd browse", "description": "Browse pending messages on
endpoints [...]
+ { "name": "cmd", "fullName": "cmd", "description": "Performs commands in
the running Camel integrations, such as start\/stop route, or change logging
levels.", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.action.CamelAction", "options": [ {
"names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name":
"browse", "fullName": "cmd browse", "description": "Browse pending messages on
endpoints [...]
{ "name": "completion", "fullName": "completion", "description": "Generate
completion script for bash\/zsh", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.Complete", "options": [ { "names":
"-h,--help", "description": "Display the help and sub-commands", "javaType":
"boolean", "type": "boolean" } ] },
{ "name": "config", "fullName": "config", "description": "Get and set user
configuration values", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.config.ConfigCommand", "options": [ {
"names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name": "get",
"fullName": "config get", "description": "Display user configuration value",
"sourceClass": "org.apache.camel.dsl.jbang.core.commands.config. [...]
{ "name": "debug", "fullName": "debug", "description": "Debug local Camel
integration", "sourceClass": "org.apache.camel.dsl.jbang.core.commands.Debug",
"options": [ { "names": "--ago", "description": "Use ago instead of yyyy-MM-dd
HH:mm:ss in timestamp.", "javaType": "boolean", "type": "boolean" }, { "names":
"--background", "description": "Run in the background", "defaultValue":
"false", "javaType": "boolean", "type": "boolean" }, { "names":
"--background-wait", "description": "How [...]
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/Ask.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/Ask.java
index dbdccd7ca25b..fef688a641cd 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/Ask.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/Ask.java
@@ -131,6 +131,10 @@ public class Ask extends CamelCommand {
// the tools the model does not get, such as starting a chaos exploration
that camel cmd chaos-explore started
Set<String> hiddenTools = Set.of();
+ // the last answer of the model, such as the report of camel cmd
chaos-explore, which it saves
+ String lastAnswer;
+ // the model that answered, and its API, such as qwen3.6 (ollama)
+ String usedModel;
private long targetPid;
private AskTools askTools;
@@ -170,6 +174,7 @@ public class Ask extends CamelCommand {
return 1;
}
warnSmallModel(client.model, client.ollamaParameterBillions());
+ usedModel = client.model + " (" + client.apiType + ")";
RuntimeHelper.ProcessInfo process = findProcess(nameOrPid);
if (process != null) {
@@ -296,6 +301,7 @@ public class Ask extends CamelCommand {
continue;
}
messages.add(LlmClient.Message.assistantWithToolCalls(response.text(),
List.of()));
+ lastAnswer = response.text();
printStats(totalUsage, startTime);
return 0;
}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ChaosExplore.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ChaosExplore.java
index 4a59b7eef5fc..fcfa6e1cad4e 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ChaosExplore.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ChaosExplore.java
@@ -16,6 +16,11 @@
*/
package org.apache.camel.dsl.jbang.core.commands;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
@@ -31,6 +36,7 @@ import
org.apache.camel.dsl.jbang.core.common.ProcessFinder.CamelProcess;
import org.apache.camel.dsl.jbang.core.common.RuntimeHelper;
import org.apache.camel.util.TimeUtils;
import org.apache.camel.util.json.JsonObject;
+import org.apache.camel.util.json.Jsoner;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;
@@ -110,6 +116,11 @@ public class ChaosExplore extends CamelCommand {
description = "Show the tool calls and results as they happen")
boolean showTools;
+ @Option(names = { "--save" }, negatable = true, defaultValue = "true",
fallbackValue = "true",
+ description = "Save the report in test/chaos/reports (as markdown,
and the journal as JSON), and each"
+ + " experiment that failed as a scenario in
test/chaos, to replay it")
+ boolean save = true;
+
@Option(names = { "--verbose" },
description = "Print debug information of the LLM calls")
boolean verbose;
@@ -133,34 +144,64 @@ public class ChaosExplore extends CamelCommand {
printer().printErr(started.error());
return started.exitCode();
}
- printer().println("Let there be chaos: " + process.name() + " (pid " +
process.pid() + "), "
+ String integration = displayName(process);
+ printer().println("Let there be chaos: " + integration + " (pid " +
process.pid() + "), "
+
TimeUtils.printDuration(started.reply().getLongOrDefault("budget", 0))
+ (focus != null && !focus.isBlank() ? ", focus: " +
focus : ""));
int answer;
+ Ask ask = ask(process, integration);
try {
- answer = ask(process).doCall();
+ answer = ask.doCall();
} catch (Exception e) {
// the LLM fails with an exit code; anything else must not cost
the report of what was explored so far
- printer().printErr("camel ask failed: " + e.getMessage());
+ // an exception without a message is known by its type
+ printer().printErr("camel ask failed: " + (e.getMessage() != null
? e.getMessage() : e.getClass().getSimpleName()));
answer = CliExitCode.FAILURE;
} finally {
// the AI may have finished it already; else its rules go now
ActionClient.execute(process, "chaos", root -> root.put("command",
"explore-finish"), 10000);
}
- JsonObject exploration =
ChaosExperimentRun.exploration(RuntimeHelper.readStatus(process.pid()));
+ JsonObject status = RuntimeHelper.readStatus(process.pid());
+ JsonObject exploration = ChaosExperimentRun.exploration(status);
List<String> lines = journal(exploration);
printer().println();
lines.forEach(printer()::println);
+ if (save && exploration != null) {
+ try {
+ Saved saved = save(Path.of("test", "chaos"),
LocalDateTime.now(),
+ new Run(integration, budget, focus, ask.usedModel,
seed(status)), exploration, ask.lastAnswer,
+ lines);
+ printer().println();
+ printer().println("Report: " + saved.report());
+ saved.scenarios()
+ .forEach(s -> printer().println("Scenario: " + s + "
(replay: camel run * --chaos=" + s + ")"));
+ } catch (IOException e) {
+ // the journal is printed already: say why it is not saved,
rather than a stack trace
+ printer().printErr("Could not save the report in test/chaos: "
+ e.getMessage());
+ }
+ }
if (failed(exploration)) {
return CliExitCode.FAILURE;
}
return answer;
}
+ /** The name of the integration as camel get chaos has it: its
CamelContext, else the name of the process. */
+ static String displayName(CamelProcess process) {
+ JsonObject status = process.status();
+ if (status != null && status.get("context") instanceof JsonObject
context) {
+ String name = context.getString("name");
+ if (name != null && !name.isBlank() && !"CamelJBang".equals(name))
{
+ return name;
+ }
+ }
+ return process.name();
+ }
+
/** camel ask, with the playbook as its question, for the integration. */
- Ask ask(CamelProcess process) {
+ Ask ask(CamelProcess process, String integration) {
Ask ask = new Ask(getMain());
- ask.question =
List.of(ChaosExplorePlaybook.text(ChaosExplorePlaybook.ASK, process.name(),
budget, focus, true));
+ ask.question =
List.of(ChaosExplorePlaybook.text(ChaosExplorePlaybook.ASK, integration,
budget, focus, true));
ask.nameOrPid = Long.toString(process.pid());
ask.url = url;
ask.apiType = apiType;
@@ -173,11 +214,135 @@ public class ChaosExplore extends CamelCommand {
ask.showTools = showTools;
ask.showStats = true;
ask.verbose = verbose;
- // the exploration runs with the budget of the user: the AI does not
start another one
- ask.hiddenTools = Set.of("chaos_explore_start");
+ // the exploration runs with the budget of the user: the AI does not
start another one, not even with a camel
+ // command, and the chaos tools are all it needs
+ ask.hiddenTools = Set.of("chaos_explore_start", "cli_exec");
return ask;
}
+ /** What the exploration ran with, for the report. */
+ record Run(String integration, String budget, String focus, String model,
Long seed) {
+ }
+
+ /** The files written: the report (markdown, with the journal as JSON next
to it), and a scenario per failure. */
+ record Saved(Path report, Path journal, List<Path> scenarios) {
+ }
+
+ private static Long seed(JsonObject status) {
+ if (status != null && status.get("chaos") instanceof JsonObject chaos
+ && chaos.get("report") instanceof JsonObject report &&
report.get("seed") instanceof Number n) {
+ return n.longValue();
+ }
+ return null;
+ }
+
+ /**
+ * Saves what the exploration found (CAMEL-26788): the report of the AI
with the journal of the experiments as
+ * markdown in reports/, the journal as JSON next to it, and each
experiment that failed as a scenario, so camel run
+ * --chaos or camel cmd chaos-load replays its verdict.
+ *
+ * @param dir the chaos folder of the project, test/chaos, which camel run
and export leave alone
+ */
+ static Saved save(
+ Path dir, LocalDateTime when, Run run, JsonObject exploration,
String report, List<String> journal)
+ throws IOException {
+ String stamp =
when.format(DateTimeFormatter.ofPattern("yyyyMMdd-HHmmss"));
+ String date = when.format(DateTimeFormatter.ofPattern("yyyy-MM-dd
HH:mm"));
+ Path reports = Files.createDirectories(dir.resolve("reports"));
+ List<Path> scenarios = new ArrayList<>();
+ List<Object> experiments = exploration.getCollection("experiments");
+ if (experiments != null) {
+ for (Object o : experiments) {
+ if (!(o instanceof JsonObject e) ||
!"FAIL".equals(e.getString("verdict"))) {
+ continue;
+ }
+ Path scenario = dir.resolve("explore-" + stamp + "-" +
e.getIntegerOrDefault("number", 0) + ".chaos");
+ Files.writeString(scenario, scenario(e, run, date, scenario));
+ scenarios.add(scenario);
+ }
+ }
+
+ StringBuilder md = new StringBuilder();
+ md.append("# Chaos exploration:
").append(run.integration()).append("\n\n");
+ md.append(date).append(" · budget ").append(run.budget());
+ if (run.focus() != null && !run.focus().isBlank()) {
+ md.append(" · focus ").append(run.focus());
+ }
+ if (run.model() != null) {
+ md.append(" · ").append(run.model());
+ }
+ if (run.seed() != null) {
+ md.append(" · seed ").append(run.seed());
+ }
+ md.append("\n\n## Report\n\n").append(report != null &&
!report.isBlank()
+ ? report.strip() : "(the AI wrote no report)").append("\n\n##
Experiments\n\n```text\n");
+ journal.forEach(line -> md.append(line).append('\n'));
+ md.append("```\n");
+ if (!scenarios.isEmpty()) {
+ md.append("\n## Scenarios\n\nEach experiment that failed, to
replay its verdict:\n\n");
+ for (Path s : scenarios) {
+ md.append("- `").append(s).append("`: `camel run *
--chaos=").append(s).append("`\n");
+ }
+ }
+ Path markdown = reports.resolve("explore-" + stamp + ".md");
+ Files.writeString(markdown, md.toString());
+
+ JsonObject json = new JsonObject();
+ json.put("integration", run.integration());
+ json.put("budget", run.budget());
+ if (run.focus() != null) {
+ json.put("focus", run.focus());
+ }
+ if (run.model() != null) {
+ json.put("model", run.model());
+ }
+ if (run.seed() != null) {
+ json.put("seed", run.seed());
+ }
+ json.put("report", report);
+ json.put("exploration", exploration);
+ json.put("scenarios", scenarios.stream().map(Path::toString).toList());
+ Path journalFile = reports.resolve("explore-" + stamp + ".json");
+ Files.writeString(journalFile, Jsoner.prettyPrint(json.toJson()));
+ return new Saved(markdown, journalFile, scenarios);
+ }
+
+ /** A failed experiment as a scenario: what it tested and found as
comments, the seed, and its rules. */
+ static String scenario(JsonObject experiment, Run run, String date, Path
file) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("# From the chaos exploration of
").append(run.integration()).append(" on ").append(date)
+ .append(", experiment
").append(experiment.getIntegerOrDefault("number", 0)).append(": ")
+ .append(experiment.getString("verdict")).append('\n');
+ String hypothesis = experiment.getString("hypothesis");
+ if (hypothesis != null) {
+ sb.append("# hypothesis: ").append(hypothesis).append('\n');
+ }
+ List<Object> expectations = experiment.getCollection("expectations");
+ if (expectations != null) {
+ for (Object o : expectations) {
+ if (!(o instanceof JsonObject x)) {
+ continue;
+ }
+ sb.append("# expect ").append(x.getString("expect")).append(":
").append(x.getString("state"))
+ .append(",
").append(x.getString("value")).append('\n');
+ }
+ }
+ String note = experiment.getString("note");
+ if (note != null) {
+ sb.append("# note: ").append(note).append('\n');
+ }
+ sb.append("# replay: camel run * --chaos=").append(file)
+ .append(" (and send it messages as the experiment did, when
its routes do not feed themselves)\n");
+ if (run.seed() != null) {
+ sb.append("seed=").append(run.seed()).append('\n');
+ }
+ List<Object> rules = experiment.getCollection("ruleTexts");
+ if (rules != null) {
+ rules.forEach(r -> sb.append(r).append('\n'));
+ }
+ return sb.toString();
+ }
+
/** The journal of the exploration: how it ended, and a line per
experiment with its expectations. */
static List<String> journal(JsonObject exploration) {
List<String> lines = new ArrayList<>();
@@ -197,7 +362,9 @@ public class ChaosExplore extends CamelCommand {
+ (count == 1 ? " experiment" : " experiments") + ", " +
failed + " failed");
if (experiments != null) {
for (Object o : experiments) {
- lines.addAll(ListChaos.experimentLines((JsonObject) o, " "));
+ if (o instanceof JsonObject e) {
+ lines.addAll(ListChaos.experimentLines(e, " "));
+ }
}
}
return lines;
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
index 3d6a42842a7f..91eda0f15abe 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
@@ -97,7 +97,8 @@ public class CamelChaosExperimentAction extends
CamelChaosAction {
String interval;
@CommandLine.Option(names = { "--wait" }, defaultValue = "2m",
- description = "The longest wait for the verdict after
the messages were sent (at most 10m)")
+ description = "The longest wait for the verdict after
the messages were sent, and at least the"
+ + " within= of the expectations (at most
10m)")
String wait;
@CommandLine.Option(names = { "--note" },
@@ -146,6 +147,10 @@ public class CamelChaosExperimentAction extends
CamelChaosAction {
feed, absolute(body), headers != null ?
String.join("\n", headers) : null,
repeat, interval)
: null;
+ String why = traffic != null ?
ChaosExperimentRun.checkFeed(RuntimeHelper.readStatus(process.pid()), feed) :
null;
+ if (why != null) {
+ throw CliException.usage(why.replace("The feed", "--feed"));
+ }
JsonObject result = ChaosExperimentRun.run(new
ChaosExperimentRun.Integration() {
@Override
public JsonObject chaos(String command, Consumer<JsonObject>
options) {
@@ -188,6 +193,10 @@ public class CamelChaosExperimentAction extends
CamelChaosAction {
if (!lines.isEmpty()) {
lines.set(0, "Experiment " + lines.get(0));
}
+ List<Object> warnings = experiment.getCollection("warnings");
+ if (warnings != null) {
+ warnings.forEach(w -> lines.add(" Warning: " + w));
+ }
JsonObject traffic = (JsonObject) experiment.get("traffic");
if (traffic != null) {
lines.add(" traffic: " + traffic(traffic));
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ChaosExplorePlaybook.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ChaosExplorePlaybook.java
index 33a80209ac75..181f53d49339 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ChaosExplorePlaybook.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ChaosExplorePlaybook.java
@@ -80,20 +80,29 @@ public final class ChaosExplorePlaybook {
route still completing its messages in time.
- hypothesis: what should hold, in a few words, such as "a
failed refund is retried, not lost"
- rule: the suggested rule with a short until= (such as
until=30s), and one expectation that \
- decides the hypothesis, with within= a little longer than
until=: expect=<route>:failed<=0 \
- (nothing fails), expect=<route>-><next route>:lost<=0 (nothing
lost between two routes), \
+ decides the hypothesis, with within= a little longer than
until=. Expect 0 where nothing may go \
+ wrong, not a guess such as failed<=5, which hides the messages
that were lost: expect=<route>:failed<=0 \
+ (nothing fails), expect=<route>-><next route>:lost<=0 (nothing
lost when a route hands messages to \
+ the next route, as the suggestions of kind lost name them), \
expect=<route>:20 (20 completed), expect=<route>:maxTime<=5s
(none slower than 5s)
- - feed: when the feed of the suggestion is kind send, its
endpoint, with body, repeat (such as \
- 20) and interval (such as 100..300); leave feed out when the
route feeds itself (kind self). \
+ - feed: a route that does not feed itself gets no messages
unless you give feed. When the feed \
+ of the suggestion is kind send, always give its endpoint, with
body (a message as the route \
+ expects it, such as JSON), repeat (such as 20) and interval
(such as 100..300); leave feed out \
+ only when the route feeds itself (kind self). \
Routes fed over http or from files cannot be fed by an
experiment yet: try them only when they \
feed themselves
+ - a rule at=dlq fails the dead letter channel, so it fires
only when a message goes there: use \
+ the combined suggestions, which add the fault that sends
messages there too. A message the \
+ dead letter channel did not deliver is handled there, so it
does not fail in its route: the \
+ experiment counts it as lost at the dead letter channel, and
fails then
- wait: longer than the within= of the expectation
- note: what you make of the result, in one sentence
4. Read the result: the verdict (FAIL, PASS or INCONCLUSIVE),
the expectations, what the routes \
- did, and the traffic. On FAIL, call %s for the failed
exchanges and their exceptions, and say why it \
+ did, the traffic, and the warnings: a rule that matched
nothing made the experiment say nothing, so \
+ fix the rule (its pattern, such as endpoint=file://parked) and
run it again. On FAIL, call %s for the failed exchanges and their exceptions,
and say why it \
broke. On INCONCLUSIVE, run it again with a longer wait. On
PASS, move on.
- 5. Go on while timeLeft of the result is more than the next
experiment needs (its until= and wait, \
- plus a minute). Stop with about a minute left.
+ 5. Go on while timeLeft of the result (millis) is more than
the next experiment needs: its wait \
+ plus a minute. When it is less, do not begin another
experiment: finish.
6. Finish with %s, and write the report:
- Summary: how many experiments, how many held and how many
broke
- Findings: per FAIL the route, what broke, the evidence
(the expectation and its actual value, \
@@ -102,7 +111,8 @@ public final class ChaosExplorePlaybook {
exist; say when it looks like a bug in Camel rather than in
the route
- What held: one line per PASS
- Not tried: the kinds of fault or routes left out, and why
(no feed, out of time)
- """.formatted(where, time, about, start, tools.report(),
tools.suggest(), tools.experiment(),
- tools.errors(), tools.finish());
+ """
+ .formatted(where, time, about, start, tools.report(),
tools.suggest(), tools.experiment(),
+ tools.errors(), tools.finish());
}
}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolRegistry.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolRegistry.java
index 0c9f44d0454f..045e389c4808 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolRegistry.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolRegistry.java
@@ -963,7 +963,8 @@ public final class ToolRegistry {
+ "INCONCLUSIVE: the wait was up
before one was decided, so wait longer than "
+ "its within=), the expectations
with their actual values, the faults per "
+ "rule, what the routes did
meanwhile, the traffic sent, and the time left "
- + "of the exploration (timeLeft,
millis). chaos_suggest has rules to try and "
+ + "of the exploration (timeLeft,
millis), and warnings for a rule that matched "
+ + "nothing (fix it and run again).
chaos_suggest has rules to try and "
+ "the feed of each route.")
.param("rule", "string",
"The rules in their one-line text form, separated by
;, one of them with expect=, such as"
@@ -982,7 +983,7 @@ public final class ToolRegistry {
"The pause between two messages in millis, or a range
such as 100..500 (at most 60000)", false)
.param("wait", "string",
"The longest wait for the verdict after the messages
were sent, such as 90s (default 2m, at most"
- + " 10m)",
+ + " 10m); it is at least the within=
of the expectations",
false)
.param("note", "string", "What to note with the experiment in
the journal", false)
.param("name", "string", CHAOS_NAME, false)
@@ -1005,26 +1006,32 @@ public final class ToolRegistry {
feed, args.get("body"),
args.get("headers"), repeat,
args.get("interval"))
: null;
- return ChaosExperimentRun.run(new
ChaosExperimentRun.Integration() {
- @Override
- public JsonObject chaos(String command,
Consumer<JsonObject> options) {
- String answer = ctx.executeAction("chaos", root ->
{
- root.put("command", command);
- options.accept(root);
- });
- return Jsoner.deserialize(answer, new
JsonObject());
- }
+ // the run checks the feed first: an experiment without
messages says nothing, so the AI gets the
+ // routes to feed instead, as a tool error (and the status
is read once for it)
+ try {
+ return ChaosExperimentRun.run(new
ChaosExperimentRun.Integration() {
+ @Override
+ public JsonObject chaos(String command,
Consumer<JsonObject> options) {
+ String answer = ctx.executeAction("chaos",
root -> {
+ root.put("command", command);
+ options.accept(root);
+ });
+ return Jsoner.deserialize(answer, new
JsonObject());
+ }
- @Override
- public JsonObject send(ChaosExperimentRun.Feed f) {
- return ctx.sendMessage(f.endpoint(), f.body(),
f.headers(), f.repeat(), f.interval());
- }
+ @Override
+ public JsonObject send(ChaosExperimentRun.Feed f) {
+ return ctx.sendMessage(f.endpoint(), f.body(),
f.headers(), f.repeat(), f.interval());
+ }
- @Override
- public JsonObject status() {
- return ctx.readFullStatus();
- }
- }, args.get("hypothesis"), rule, traffic, wait,
args.get("note")).toJson();
+ @Override
+ public JsonObject status() {
+ return ctx.readFullStatus();
+ }
+ }, args.get("hypothesis"), rule, traffic, wait,
args.get("note")).toJson();
+ } catch (IllegalArgumentException e) {
+ throw new ToolExecutionException(e.getMessage(), e);
+ }
}));
register(tool("chaos_explore_finish",
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
index 742c29a7c6dc..4be77ccc7705 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
@@ -17,7 +17,10 @@
package org.apache.camel.dsl.jbang.core.common;
import java.util.List;
+import java.util.StringJoiner;
import java.util.function.Consumer;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.camel.util.TimeUtils;
import org.apache.camel.util.json.JsonObject;
@@ -132,6 +135,18 @@ public final class ChaosExperimentRun {
static JsonObject run(
Integration integration, String hypothesis, String rules, Feed
feed, long wait, String note, Clock clock)
throws InterruptedException {
+ if (feed != null) {
+ // a feed that reaches no route makes an experiment without
messages, whose verdict says nothing
+ String why = checkFeed(integration.status(), feed.endpoint());
+ if (why != null) {
+ throw new IllegalArgumentException(why);
+ }
+ }
+ // an expectation such as failed<=0 is decided when its within= is up:
a shorter wait leaves it undecided
+ long within = longestWithin(rules);
+ if (within + POLL * 2 > wait) {
+ wait = Math.min(within + POLL * 2, MAX_WAIT);
+ }
JsonObject begun = integration.chaos("experiment-begin", root -> {
if (hypothesis != null && !hypothesis.isBlank()) {
root.put("hypothesis", hypothesis);
@@ -139,6 +154,8 @@ public final class ChaosExperimentRun {
root.put("rule", rules);
});
int number = begun.getIntegerOrDefault("number", 0);
+ // a rule that matches nothing in the integration now (often a typo)
+ List<Object> warnings = begun.getCollection("warnings");
JsonObject traffic = null;
long waited = 0;
try {
@@ -170,6 +187,9 @@ public final class ChaosExperimentRun {
answer.put("traffic", traffic);
}
answer.put("waited", waited);
+ if (warnings != null && !warnings.isEmpty()) {
+ answer.put("warnings", warnings);
+ }
JsonObject exploration = exploration(integration.status());
if (exploration != null && exploration.containsKey("timeLeft")) {
answer.put("timeLeft", exploration.getLong("timeLeft"));
@@ -177,6 +197,22 @@ public final class ChaosExperimentRun {
return answer;
}
+ /** The longest within= of the rules, in millis, or 0 (an expectation
without it has the time of its rule). */
+ static long longestWithin(String rules) {
+ long answer = 0;
+ Matcher m = WITHIN.matcher(rules);
+ while (m.find()) {
+ try {
+ answer = Math.max(answer,
TimeUtils.toMilliSeconds(m.group(1)));
+ } catch (RuntimeException e) {
+ // not a time: camel-chaos refuses the rule with the reason
+ }
+ }
+ return answer;
+ }
+
+ private static final Pattern WITHIN =
Pattern.compile("(?:^|[\\s;])within=([^\\s;]+)");
+
/** Sends the traffic: the summary of the send, or why it could not be
sent. */
private static JsonObject traffic(Integration integration, Feed feed) {
JsonObject answer;
@@ -219,6 +255,43 @@ public final class ChaosExperimentRun {
}
}
+ /**
+ * Why the feed reaches no route of the integration, or null when it does
(or the status does not tell): the feed is
+ * the id of a route, a pattern, or the endpoint a route starts from, as
camel cmd send finds its target.
+ */
+ public static String checkFeed(JsonObject status, String feed) {
+ List<Object> routes = status != null ? status.getCollection("routes")
: null;
+ if (routes == null || routes.isEmpty() || feed.endsWith("*")) {
+ return null;
+ }
+ String endpoint = normalize(feed);
+ StringJoiner inputs = new StringJoiner(", ");
+ for (Object o : routes) {
+ if (!(o instanceof JsonObject route)) {
+ continue;
+ }
+ String id = route.getString("routeId");
+ String from = route.getString("from");
+ if (feed.equals(id)) {
+ return null;
+ }
+ if (from != null) {
+ String input = normalize(from);
+ if (input.equals(endpoint) || input.startsWith(endpoint +
"?")) {
+ return null;
+ }
+ }
+ inputs.add(id + " (" + from + ")");
+ }
+ return "The feed " + feed + " is not where a route starts, so no
message would reach the integration: give the"
+ + " id of a route, or the endpoint it starts from: " + inputs;
+ }
+
+ // kafka://refunds and kafka:refunds are the same endpoint
+ private static String normalize(String uri) {
+ return uri.replace("://", ":");
+ }
+
/** The experiment with the number in the report of the status, or null. */
static JsonObject experiment(JsonObject status, int number) {
JsonObject exploration = exploration(status);
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ChaosExploreTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ChaosExploreTest.java
index fa94b0c72251..0741551c995a 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ChaosExploreTest.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ChaosExploreTest.java
@@ -16,13 +16,18 @@
*/
package org.apache.camel.dsl.jbang.core.commands;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.LocalDateTime;
import java.util.List;
import java.util.Map;
import org.apache.camel.dsl.jbang.core.common.ProcessFinder.CamelProcess;
import org.apache.camel.util.json.JsonArray;
import org.apache.camel.util.json.JsonObject;
+import org.apache.camel.util.json.Jsoner;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
import picocli.CommandLine;
import static org.assertj.core.api.Assertions.assertThat;
@@ -39,7 +44,7 @@ class ChaosExploreTest {
CommandLine.populateCommand(command, "shop", "--budget=30m",
"--focus=refunds", "--model=qwen3.6",
"--api-type=ollama", "--tools=core", "--show-tools");
- Ask ask = command.ask(new CamelProcess(4711, "shop", null));
+ Ask ask = command.ask(new CamelProcess(4711, "shop.camel.yaml", null),
"shop");
assertThat(ask.question).singleElement().asString()
.contains("explore how the integration shop copes", "The
exploration is running (30m)", "chaos_experiment");
@@ -52,8 +57,19 @@ class ChaosExploreTest {
assertThat(ask.timeout).isEqualTo(300);
assertThat(ask.maxIterations).isEqualTo(100);
assertThat(ask.yes).isFalse();
- // the exploration runs with the budget of the user: the AI does not
get the tool to start another one
- assertThat(ask.hiddenTools).containsExactly("chaos_explore_start");
+ // the exploration runs with the budget of the user: the AI does not
get the tools to start another one
+
assertThat(ask.hiddenTools).containsExactlyInAnyOrder("chaos_explore_start",
"cli_exec");
+ }
+
+ @Test
+ void theIntegrationIsNamedAsCamelGetChaosNamesIt() {
+ // camel run * names the process after its files, and the CamelContext
by --name
+ JsonObject status = new JsonObject(Map.of("context", new
JsonObject(Map.of("name", "shop"))));
+ assertThat(ChaosExplore.displayName(new CamelProcess(1,
"chaos.camel.yaml parked.camel.yaml", status)))
+ .isEqualTo("shop");
+ JsonObject unnamed = new JsonObject(Map.of("context", new
JsonObject(Map.of("name", "CamelJBang"))));
+ assertThat(ChaosExplore.displayName(new CamelProcess(1,
"shop.camel.yaml", unnamed))).isEqualTo("shop.camel.yaml");
+ assertThat(ChaosExplore.displayName(new CamelProcess(1,
"shop.camel.yaml", null))).isEqualTo("shop.camel.yaml");
}
@Test
@@ -80,4 +96,63 @@ class ChaosExploreTest {
assertThat(ChaosExplore.journal(null)).containsExactly(
"No journal of the exploration in the status of the
integration");
}
+
+ @Test
+ void whatTheExplorationFoundIsSaved(@TempDir Path dir) throws Exception {
+ JsonObject failed = new JsonObject(
+ Map.of("number", 1, "verdict", "FAIL", "open", false,
"endedBy", "ended",
+ "hypothesis", "nothing is lost after the hand-off",
"rules", new JsonArray(List.of("r1", "r2")),
+ "ruleTexts", new JsonArray(
+ List.of("route=shipping error rate=0.2
until=30s"
+ + " expect=orders->shipping:lost<=0
within=45s",
+ "endpoint=seda:dlq error at=dlq
until=30s")),
+ "expectations", new JsonArray(
+ List.of(new JsonObject(
+ Map.of("expect",
"orders->shipping:lost<=0", "state", "unmet", "value", "11 lost")))),
+ "note", "the seda hand-off has no error handling"));
+ JsonObject held = new JsonObject(
+ Map.of("number", 2, "verdict", "PASS", "open", false,
+ "rules", new JsonArray(List.of("r3")), "ruleTexts",
new JsonArray(List.of("route=refunds delay"))));
+ JsonObject exploration = new JsonObject(
+ Map.of("running", false, "finishedBy", "finished",
+ "experiments", new JsonArray(List.of(failed, held))));
+ ChaosExplore.Run run = new ChaosExplore.Run("shop", "10m", "orders",
"qwen3.6 (ollama)", -42L);
+
+ ChaosExplore.Saved saved = ChaosExplore.save(dir,
LocalDateTime.of(2026, 11, 9, 14, 2, 30), run, exploration,
+ "## Summary\n1 broke", List.of("Chaos exploration finished: 2
experiments, 1 failed"));
+
+
assertThat(saved.report()).isEqualTo(dir.resolve("reports/explore-20261109-140230.md"));
+ assertThat(Files.readString(saved.report())).contains("# Chaos
exploration: shop",
+ "2026-11-09 14:02 · budget 10m · focus orders · qwen3.6
(ollama) · seed -42",
+ "## Report\n\n## Summary\n1 broke", "Chaos exploration
finished: 2 experiments, 1 failed",
+ "## Scenarios", "camel run * --chaos=" +
dir.resolve("explore-20261109-140230-1.chaos"));
+ JsonObject journal = (JsonObject)
Jsoner.deserialize(Files.readString(saved.journal()));
+ assertThat(journal.getString("integration")).isEqualTo("shop");
+ assertThat(journal.getLong("seed")).isEqualTo(-42);
+ assertThat(journal.get("exploration")).isInstanceOf(Map.class);
+ // only the experiment that failed is a scenario: comments, the seed,
and its rules together
+
assertThat(saved.scenarios()).containsExactly(dir.resolve("explore-20261109-140230-1.chaos"));
+ assertThat(Files.readString(saved.scenarios().get(0))).isEqualTo(
+ "# From the chaos exploration of shop on 2026-11-09 14:02,
experiment 1: FAIL\n"
+ + "#
hypothesis: nothing is lost after the hand-off\n"
+ + "#
expect orders->shipping:lost<=0: unmet, 11 lost\n"
+ + "#
note: the seda hand-off has no error handling\n"
+ + "#
replay: camel run * --chaos="
+ +
dir.resolve("explore-20261109-140230-1.chaos")
+ + "
(and send it messages as the experiment did,"
+ + "
when its routes do not feed themselves)\n"
+ +
"seed=-42\n"
+ +
"route=shipping error rate=0.2 until=30s"
+ + "
expect=orders->shipping:lost<=0 within=45s\n"
+ +
"endpoint=seda:dlq error at=dlq until=30s\n");
+ }
+
+ @Test
+ void saveIsOnByDefault() {
+ ChaosExplore command = new ChaosExplore(new CamelJBangMain());
+ CommandLine.populateCommand(command);
+ assertThat(command.save).isTrue();
+ CommandLine.populateCommand(command, "--no-save");
+ assertThat(command.save).isFalse();
+ }
}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
index a4ec332abbf1..5ec3f1c84b8f 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
@@ -573,6 +573,13 @@ class CamelChaosActionTest extends
ActionCommandTestSupport {
" waited 2m0s for the verdict; 8m15s left of the exploration",
" An expectation was not decided when it ended: wait longer
than its within= (--wait)");
+ JsonObject typo = new JsonObject(
+ Map.of("number", 4, "verdict", "PASS", "elapsed", 1_000L,
+ "warnings",
+ new JsonArray(List.of("r5: endpoint=file:/parked
matches no endpoint of the context now"))));
+ assertThat(CamelChaosExperimentAction.lines(typo))
+ .contains(" Warning: r5: endpoint=file:/parked matches no
endpoint of the context now");
+
// another reason: waiting longer would not help (CAMEL-26790)
JsonObject idle = new JsonObject(
Map.of("number", 4, "verdict", "INCONCLUSIVE", "rules", new
JsonArray(List.of("r4")),
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
index b799e19b21d7..e0b558c1cbb6 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
@@ -44,6 +44,7 @@ class ChaosExperimentRunTest {
RuntimeException sendFails;
RuntimeException endFails;
RuntimeException beginFails;
+ List<String> beginWarnings;
JsonObject ended = experiment("FAIL", false);
@Override
@@ -56,7 +57,11 @@ class ChaosExperimentRunTest {
if (beginFails != null) {
throw beginFails;
}
- return experiment("PENDING", true);
+ JsonObject begun = experiment("PENDING", true);
+ if (beginWarnings != null) {
+ begun.put("warnings", new JsonArray(beginWarnings));
+ }
+ return begun;
}
if (endFails != null) {
throw endFails;
@@ -121,6 +126,8 @@ class ChaosExperimentRunTest {
@Test
void theTrafficThenTheWaitUntilTheVerdict() throws Exception {
+ // read first by the check of the feed
+ integration.shows("PENDING", true);
integration.shows("PENDING", true);
integration.shows("PENDING", true);
integration.shows("FAIL", true);
@@ -147,7 +154,9 @@ class ChaosExperimentRunTest {
integration.shows("PENDING", true);
integration.ended = experiment("INCONCLUSIVE", false);
- JsonObject answer = ChaosExperimentRun.run(integration, null, RULE,
null, 5_000, null, clock);
+ // an expectation without within= has the time of its rule, which the
wait does not know
+ JsonObject answer = ChaosExperimentRun.run(integration, null,
"route=refunds error expect=refunds:failed<=0", null,
+ 5_000, null, clock);
assertThat(answer.getString("verdict")).isEqualTo("INCONCLUSIVE");
assertThat(answer.getLong("waited")).isEqualTo(5_000);
@@ -194,6 +203,63 @@ class ChaosExperimentRunTest {
assertThat(integration.sent).isNull();
}
+ @Test
+ void aFeedThatReachesNoRouteIsRefusedBeforeTheExperiment() {
+ JsonObject status = new JsonObject(
+ Map.of("routes", new JsonArray(
+ List.of(
+ new JsonObject(Map.of("routeId", "orders",
"from", "timer://orders?period=1000")),
+ new JsonObject(
+ Map.of("routeId", "refunds", "from",
"kafka://refunds?brokers=localhost:9092"))))));
+ integration.statuses.add(status);
+
+ // the route id, the endpoint it starts from (with or without //), and
a pattern reach a route
+ assertThat(ChaosExperimentRun.checkFeed(status, "refunds")).isNull();
+ assertThat(ChaosExperimentRun.checkFeed(status,
"kafka:refunds")).isNull();
+ assertThat(ChaosExperimentRun.checkFeed(status,
"kafka://refunds")).isNull();
+ assertThat(ChaosExperimentRun.checkFeed(status, "ref*")).isNull();
+ // an endpoint no route starts from: no message would arrive, so the
verdict would say nothing
+ assertThatThrownBy(() -> ChaosExperimentRun.run(integration, null,
RULE,
+ new ChaosExperimentRun.Feed("direct:refunds", "R-1", null, 5,
null), 60_000, null, clock))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("The feed direct:refunds is not where a route
starts, so no message would reach the"
+ + " integration: give the id of a route, or the
endpoint it starts from:"
+ + " orders (timer://orders?period=1000), refunds
(kafka://refunds?brokers=localhost:9092)");
+ assertThat(integration.actions).isEmpty();
+ }
+
+ @Test
+ void aRuleThatMatchesNothingComesWithItsWarning() throws Exception {
+ integration.shows("PENDING", true);
+ integration.beginWarnings = List.of("r1: endpoint=file:/parked matches
no endpoint of the context now");
+ integration.ended = experiment("PASS", false);
+
+ JsonObject answer = ChaosExperimentRun.run(integration, null, RULE,
null, 1_000, null, clock);
+
+ // the PASS says nothing: the AI reads why
+ assertThat(answer.<List<Object>> getCollection("warnings"))
+ .containsExactly("r1: endpoint=file:/parked matches no
endpoint of the context now");
+ }
+
+ @Test
+ void theWaitIsAtLeastTheWithinOfTheExpectations() throws Exception {
+ integration.shows("PENDING", true);
+ integration.ended = experiment("PASS", false);
+
+ // RULE has within=30s: a wait of 5s would end it before failed<=0 is
decided
+ JsonObject answer = ChaosExperimentRun.run(integration, null, RULE,
null, 5_000, null, clock);
+
+ assertThat(answer.getLong("waited")).isEqualTo(32_000);
+ assertThat(ChaosExperimentRun.longestWithin("route=a error expect=a:1
within=1m; route=b drop expect=b:1 within=90s"))
+ .isEqualTo(90_000);
+ // the ; between the rules is not part of the time, also when it
follows the time without a space
+ assertThat(ChaosExperimentRun.longestWithin("route=a error expect=a:1
within=3m; route=b drop expect=b:1 within=90s"))
+ .isEqualTo(180_000);
+ assertThat(ChaosExperimentRun.longestWithin("route=a error expect=a:1
within=3m;route=b drop within=1m"))
+ .isEqualTo(180_000);
+ assertThat(ChaosExperimentRun.longestWithin("route=a error")).isZero();
+ }
+
@Test
void theWait() {
assertThat(ChaosExperimentRun.waitMillis(null)).isEqualTo(ChaosExperimentRun.DEFAULT_WAIT);
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
b/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
index 2b32127c9287..4d03a9ef8a3f 100644
--- a/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
+++ b/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
@@ -786,7 +786,8 @@ rules, send its traffic, wait for the verdict, and end it
range such as 100..500 to draw each pause at
random (at most 60s)
--wait=<wait> The longest wait for the verdict after the messages
- were sent (at most 10m)
+ were sent, and at least the within= of the
+ expectations (at most 10m)
Default: 2m
--note=<note> What to note with the experiment when it ends
--end End the open experiment (with --note) instead of
@@ -829,9 +830,10 @@ failed<=0 within=70s' \
camel cmd chaos-experiment --end --note='stopped by hand'
== camel cmd chaos-explore --help
-Usage: camel cmd chaos-explore [-h] [--show-tools] [--verbose] [--yes]
- [--api-key=<apiKey>] [--api-type=<apiType>]
- [--budget=<budget>] [--focus=<focus>]
+Usage: camel cmd chaos-explore [-h] [--[no-]save] [--show-tools] [--verbose]
+ [--yes] [--api-key=<apiKey>]
+ [--api-type=<apiType>] [--budget=<budget>]
+ [--focus=<focus>]
[--max-iterations=<maxIterations>]
[--model=<model>] [--timeout=<timeout>]
[--tools=<tools>] [--url=<url>] [<name>]
@@ -869,6 +871,10 @@ with faults, for a time budget, and reports what held and
what broke
calls that camel ask asks for first, without
asking
--show-tools Show the tool calls and results as they happen
+ --[no-]save Save the report in test/chaos/reports (as
+ markdown, and the journal as JSON), and each
+ experiment that failed as a scenario in
+ test/chaos, to replay it
--verbose Print debug information of the LLM calls
This command starts a chaos exploration with the budget (camel cmd
diff --git
a/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/RuntimeTools.java
b/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/RuntimeTools.java
index 95e1d96f308f..813e34815deb 100644
---
a/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/RuntimeTools.java
+++
b/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/RuntimeTools.java
@@ -1295,8 +1295,9 @@ public class RuntimeTools {
the experiment, which removes its rules. Returns the verdict
(FAIL: an expectation is unmet; PASS: \
all are met; INCONCLUSIVE: the wait was up before one was
decided, so wait longer than its \
within=), the expectations with their actual values, the
faults per rule, what the routes did \
- meanwhile, the traffic sent, and the time left of the
exploration (timeLeft, millis). \
- camel_runtime_chaos_suggest has rules to try and the feed of
each route.""")
+ meanwhile, the traffic sent, the time left of the
exploration (timeLeft, millis), and warnings \
+ for a rule that matched nothing (fix it and run again).
camel_runtime_chaos_suggest has rules to \
+ try and the feed of each route.""")
public JsonObject camel_runtime_chaos_experiment(
@ToolArg(description = CHAOS_NAME_OR_PID_DESC, required = false)
String nameOrPid,
@ToolArg(description = "The rules in their one-line text form,
separated by ;, one of them with expect=,"
@@ -1313,7 +1314,7 @@ public class RuntimeTools {
@ToolArg(description = "The pause between two messages in millis,
or a range such as 100..500",
required = false) String interval,
@ToolArg(description = "The longest wait for the verdict after the
messages were sent, such as 90s"
- + " (default 2m, at most 10m)",
+ + " (default 2m, at most 10m); it is at
least the within= of the expectations",
required = false) String wait,
@ToolArg(description = "What to note with the experiment in the
journal", required = false) String note) {
requireArg("rule", rule);
