This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 72fbc1a97df8 CAMEL-26786: camel-jbang - camel cmd chaos-experiment:
one experiment in one call, with its traffic and the wait for its verdict
(#29498)
72fbc1a97df8 is described below
commit 72fbc1a97df801fa6cdb0c0b0d66c279ad516695
Author: Claus Ibsen <[email protected]>
AuthorDate: Mon Nov 9 22:47:27 2026 +0100
CAMEL-26786: camel-jbang - camel cmd chaos-experiment: one experiment in
one call, with its traffic and the wait for its verdict (#29498)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/docs/chaos.adoc | 4 +
components/camel-chaos/src/main/docs/chaos.adoc | 4 +
.../modules/ROOT/pages/camel-jbang-managing.adoc | 30 +++
.../camel-jbang-cmd-chaos-experiment.adoc | 36 +++
.../camel-jbang-cmd-chaos-explore-finish.adoc | 26 +++
.../camel-jbang-cmd-chaos-explore-start.adoc | 28 +++
.../ROOT/pages/jbang-commands/camel-jbang-cmd.adoc | 3 +
.../META-INF/camel-jbang-commands-metadata.json | 2 +-
.../dsl/jbang/core/commands/CamelJBangMain.java | 3 +
.../action/CamelChaosExperimentAction.java | 241 ++++++++++++++++++++
.../action/CamelChaosExploreFinishAction.java | 71 ++++++
.../action/CamelChaosExploreStartAction.java | 75 +++++++
.../dsl/jbang/core/commands/process/ListChaos.java | 25 ++-
.../dsl/jbang/core/common/ChaosExperimentRun.java | 245 +++++++++++++++++++++
.../core/commands/action/CamelChaosActionTest.java | 102 ++++++++-
.../jbang/core/common/ChaosExperimentRunTest.java | 204 +++++++++++++++++
.../src/test/resources/golden/cli-help.txt | 211 ++++++++++++++----
17 files changed, 1261 insertions(+), 49 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
index 00bdae584b34..66d44391258b 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/chaos.adoc
@@ -420,6 +420,10 @@ $ camel get chaos --report
note: orders has no error handler
----
+The Camel CLI runs an experiment in one call: `camel cmd chaos-experiment`
begins it, sends its traffic (`--feed` with
+`--body`, `--repeat` and `--interval`, as `camel cmd send`), waits until its
expectations are decided (at most
+`--wait`), and ends it. `camel cmd chaos-explore-start` and
`chaos-explore-finish` start and finish the exploration.
+
== Scenarios
A scenario is the rules of an experiment in a file of the project, so it can
be run again, shared and kept with the
diff --git a/components/camel-chaos/src/main/docs/chaos.adoc
b/components/camel-chaos/src/main/docs/chaos.adoc
index 00bdae584b34..66d44391258b 100644
--- a/components/camel-chaos/src/main/docs/chaos.adoc
+++ b/components/camel-chaos/src/main/docs/chaos.adoc
@@ -420,6 +420,10 @@ $ camel get chaos --report
note: orders has no error handler
----
+The Camel CLI runs an experiment in one call: `camel cmd chaos-experiment`
begins it, sends its traffic (`--feed` with
+`--body`, `--repeat` and `--interval`, as `camel cmd send`), waits until its
expectations are decided (at most
+`--wait`), and ends it. `camel cmd chaos-explore-start` and
`chaos-explore-finish` start and finish the exploration.
+
== Scenarios
A scenario is the rules of an experiment in a file of the project, so it can
be run again, shared and kept with the
diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
b/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
index db2a227f25d0..d41e7b421294 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-managing.adoc
@@ -913,6 +913,36 @@ why and the `chaos-add` command that adds it.
been tried per route (the kinds of fault, how it went, and what the
suggestions offer that no rule tried yet). Chaos is for
development only: it stays off with profile `prod`, and the commands cannot
switch it on.
+=== Chaos explorations
+
+An exploration is a time to try chaos rules in, which camel-chaos enforces: at
its deadline the rules are removed,
+whatever happens. In it, `camel cmd chaos-experiment` runs one experiment in
one call. It adds the rules (one with an
+expectation that decides the hypothesis), sends the messages of `--feed` as
`camel cmd send` does, waits until the
+expectations are decided (at most `--wait`), ends the experiment and prints
what came of it:
+
+[source,bash]
+----
+camel cmd chaos-explore-start --budget=10m --focus=refunds
+camel cmd chaos-experiment --hypothesis='a failed refund is read again' \
+ --rule='route=refunds error rate=every:5 expect=refunds:failed<=0
within=30s' \
+ --feed=refunds --body=file:test/chaos/refunds-fails/ --repeat=20
--interval=200ms
+camel cmd chaos-explore-finish
+----
+
+[source,text]
+----
+Experiment 1 FAIL a failed refund is read again (r1, ended after 6s)
+ expect refunds:failed<=0: unmet, 4 failed
+ routes: refunds 16 completed, 4 failed
+ traffic: 20 messages sent to direct://refunds in 4s
+ waited 1s for the verdict; 9m51s left of the exploration
+----
+
+The verdict is `FAIL` when an expectation is unmet (exit code 1), `PASS` when
all are met, and `INCONCLUSIVE` when the
+wait was up before an expectation was decided. `--feed` takes the endpoint, or
the id of the route, as the `feed:` line
+of `chaos-suggest` has it. `camel get chaos --report` shows the exploration
with its experiments. See
+xref:next@components:others:chaos.adoc#_an_exploration[An exploration] in the
camel-chaos documentation.
+
=== Chaos scenarios
A scenario keeps the rules of an experiment in a file of the project, such as
`test/chaos/outage.chaos`: one rule per line,
diff --git
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
new file mode 100644
index 000000000000..f058c4f411d6
--- /dev/null
+++
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-experiment.adoc
@@ -0,0 +1,36 @@
+
+// AUTO-GENERATED by camel-package-maven-plugin - DO NOT EDIT THIS FILE
+= camel cmd chaos-experiment
+
+Run one chaos experiment in the exploration of a running integration: add its
rules, send its traffic, wait for the verdict, and end it
+
+
+== Usage
+
+[source,bash]
+----
+camel cmd chaos-experiment [options]
+----
+
+
+
+== Options
+
+[cols="2,5,1,2",options="header"]
+|===
+| Option | Description | Default | Type
+| `--body` | The body of the messages: text, file:<path>, or file:<folder> for
a message per file | | String
+| `--end` | End the open experiment (with --note) instead of running a new one
| | boolean
+| `--feed` | The endpoint, or the id of the route, to send the messages of the
experiment to | | String
+| `--header` | A header of the messages as key=value; can be given more than
once | | List
+| `--hypothesis` | What the experiment tests, in a few words, such as 'orders
survive a payment outage' | | String
+| `--interval` | The pause between two messages, such as 200ms, or a range
such as 100..500 to draw each pause at random (at most 60s) | | String
+| `--note` | What to note with the experiment when it ends | | String
+| `--repeat` | How many messages to send in all (at most 10000) | | int
+| `--rule` | A rule in its one-line text form, as an entry of
camel.chaos.rules; can be given more than once, and one of them needs an
expectation (expect=) | | List
+| `--timeout` | How long to wait for the integration to answer, in millis or
as a duration such as 30s | 10000 | long
+| `--wait` | The longest wait for the verdict after the messages were sent (at
most 10m) | 2m | String
+| `-h,--help` | Display the help and sub-commands | | boolean
+|===
+
+
diff --git
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore-finish.adoc
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore-finish.adoc
new file mode 100644
index 000000000000..7b480686f24b
--- /dev/null
+++
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore-finish.adoc
@@ -0,0 +1,26 @@
+
+// AUTO-GENERATED by camel-package-maven-plugin - DO NOT EDIT THIS FILE
+= camel cmd chaos-explore-finish
+
+Finish the chaos exploration of a running integration before its deadline
+
+
+== Usage
+
+[source,bash]
+----
+camel cmd chaos-explore-finish [options]
+----
+
+
+
+== Options
+
+[cols="2,5,1,2",options="header"]
+|===
+| Option | Description | Default | Type
+| `--timeout` | How long to wait for the integration to answer, in millis or
as a duration such as 30s | 10000 | long
+| `-h,--help` | Display the help and sub-commands | | boolean
+|===
+
+
diff --git
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore-start.adoc
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore-start.adoc
new file mode 100644
index 000000000000..3147bbe800b0
--- /dev/null
+++
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd-chaos-explore-start.adoc
@@ -0,0 +1,28 @@
+
+// AUTO-GENERATED by camel-package-maven-plugin - DO NOT EDIT THIS FILE
+= camel cmd chaos-explore-start
+
+Start a chaos exploration in a running integration: a time to try chaos rules
in, that camel-chaos enforces
+
+
+== Usage
+
+[source,bash]
+----
+camel cmd chaos-explore-start [options]
+----
+
+
+
+== Options
+
+[cols="2,5,1,2",options="header"]
+|===
+| Option | Description | Default | Type
+| `--budget` | How long the exploration runs, between 10s and 2h | 10m | String
+| `--focus` | What the exploration is about, such as a route | | String
+| `--timeout` | How long to wait for the integration to answer, in millis or
as a duration such as 30s | 10000 | long
+| `-h,--help` | Display the help and sub-commands | | boolean
+|===
+
+
diff --git
a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd.adoc
b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd.adoc
index f6a8cd0288cb..1c4a85c0047e 100644
--- a/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd.adoc
+++ b/docs/user-manual/modules/ROOT/pages/jbang-commands/camel-jbang-cmd.adoc
@@ -22,6 +22,9 @@ camel cmd [options]
| xref:jbang-commands/camel-jbang-cmd-chaos-add.adoc[chaos-add] | Add chaos
rules to a running integration (camel-chaos, for development only)
| xref:jbang-commands/camel-jbang-cmd-chaos-change.adoc[chaos-change] | Change
a chaos rule of a running integration (camel-chaos, for development only)
| xref:jbang-commands/camel-jbang-cmd-chaos-clear.adoc[chaos-clear] | Remove
all chaos rules of running integrations (camel-chaos, for development only)
+| xref:jbang-commands/camel-jbang-cmd-chaos-experiment.adoc[chaos-experiment]
| Run one chaos experiment in the exploration of a running integration: add its
rules, send its traffic, wait for the verdict, and end it
+|
xref:jbang-commands/camel-jbang-cmd-chaos-explore-finish.adoc[chaos-explore-finish]
| Finish the chaos exploration of a running integration before its deadline
+|
xref:jbang-commands/camel-jbang-cmd-chaos-explore-start.adoc[chaos-explore-start]
| Start a chaos exploration in a running integration: a time to try chaos
rules in, that camel-chaos enforces
| xref:jbang-commands/camel-jbang-cmd-chaos-load.adoc[chaos-load] | Load a
chaos scenario into a running integration (camel-chaos, for development only)
| xref:jbang-commands/camel-jbang-cmd-chaos-remove.adoc[chaos-remove] | Remove
a chaos rule of a running integration (camel-chaos, for development only)
| xref:jbang-commands/camel-jbang-cmd-chaos-suggest.adoc[chaos-suggest] |
Suggest chaos rules to try in a running integration, per route (camel-chaos,
for development only)
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
b/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
index d04b44c65d5d..ccde7b460870 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
+++
b/dsl/camel-jbang/camel-jbang-core/src/generated/resources/META-INF/camel-jbang-commands-metadata.json
@@ -3,7 +3,7 @@
{ "name": "ask", "fullName": "ask", "description": "Ask a question about a
running Camel application using AI", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.Ask", "options": [ { "names":
"--api-key", "description": "API key. Also reads ANTHROPIC_API_KEY,
OPENAI_API_KEY, WATSONX_APIKEY, or LLM_API_KEY env vars", "javaType":
"java.lang.String", "type": "string" }, { "names": "--api-type", "description":
"API type: 'ollama', 'openai', 'anthropic', or 'watsonx'", "javaType": [...]
{ "name": "bind", "fullName": "bind", "description": "DEPRECATED: Bind
source and sink Kamelets as a new Camel integration", "deprecated": true,
"sourceClass": "org.apache.camel.dsl.jbang.core.commands.bind.Bind", "options":
[ { "names": "--error-handler", "description": "Add error handler
(none|log|sink:<endpoint>). Sink endpoints are expected in the format
\"[[apigroup\/]version:]kind:[namespace\/]name\", plain Camel URIs or Kamelet
name.", "javaType": "java.lang.String", "type": " [...]
{ "name": "catalog", "fullName": "catalog", "description": "List artifacts
from Camel Catalog", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.catalog.CatalogCommand", "options": [
{ "names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name":
"component", "fullName": "catalog component", "description": "List components
from the Camel Catalog", "sourceClass": "org.apache.camel.dsl.jbang.co [...]
- { "name": "cmd", "fullName": "cmd", "description": "Performs commands in
the running Camel integrations, such as start\/stop route, or change logging
levels.", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.action.CamelAction", "options": [ {
"names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name":
"browse", "fullName": "cmd browse", "description": "Browse pending messages on
endpoints [...]
+ { "name": "cmd", "fullName": "cmd", "description": "Performs commands in
the running Camel integrations, such as start\/stop route, or change logging
levels.", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.action.CamelAction", "options": [ {
"names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name":
"browse", "fullName": "cmd browse", "description": "Browse pending messages on
endpoints [...]
{ "name": "completion", "fullName": "completion", "description": "Generate
completion script for bash\/zsh", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.Complete", "options": [ { "names":
"-h,--help", "description": "Display the help and sub-commands", "javaType":
"boolean", "type": "boolean" } ] },
{ "name": "config", "fullName": "config", "description": "Get and set user
configuration values", "sourceClass":
"org.apache.camel.dsl.jbang.core.commands.config.ConfigCommand", "options": [ {
"names": "-h,--help", "description": "Display the help and sub-commands",
"javaType": "boolean", "type": "boolean" } ], "subcommands": [ { "name": "get",
"fullName": "config get", "description": "Display user configuration value",
"sourceClass": "org.apache.camel.dsl.jbang.core.commands.config. [...]
{ "name": "debug", "fullName": "debug", "description": "Debug local Camel
integration", "sourceClass": "org.apache.camel.dsl.jbang.core.commands.Debug",
"options": [ { "names": "--ago", "description": "Use ago instead of yyyy-MM-dd
HH:mm:ss in timestamp.", "javaType": "boolean", "type": "boolean" }, { "names":
"--background", "description": "Run in the background", "defaultValue":
"false", "javaType": "boolean", "type": "boolean" }, { "names":
"--background-wait", "description": "How [...]
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/CamelJBangMain.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/CamelJBangMain.java
index fb056fd2da4e..89693244f2aa 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/CamelJBangMain.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/CamelJBangMain.java
@@ -262,6 +262,9 @@ public class CamelJBangMain implements Callable<Integer> {
.addSubcommand("chaos-add", new CommandLine(new
CamelChaosAddAction(this)))
.addSubcommand("chaos-change", new CommandLine(new
CamelChaosChangeAction(this)))
.addSubcommand("chaos-clear", new CommandLine(new
CamelChaosClearAction(this)))
+ .addSubcommand("chaos-experiment", new CommandLine(new
CamelChaosExperimentAction(this)))
+ .addSubcommand("chaos-explore-finish", new CommandLine(new
CamelChaosExploreFinishAction(this)))
+ .addSubcommand("chaos-explore-start", new CommandLine(new
CamelChaosExploreStartAction(this)))
.addSubcommand("chaos-load", new CommandLine(new
CamelChaosLoadAction(this)))
.addSubcommand("chaos-remove", new CommandLine(new
CamelChaosRemoveAction(this)))
.addSubcommand("chaos-suggest", new CommandLine(new
CamelChaosSuggestAction(this)))
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
new file mode 100644
index 000000000000..e2b2404ec08c
--- /dev/null
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExperimentAction.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.dsl.jbang.core.commands.action;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Consumer;
+
+import org.apache.camel.cli.connector.RepeatedSend;
+import org.apache.camel.dsl.jbang.core.commands.CamelJBangMain;
+import org.apache.camel.dsl.jbang.core.commands.process.ListChaos;
+import org.apache.camel.dsl.jbang.core.common.ChaosExperimentRun;
+import org.apache.camel.dsl.jbang.core.common.CliException;
+import org.apache.camel.dsl.jbang.core.common.CliExitCode;
+import org.apache.camel.dsl.jbang.core.common.ProcessFinder.CamelProcess;
+import org.apache.camel.dsl.jbang.core.common.RuntimeHelper;
+import org.apache.camel.util.TimeUtils;
+import org.apache.camel.util.json.JsonObject;
+import picocli.CommandLine;
+
[email protected](name = "chaos-experiment",
+ description = "Run one chaos experiment in the
exploration of a running integration: add its rules,"
+ + " send its traffic, wait for the verdict,
and end it",
+ sortOptions = false, showDefaultValues = true,
+ footer = {
+ "%nAn experiment is part of an exploration (camel
cmd chaos-explore-start). It tests a",
+ "hypothesis with rules, one of them with an
expectation (expect=) that decides it.",
+ "The rules are added together, the messages of
--feed are sent (as camel cmd send",
+ "sends them, with --body, --header, --repeat and
--interval), and then it waits until",
+ "the expectations are decided, or --wait is up.
The experiment is ended then, its",
+ "rules are removed, and what came of it is
printed: the verdict, the expectations,",
+ "what the routes did, and the traffic. It stays
in the journal of the exploration",
+ "(camel get chaos --report).",
+ "%nThe verdict is FAIL when an expectation is
unmet (exit code 1), PASS when all are",
+ "met, and INCONCLUSIVE when it ended before an
expectation was decided: wait longer",
+ "than the within= of the expectation. --feed can
be the id of a route, as the feed:",
+ "line of camel cmd chaos-suggest has it.",
+ "%nExamples:",
+ " camel cmd chaos-experiment --hypothesis='a
failed refund is read again' \\",
+ " --rule='route=refunds error rate=every:5
expect=refunds:failed<=0 within=30s' \\",
+ " --feed=refunds
--body=file:test/chaos/refunds-fails/ --repeat=20 --interval=200ms",
+ " camel cmd chaos-experiment myApp
--hypothesis='orders survive a payment outage' \\",
+ " --rule='endpoint=http:payment* error
status=503 until=1m expect=orders:failed<=0 within=70s' \\",
+ " --wait=90s",
+ " camel cmd chaos-experiment --end
--note='stopped by hand'" })
+public class CamelChaosExperimentAction extends CamelChaosAction {
+
+ @CommandLine.Parameters(description = "Name or pid of the running Camel
integration (must match one); without it,"
+ + " the only one running, or the
only one with chaos switched on",
+ arity = "0..1")
+ String name;
+
+ @CommandLine.Option(names = { "--hypothesis" },
+ description = "What the experiment tests, in a few
words, such as 'orders survive a payment outage'")
+ String hypothesis;
+
+ @CommandLine.Option(names = { "--rule" },
+ description = "A rule in its one-line text form, as an
entry of camel.chaos.rules; can be given more"
+ + " than once, and one of them needs an
expectation (expect=)")
+ List<String> rules;
+
+ @CommandLine.Option(names = { "--feed" },
+ description = "The endpoint, or the id of the route,
to send the messages of the experiment to")
+ String feed;
+
+ @CommandLine.Option(names = { "--body" },
+ description = "The body of the messages: text,
file:<path>, or file:<folder> for a message per file")
+ String body;
+
+ @CommandLine.Option(names = { "--header" },
+ description = "A header of the messages as key=value;
can be given more than once")
+ List<String> headers;
+
+ @CommandLine.Option(names = { "--repeat" },
+ description = "How many messages to send in all (at
most " + RepeatedSend.MAX_MESSAGES + ")")
+ int repeat;
+
+ @CommandLine.Option(names = { "--interval" },
+ description = "The pause between two messages, such as
200ms, or a range such as 100..500 to draw"
+ + " each pause at random (at most 60s)")
+ String interval;
+
+ @CommandLine.Option(names = { "--wait" }, defaultValue = "2m",
+ description = "The longest wait for the verdict after
the messages were sent (at most 10m)")
+ String wait;
+
+ @CommandLine.Option(names = { "--note" },
+ description = "What to note with the experiment when
it ends")
+ String note;
+
+ @CommandLine.Option(names = { "--end" },
+ description = "End the open experiment (with --note)
instead of running a new one")
+ boolean end;
+
+ public CamelChaosExperimentAction(CamelJBangMain main) {
+ super(main);
+ }
+
+ @Override
+ public Integer doCall() throws Exception {
+ if (end) {
+ CamelProcess process = findChaosTarget(name);
+ JsonObject ended = chaos(process, "experiment-end", root -> {
+ if (note != null && !note.isBlank()) {
+ root.put("note", note);
+ }
+ });
+ print(ended);
+ return exitCode(ended);
+ }
+ if (rules == null || rules.isEmpty()) {
+ throw CliException.usage("--rule is required: the rules of the
experiment, one of them with expect=");
+ }
+ if (feed == null && (body != null || headers != null || repeat > 0 ||
interval != null)) {
+ throw CliException.usage("--body, --header, --repeat and
--interval go with --feed: where to send the messages");
+ }
+ if (repeat < 0 || repeat > RepeatedSend.MAX_MESSAGES) {
+ throw CliException.usage("--repeat must be between 1 and " +
RepeatedSend.MAX_MESSAGES + ", was: " + repeat);
+ }
+ long waitMillis;
+ try {
+ RepeatedSend.interval(interval);
+ waitMillis = ChaosExperimentRun.waitMillis(wait);
+ } catch (IllegalArgumentException e) {
+ throw CliException.usage(e.getMessage().replace("The interval",
"--interval").replace("The wait", "--wait"));
+ }
+ CamelProcess process = findChaosTarget(name);
+ ChaosExperimentRun.Feed traffic = feed != null
+ ? new ChaosExperimentRun.Feed(
+ feed, absolute(body), headers != null ?
String.join("\n", headers) : null,
+ repeat, interval)
+ : null;
+ JsonObject result = ChaosExperimentRun.run(new
ChaosExperimentRun.Integration() {
+ @Override
+ public JsonObject chaos(String command, Consumer<JsonObject>
options) {
+ return CamelChaosExperimentAction.this.chaos(process, command,
options);
+ }
+
+ @Override
+ public JsonObject send(ChaosExperimentRun.Feed f) {
+ return RuntimeHelper.sendMessage(process.pid(), f.endpoint(),
f.body(), f.headers(), f.repeat(),
+ f.interval());
+ }
+
+ @Override
+ public JsonObject status() {
+ return RuntimeHelper.readStatus(process.pid());
+ }
+ }, hypothesis, String.join(";", rules), traffic, waitMillis, note);
+ print(result);
+ return exitCode(result);
+ }
+
+ // a file or a folder of messages here is sent with its absolute path, as
the integration may run elsewhere
+ private static String absolute(String body) {
+ if (body != null && body.startsWith("file:")) {
+ File f = new File(body.substring(5));
+ if (f.exists()) {
+ return "file:" + f.getAbsolutePath();
+ }
+ }
+ return body;
+ }
+
+ private void print(JsonObject experiment) {
+ lines(experiment).forEach(printer()::println);
+ }
+
+ /** What came of the experiment: its lines as the report has them, the
traffic, and the time it waited. */
+ static List<String> lines(JsonObject experiment) {
+ List<String> lines = new
ArrayList<>(ListChaos.experimentLines(experiment, ""));
+ if (!lines.isEmpty()) {
+ lines.set(0, "Experiment " + lines.get(0));
+ }
+ JsonObject traffic = (JsonObject) experiment.get("traffic");
+ if (traffic != null) {
+ lines.add(" traffic: " + traffic(traffic));
+ }
+ StringBuilder sb = new StringBuilder();
+ if (experiment.containsKey("waited")) {
+ sb.append("waited
").append(time(experiment.getLong("waited"))).append(" for the verdict");
+ }
+ if (experiment.containsKey("timeLeft")) {
+ sb.append(sb.isEmpty() ? "" : ";
").append(TimeUtils.printDuration(experiment.getLong("timeLeft")))
+ .append(" left of the exploration");
+ }
+ if (!sb.isEmpty()) {
+ lines.add(" " + sb);
+ }
+ if ("INCONCLUSIVE".equals(experiment.getString("verdict"))) {
+ lines.add(" An expectation was not decided when it ended: wait
longer than its within= (--wait)");
+ }
+ return lines;
+ }
+
+ private static String traffic(JsonObject traffic) {
+ String endpoint = traffic.getString("endpoint");
+ String error = traffic.getString("error");
+ if (error != null) {
+ return "not sent to " + endpoint + ": " + error;
+ }
+ if (traffic.containsKey("sent")) {
+ long sent = traffic.getLongOrDefault("sent", 0);
+ long failed = traffic.getLongOrDefault("failed", 0);
+ return sent + (sent == 1 ? " message" : " messages") + " sent to "
+ endpoint
+ + (failed > 0 ? " (" + failed + " failed)" : "") + " in "
+ + time(traffic.getLongOrDefault("elapsed", 0));
+ }
+ // a single message: the answer of the send
+ return "1 message sent to " + endpoint +
("success".equals(traffic.getString("status")) ? "" : " (failed)");
+ }
+
+ // a time under a second in millis, as a send of a few messages takes
+ private static String time(long millis) {
+ return TimeUtils.printDuration(millis, millis < 1000);
+ }
+
+ private static int exitCode(JsonObject experiment) {
+ return "FAIL".equals(experiment.getString("verdict")) ?
CliExitCode.FAILURE : CliExitCode.OK;
+ }
+
+ @Override
+ protected String describe(JsonObject answer) {
+ return String.join("\n", lines(answer));
+ }
+}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExploreFinishAction.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExploreFinishAction.java
new file mode 100644
index 000000000000..5b3c4ed771b9
--- /dev/null
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExploreFinishAction.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.dsl.jbang.core.commands.action;
+
+import java.util.List;
+
+import org.apache.camel.dsl.jbang.core.commands.CamelJBangMain;
+import org.apache.camel.dsl.jbang.core.common.ProcessFinder.CamelProcess;
+import org.apache.camel.util.TimeUtils;
+import org.apache.camel.util.json.JsonObject;
+import picocli.CommandLine;
+
[email protected](name = "chaos-explore-finish",
+ description = "Finish the chaos exploration of a running
integration before its deadline",
+ sortOptions = false, showDefaultValues = true,
+ footer = {
+ "%nThe rules are removed (an open experiment is
ended with what came of it so far), and",
+ "the journal stays in the report (camel get chaos
--report) until the next",
+ "exploration starts or the report is reset.",
+ "%nExamples:",
+ " camel cmd chaos-explore-finish",
+ " camel cmd chaos-explore-finish myApp" })
+public class CamelChaosExploreFinishAction extends CamelChaosAction {
+
+ @CommandLine.Parameters(description = "Name or pid of the running Camel
integration (must match one); without it,"
+ + " the only one running, or the
only one with chaos switched on",
+ arity = "0..1")
+ String name;
+
+ public CamelChaosExploreFinishAction(CamelJBangMain main) {
+ super(main);
+ }
+
+ @Override
+ public Integer doCall() throws Exception {
+ CamelProcess process = findChaosTarget(name);
+ printer().println(describe(chaos(process, "explore-finish", null)));
+ return 0;
+ }
+
+ @Override
+ protected String describe(JsonObject answer) {
+ long ran = answer.getLongOrDefault("finishedAt", 0) -
answer.getLongOrDefault("startedAt", 0);
+ List<Object> rules = answer.getCollection("rules");
+ List<Object> experiments = answer.getCollection("experiments");
+ StringBuilder sb = new StringBuilder("Chaos exploration finished after
").append(TimeUtils.printDuration(ran))
+ .append(": ").append(rules != null ? rules.size() :
0).append(" rules tried, ")
+ .append(answer.getLongOrDefault("removed", 0)).append("
removed");
+ if (experiments != null && !experiments.isEmpty()) {
+ long failed = experiments.stream()
+ .filter(o -> o instanceof JsonObject jo &&
"FAIL".equals(jo.getString("verdict"))).count();
+ sb.append(",
").append(experiments.size()).append(experiments.size() == 1 ? " experiment" :
" experiments")
+ .append(" (").append(failed).append(" failed)");
+ }
+ return sb.toString();
+ }
+}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExploreStartAction.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExploreStartAction.java
new file mode 100644
index 000000000000..4a8cc6d20f6a
--- /dev/null
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosExploreStartAction.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.dsl.jbang.core.commands.action;
+
+import org.apache.camel.dsl.jbang.core.commands.CamelJBangMain;
+import org.apache.camel.dsl.jbang.core.common.ProcessFinder.CamelProcess;
+import org.apache.camel.util.TimeUtils;
+import org.apache.camel.util.json.JsonObject;
+import picocli.CommandLine;
+
[email protected](name = "chaos-explore-start",
+ description = "Start a chaos exploration in a running
integration: a time to try chaos rules in,"
+ + " that camel-chaos enforces",
+ sortOptions = false, showDefaultValues = true,
+ footer = {
+ "%nWhile the exploration runs, camel-chaos keeps
its journal: the baseline (the counts",
+ "per route before its first rule), the rules
tried, and the experiments",
+ "(camel cmd chaos-experiment). A rule ends at the
deadline at the latest, and at the",
+ "deadline the rules are removed and the
exploration is finished, whatever happens.",
+ "One exploration runs at a time. camel get chaos
--report shows it.",
+ "%nExamples:",
+ " camel cmd chaos-explore-start",
+ " camel cmd chaos-explore-start myApp
--budget=30m --focus=orders" })
+public class CamelChaosExploreStartAction extends CamelChaosAction {
+
+ @CommandLine.Parameters(description = "Name or pid of the running Camel
integration (must match one); without it,"
+ + " the only one running, or the
only one with chaos switched on",
+ arity = "0..1")
+ String name;
+
+ @CommandLine.Option(names = { "--budget" }, defaultValue = "10m",
+ description = "How long the exploration runs, between
10s and 2h")
+ String budget;
+
+ @CommandLine.Option(names = { "--focus" },
+ description = "What the exploration is about, such as
a route")
+ String focus;
+
+ public CamelChaosExploreStartAction(CamelJBangMain main) {
+ super(main);
+ }
+
+ @Override
+ public Integer doCall() throws Exception {
+ CamelProcess process = findChaosTarget(name);
+ printer().println(describe(chaos(process, "explore-start", root -> {
+ root.put("budget", budget);
+ if (focus != null && !focus.isBlank()) {
+ root.put("focus", focus);
+ }
+ })));
+ return 0;
+ }
+
+ @Override
+ protected String describe(JsonObject answer) {
+ String focus = answer.getString("focus");
+ return "Chaos exploration started: " +
TimeUtils.printDuration(answer.getLongOrDefault("budget", 0))
+ + (focus != null ? " (focus: " + focus + ")" : "");
+ }
+}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
index f81650fb1806..ed5f4bc11332 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/process/ListChaos.java
@@ -350,7 +350,18 @@ public class ListChaos extends ProcessWatchCommand {
/** An experiment: its verdict and hypothesis, its rules, its
expectations, and what the routes did meanwhile. */
private void printExperiment(JsonObject experiment) {
- StringBuilder sb = new StringBuilder("
").append(experiment.getIntegerOrDefault("number", 0)).append(' ')
+ experimentLines(experiment, " ").forEach(printer()::println);
+ }
+
+ /**
+ * The lines of an experiment, as camel get chaos --report and camel cmd
chaos-experiment print it: its verdict and
+ * hypothesis, its rules and how long it ran, its expectations, what the
routes did meanwhile, and its note.
+ *
+ * @param indent the indent of the first line; the others are indented two
more
+ */
+ public static List<String> experimentLines(JsonObject experiment, String
indent) {
+ List<String> lines = new ArrayList<>();
+ StringBuilder sb = new
StringBuilder(indent).append(experiment.getIntegerOrDefault("number",
0)).append(' ')
.append(experiment.getString("verdict"));
String hypothesis = experiment.getString("hypothesis");
if (hypothesis != null) {
@@ -368,23 +379,25 @@ public class ListChaos extends ProcessWatchCommand {
? "ended at the deadline" : "finished".equals(by) ? "ended
with the exploration" : "ended")
.append(" after ").append(elapsed);
}
- printer().println(sb.append(')').toString());
+ lines.add(sb.append(')').toString());
+ String more = indent + " ";
List<Object> expectations = experiment.getCollection("expectations");
if (expectations != null) {
for (Object o : expectations) {
JsonObject x = (JsonObject) o;
- printer().println(" expect " + x.getString("expect") +
": " + x.getString("state") + ", "
- + x.getString("value"));
+ lines.add(more + "expect " + x.getString("expect") + ": " +
x.getString("state") + ", "
+ + x.getString("value"));
}
}
List<Object> routes = experiment.getCollection("routes");
if (routes != null && !routes.isEmpty()) {
- printer().println(" routes: " + routeCounts(routes));
+ lines.add(more + "routes: " + routeCounts(routes));
}
String note = experiment.getString("note");
if (note != null) {
- printer().println(" note: " + note);
+ lines.add(more + "note: " + note);
}
+ return lines;
}
private static String routeCounts(List<Object> routes) {
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
new file mode 100644
index 000000000000..40b0a5a1b751
--- /dev/null
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRun.java
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.dsl.jbang.core.common;
+
+import java.util.List;
+import java.util.function.Consumer;
+
+import org.apache.camel.util.TimeUtils;
+import org.apache.camel.util.json.JsonObject;
+
+/**
+ * One chaos experiment in one call (CAMEL-26786), as camel cmd
chaos-experiment and the chaos_experiment tool run it:
+ * it begins the experiment in the running exploration (its hypothesis and
rules), sends the traffic of its feed, waits
+ * until its expectations are decided, and ends it. The answer is what came of
it (as the journal has it), with the
+ * traffic that was sent and the time left of the exploration.
+ * <p/>
+ * The wait reads the report in the status of the integration, which is
written every second. It ends as soon as an
+ * expectation is unmet (FAIL) or all are met (PASS), when the exploration
ended the experiment at its deadline, or
+ * after the longest wait; an experiment ended then with an expectation not
decided is INCONCLUSIVE. It is ended in any
+ * case, also when the traffic could not be sent, so its rules do not stay.
+ */
+public final class ChaosExperimentRun {
+
+ /** How long to wait for the verdict after the traffic, by default. */
+ public static final long DEFAULT_WAIT = 2 * 60_000L;
+ /** The longest wait for the verdict (the exploration ends the experiment
at its deadline anyway). */
+ public static final long MAX_WAIT = 10 * 60_000L;
+ // the status of the integration is written every second
+ private static final long POLL = 1000;
+
+ private ChaosExperimentRun() {
+ }
+
+ /** What the experiment needs of the integration: the CLI and the tools
each reach it their own way. */
+ public interface Integration {
+
+ /**
+ * Sends the chaos action, and returns the answer.
+ *
+ * @throws RuntimeException when the integration refuses it, with the
reason
+ */
+ JsonObject chaos(String command, Consumer<JsonObject> options);
+
+ /** Sends the messages of the feed, and returns the summary of the
send action. */
+ JsonObject send(Feed feed);
+
+ /** The status of the integration, as it wrote it last, or null. */
+ JsonObject status();
+ }
+
+ /**
+ * The traffic of an experiment, as camel cmd send sends it.
+ *
+ * @param endpoint the endpoint or the id of the route to send to
+ * @param body the body, file:<path>, or file:<folder> for
a message per file
+ * @param headers the headers as key=value lines, or null
+ * @param repeat how many messages in all; 0 to send once (each file of
a folder once)
+ * @param interval the pause between two messages, such as 200ms or a
range such as 100..500, or null
+ */
+ public record Feed(String endpoint, String body, String headers, int
repeat, String interval) {
+ }
+
+ /** The time, and a pause; the tests give a clock that does not wait. */
+ interface Clock {
+ long now();
+
+ void sleep(long millis) throws InterruptedException;
+ }
+
+ private static final Clock SYSTEM = new Clock() {
+ @Override
+ public long now() {
+ return System.currentTimeMillis();
+ }
+
+ @Override
+ public void sleep(long millis) throws InterruptedException {
+ Thread.sleep(millis);
+ }
+ };
+
+ /** The longest wait as text, such as 2m: 2 minutes when empty. */
+ public static long waitMillis(String text) {
+ if (text == null || text.isBlank()) {
+ return DEFAULT_WAIT;
+ }
+ long millis;
+ try {
+ millis = TimeUtils.toMilliSeconds(text.strip());
+ } catch (RuntimeException e) {
+ throw new IllegalArgumentException("The wait is not a time, such
as 90s or 2m: " + text);
+ }
+ if (millis < 0 || millis > MAX_WAIT) {
+ throw new IllegalArgumentException("The wait must be at most 10m,
was: " + text);
+ }
+ return millis;
+ }
+
+ /**
+ * Runs the experiment.
+ *
+ * @param hypothesis what it tests, such as "a failed refund is read
again" (optional)
+ * @param rules its rules in their one-line form, separated by ;,
one of them with an expectation
+ * @param feed the traffic, or null when the routes feed themselves
(a timer) or are fed otherwise
+ * @param wait the longest wait for the verdict after the traffic,
in millis
+ * @param note what to note with it when it ends (optional)
+ * @return the experiment as it ended, with traffic, waited and
timeLeft
+ */
+ public static JsonObject run(Integration integration, String hypothesis,
String rules, Feed feed, long wait, String note) {
+ try {
+ return run(integration, hypothesis, rules, feed, wait, note,
SYSTEM);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException("Interrupted while waiting for the
verdict of the experiment", e);
+ }
+ }
+
+ static JsonObject run(
+ Integration integration, String hypothesis, String rules, Feed
feed, long wait, String note, Clock clock)
+ throws InterruptedException {
+ JsonObject begun = integration.chaos("experiment-begin", root -> {
+ if (hypothesis != null && !hypothesis.isBlank()) {
+ root.put("hypothesis", hypothesis);
+ }
+ root.put("rule", rules);
+ });
+ int number = begun.getIntegerOrDefault("number", 0);
+ JsonObject traffic = null;
+ long waited = 0;
+ try {
+ if (feed != null) {
+ traffic = traffic(integration, feed);
+ }
+ long start = clock.now();
+ while (true) {
+ JsonObject seen = experiment(integration.status(), number);
+ if (seen != null && (!seen.getBooleanOrDefault("open", true)
+ || !"PENDING".equals(seen.getString("verdict")))) {
+ break;
+ }
+ if (clock.now() - start >= wait) {
+ break;
+ }
+ clock.sleep(POLL);
+ }
+ waited = clock.now() - start;
+ } finally {
+ // ended whatever happened, so its rules do not stay
+ JsonObject result = end(integration, number, note, clock);
+ if (result != null) {
+ begun = result;
+ }
+ }
+ JsonObject answer = new JsonObject(begun);
+ if (traffic != null) {
+ answer.put("traffic", traffic);
+ }
+ answer.put("waited", waited);
+ JsonObject exploration = exploration(integration.status());
+ if (exploration != null && exploration.containsKey("timeLeft")) {
+ answer.put("timeLeft", exploration.getLong("timeLeft"));
+ }
+ return answer;
+ }
+
+ /** Sends the traffic: the summary of the send, or why it could not be
sent. */
+ private static JsonObject traffic(Integration integration, Feed feed) {
+ JsonObject answer;
+ try {
+ answer = integration.send(feed);
+ } catch (RuntimeException e) {
+ answer = new JsonObject();
+ answer.put("status", "failed");
+ answer.put("error", e.getMessage());
+ }
+ if (answer == null) {
+ answer = new JsonObject();
+ answer.put("status", "failed");
+ }
+ if (!answer.containsKey("endpoint")) {
+ answer.put("endpoint", feed.endpoint());
+ }
+ return answer;
+ }
+
+ /** Ends the experiment: what came of it, also when the exploration ended
it at its deadline meanwhile. */
+ private static JsonObject end(Integration integration, int number, String
note, Clock clock)
+ throws InterruptedException {
+ try {
+ return integration.chaos("experiment-end", root -> {
+ if (note != null && !note.isBlank()) {
+ root.put("note", note);
+ }
+ });
+ } catch (RuntimeException e) {
+ // no experiment is open: the exploration ended it, which the next
status says
+ for (int i = 0; i < 3; i++) {
+ JsonObject seen = experiment(integration.status(), number);
+ if (seen != null && !seen.getBooleanOrDefault("open", true)) {
+ return seen;
+ }
+ clock.sleep(POLL);
+ }
+ throw e;
+ }
+ }
+
+ /** The experiment with the number in the report of the status, or null. */
+ static JsonObject experiment(JsonObject status, int number) {
+ JsonObject exploration = exploration(status);
+ List<Object> experiments = exploration != null ?
exploration.getCollection("experiments") : null;
+ if (experiments == null) {
+ return null;
+ }
+ for (Object o : experiments) {
+ if (o instanceof JsonObject experiment &&
experiment.getIntegerOrDefault("number", -1) == number) {
+ return experiment;
+ }
+ }
+ return null;
+ }
+
+ /** The exploration in the report of the chaos console in the status, or
null. */
+ static JsonObject exploration(JsonObject status) {
+ if (status == null || !(status.get("chaos") instanceof JsonObject
chaos)
+ || !(chaos.get("report") instanceof JsonObject report)) {
+ return null;
+ }
+ return report.get("exploration") instanceof JsonObject exploration ?
exploration : null;
+ }
+}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
index dfab28212f77..6a84a4bd54ce 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/action/CamelChaosActionTest.java
@@ -36,8 +36,9 @@ import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/**
- * camel cmd chaos-add, chaos-change, chaos-remove, chaos-clear, chaos-load
and chaos-suggest: the chaos action they
- * send, what they print, and that a refused rule is an error with the reason
of the integration.
+ * camel cmd chaos-add, chaos-change, chaos-remove, chaos-clear, chaos-load,
chaos-suggest, chaos-explore-start,
+ * chaos-explore-finish and chaos-experiment: the chaos action they send, what
they print, and that a refused rule is an
+ * error with the reason of the integration.
*/
@ExtendWith(MockitoExtension.class)
class CamelChaosActionTest extends ActionCommandTestSupport {
@@ -481,6 +482,103 @@ class CamelChaosActionTest extends
ActionCommandTestSupport {
assertThat(printer.getLines()).containsExactly("Removed chaos rules:
3");
}
+ @Test
+ void exploreStart() throws Exception {
+ writeStatusFile(TEST_PID, "orders");
+
+ assertThat(callWithResponse(command(new
CamelChaosExploreStartAction(main()), "orders", "--budget=30m",
+ "--focus=orders"), new JsonObject(Map.of("running", true,
"budget", 1_800_000L, "focus", "orders"))))
+ .isZero();
+
+ JsonObject action = readActionFile(TEST_PID);
+ assertThat(action.getString("command")).isEqualTo("explore-start");
+ assertThat(action.getString("budget")).isEqualTo("30m");
+ assertThat(action.getString("focus")).isEqualTo("orders");
+ assertThat(printer.getLines()).containsExactly("Chaos exploration
started: 30m0s (focus: orders)");
+ }
+
+ @Test
+ void exploreFinish() throws Exception {
+ writeStatusFile(TEST_PID, "orders");
+ JsonObject answer = new JsonObject(
+ Map.of("running", false, "startedAt", 1_000_000L,
"finishedAt", 1_250_000L,
+ "rules", new JsonArray(List.of("r1", "r2", "r3")),
"removed", 1L,
+ "experiments", new JsonArray(
+ List.of(new JsonObject(Map.of("number", 1,
"verdict", "FAIL")),
+ new JsonObject(Map.of("number", 2,
"verdict", "PASS"))))));
+
+ assertThat(callWithResponse(command(new
CamelChaosExploreFinishAction(main()), "orders"), answer)).isZero();
+
+
assertThat(readActionFile(TEST_PID).getString("command")).isEqualTo("explore-finish");
+ assertThat(printer.getLines()).containsExactly(
+ "Chaos exploration finished after 4m10s: 3 rules tried, 1
removed, 2 experiments (1 failed)");
+ }
+
+ @Test
+ void experimentEndsTheOpenOne() throws Exception {
+ writeStatusFile(TEST_PID, "orders");
+ JsonObject answer = new JsonObject(
+ Map.of("number", 3, "open", false, "verdict", "FAIL",
"endedBy", "ended",
+ "hypothesis", "an order never fails", "elapsed",
64_000L, "rules", new JsonArray(List.of("r4")),
+ "expectations", new JsonArray(
+ List.of(new JsonObject(
+ Map.of("expect", "orders:failed<=0",
"state", "unmet", "value", "9 failed")))),
+ "note", "stopped by hand"));
+
+ // a FAIL verdict is exit code 1, as camel run --chaos has it
+ assertThat(callWithResponse(command(new
CamelChaosExperimentAction(main()), "orders", "--end",
+ "--note=stopped by hand"),
answer)).isEqualTo(CliExitCode.FAILURE);
+
+ JsonObject action = readActionFile(TEST_PID);
+ assertThat(action.getString("command")).isEqualTo("experiment-end");
+ assertThat(action.getString("note")).isEqualTo("stopped by hand");
+ assertThat(printer.getLines()).containsExactly(
+ "Experiment 3 FAIL an order never fails (r4, ended after
1m4s)",
+ "expect orders:failed<=0: unmet, 9 failed", "note: stopped by
hand");
+ }
+
+ @Test
+ void experimentNeedsItsRulesAndAFeedForItsMessages() {
+ assertThatThrownBy(() -> command(new
CamelChaosExperimentAction(main()), "orders").doCall())
+
.isInstanceOf(CliException.class).hasMessageStartingWith("--rule is required")
+ .extracting(e -> ((CliException)
e).getExitCode()).isEqualTo(CliExitCode.USAGE);
+ assertThatThrownBy(() -> command(new
CamelChaosExperimentAction(main()), "orders",
+ "--rule=route=orders error expect=orders:failed<=0",
"--repeat=20").doCall())
+ .isInstanceOf(CliException.class).hasMessageContaining("go
with --feed");
+ assertThatThrownBy(() -> command(new
CamelChaosExperimentAction(main()), "orders",
+ "--rule=route=orders error expect=orders:failed<=0",
"--wait=1h").doCall())
+ .isInstanceOf(CliException.class).hasMessage("--wait must be
at most 10m, was: 1h");
+ assertThatThrownBy(() -> command(new
CamelChaosExperimentAction(main()), "orders",
+ "--rule=route=orders error expect=orders:failed<=0",
"--feed=orders", "--interval=500..100").doCall())
+
.isInstanceOf(CliException.class).hasMessageContaining("--interval must go from
the shortest");
+ }
+
+ @Test
+ void experimentPrintsTheTrafficAndTheTimeLeft() {
+ JsonObject experiment = new JsonObject(
+ Map.of("number", 2, "open", false, "verdict", "INCONCLUSIVE",
+ "endedBy", "ended", "elapsed", 125_000L, "rules", new
JsonArray(List.of("r3")),
+ "expectations", new JsonArray(
+ List.of(new JsonObject(
+ Map.of("expect", "orders:100",
"state", "pending", "value", "41 completed")))),
+ "traffic", new JsonObject(
+ Map.of("endpoint", "direct://orders", "sent",
20L, "failed", 2L,
+ "elapsed", 950L)),
+ "waited", 120_000L, "timeLeft", 495_000L));
+
+
assertThat(CamelChaosExperimentAction.lines(experiment)).containsExactly(
+ "Experiment 2 INCONCLUSIVE (r3, ended after 2m5s)",
+ " expect orders:100: pending, 41 completed",
+ " traffic: 20 messages sent to direct://orders (2 failed) in
950ms",
+ " waited 2m0s for the verdict; 8m15s left of the exploration",
+ " An expectation was not decided when it ended: wait longer
than its within= (--wait)");
+
+ JsonObject notSent = new JsonObject(
+ Map.of("number", 3, "verdict", "PASS", "elapsed", 1_000L,
+ "traffic", new JsonObject(Map.of("endpoint",
"refunds", "status", "failed", "error", "No endpoint"))));
+ assertThat(CamelChaosExperimentAction.lines(notSent)).contains("
traffic: not sent to refunds: No endpoint");
+ }
+
/** The status of the integration orders, with the chaos dev console as
the CLI connector puts it there. */
private static void writeChaosStatus(boolean enabled, String... ids)
throws Exception {
JsonObject chaos = new JsonObject();
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
new file mode 100644
index 000000000000..b799e19b21d7
--- /dev/null
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/common/ChaosExperimentRunTest.java
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.dsl.jbang.core.common;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Consumer;
+
+import org.apache.camel.util.json.JsonArray;
+import org.apache.camel.util.json.JsonObject;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * One chaos experiment in one call (CAMEL-26786): begin, traffic, the wait
for the verdict in the status, and the end,
+ * against a fake integration and a clock that does not wait.
+ */
+class ChaosExperimentRunTest {
+
+ /** The integration: what it was asked, and the status it shows, one per
read (the last one stays). */
+ private static final class FakeIntegration implements
ChaosExperimentRun.Integration {
+ final List<JsonObject> actions = new ArrayList<>();
+ final Deque<JsonObject> statuses = new ArrayDeque<>();
+ ChaosExperimentRun.Feed sent;
+ RuntimeException sendFails;
+ RuntimeException endFails;
+ RuntimeException beginFails;
+ JsonObject ended = experiment("FAIL", false);
+
+ @Override
+ public JsonObject chaos(String command, Consumer<JsonObject> options) {
+ JsonObject action = new JsonObject();
+ action.put("command", command);
+ options.accept(action);
+ actions.add(action);
+ if ("experiment-begin".equals(command)) {
+ if (beginFails != null) {
+ throw beginFails;
+ }
+ return experiment("PENDING", true);
+ }
+ if (endFails != null) {
+ throw endFails;
+ }
+ return ended;
+ }
+
+ @Override
+ public JsonObject send(ChaosExperimentRun.Feed feed) {
+ sent = feed;
+ if (sendFails != null) {
+ throw sendFails;
+ }
+ return new JsonObject(Map.of("status", "success", "endpoint",
"direct://refunds", "sent", 20L, "failed", 0L));
+ }
+
+ @Override
+ public JsonObject status() {
+ return statuses.size() > 1 ? statuses.poll() : statuses.peek();
+ }
+
+ void shows(String verdict, boolean open) {
+ statuses.add(statusWith(experiment(verdict, open)));
+ }
+ }
+
+ /** A clock that moves on when it is asked to sleep. */
+ private static final class FakeClock implements ChaosExperimentRun.Clock {
+ long now = 1_000_000;
+ int sleeps;
+
+ @Override
+ public long now() {
+ return now;
+ }
+
+ @Override
+ public void sleep(long millis) {
+ sleeps++;
+ now += millis;
+ }
+ }
+
+ private static JsonObject experiment(String verdict, boolean open) {
+ return new JsonObject(Map.of("number", 1, "verdict", verdict, "open",
open, "rules", new JsonArray(List.of("r1"))));
+ }
+
+ private static JsonObject statusWith(JsonObject experiment) {
+ JsonObject exploration = new JsonObject(
+ Map.of("running", true, "timeLeft", 300_000L, "experiments",
+ new JsonArray(List.of(experiment))));
+ JsonObject report = new JsonObject(Map.of("exploration", exploration));
+ return new JsonObject(Map.of("chaos", new JsonObject(Map.of("enabled",
true, "report", report))));
+ }
+
+ private final FakeIntegration integration = new FakeIntegration();
+ private final FakeClock clock = new FakeClock();
+
+ private static final String RULE = "route=refunds error rate=every:5
expect=refunds:failed<=0 within=30s";
+ private static final ChaosExperimentRun.Feed FEED
+ = new ChaosExperimentRun.Feed("refunds", "file:/feed/", null, 20,
"200ms");
+
+ @Test
+ void theTrafficThenTheWaitUntilTheVerdict() throws Exception {
+ integration.shows("PENDING", true);
+ integration.shows("PENDING", true);
+ integration.shows("FAIL", true);
+
+ JsonObject answer = ChaosExperimentRun.run(integration, "a failed
refund is read again", RULE, FEED, 60_000,
+ "kafka commits a failed record", clock);
+
+ assertThat(integration.actions).extracting(a -> a.getString("command"))
+ .containsExactly("experiment-begin", "experiment-end");
+
assertThat(integration.actions.get(0).getString("hypothesis")).isEqualTo("a
failed refund is read again");
+
assertThat(integration.actions.get(0).getString("rule")).isEqualTo(RULE);
+
assertThat(integration.actions.get(1).getString("note")).isEqualTo("kafka
commits a failed record");
+ assertThat(integration.sent).isEqualTo(FEED);
+ // two reads that were pending, a second apart
+ assertThat(clock.sleeps).isEqualTo(2);
+ assertThat(answer.getString("verdict")).isEqualTo("FAIL");
+ assertThat(answer.getLong("waited")).isEqualTo(2_000);
+ assertThat(answer.getLong("timeLeft")).isEqualTo(300_000);
+ assertThat(((JsonObject)
answer.get("traffic")).getLong("sent")).isEqualTo(20);
+ }
+
+ @Test
+ void theWaitIsUpBeforeTheVerdict() throws Exception {
+ integration.shows("PENDING", true);
+ integration.ended = experiment("INCONCLUSIVE", false);
+
+ JsonObject answer = ChaosExperimentRun.run(integration, null, RULE,
null, 5_000, null, clock);
+
+ assertThat(answer.getString("verdict")).isEqualTo("INCONCLUSIVE");
+ assertThat(answer.getLong("waited")).isEqualTo(5_000);
+ assertThat(answer).doesNotContainKey("traffic");
+ assertThat(integration.actions.get(0)).doesNotContainKey("hypothesis");
+ assertThat(integration.actions.get(1)).doesNotContainKey("note");
+ }
+
+ @Test
+ void endedWhenTheTrafficCannotBeSent() throws Exception {
+ integration.shows("PENDING", true);
+ integration.sendFails = new IllegalArgumentException("Header must be
in key=value format, was: x");
+ integration.ended = experiment("INCONCLUSIVE", false);
+
+ JsonObject answer = ChaosExperimentRun.run(integration, null, RULE,
FEED, 2_000, null, clock);
+
+ JsonObject traffic = (JsonObject) answer.get("traffic");
+ assertThat(traffic.getString("status")).isEqualTo("failed");
+ assertThat(traffic.getString("error")).isEqualTo("Header must be in
key=value format, was: x");
+ assertThat(traffic.getString("endpoint")).isEqualTo("refunds");
+ assertThat(integration.actions).extracting(a ->
a.getString("command")).contains("experiment-end");
+ }
+
+ @Test
+ void theExplorationEndedItAtItsDeadline() throws Exception {
+ integration.shows("PENDING", true);
+ // ended by the exploration: the end is refused, and the status has
what came of it
+ integration.shows("FAIL", false);
+ integration.endFails = new IllegalStateException("No experiment is
open");
+
+ JsonObject answer = ChaosExperimentRun.run(integration, null, RULE,
null, 60_000, null, clock);
+
+ assertThat(answer.getString("verdict")).isEqualTo("FAIL");
+ assertThat(answer.getBoolean("open")).isFalse();
+ }
+
+ @Test
+ void aRefusedExperimentIsNotEnded() {
+ integration.beginFails = new IllegalStateException("Experiment 1 is
open: end it first");
+
+ assertThatThrownBy(() -> ChaosExperimentRun.run(integration, null,
RULE, FEED, 60_000, null, clock))
+ .hasMessage("Experiment 1 is open: end it first");
+ assertThat(integration.actions).hasSize(1);
+ assertThat(integration.sent).isNull();
+ }
+
+ @Test
+ void theWait() {
+
assertThat(ChaosExperimentRun.waitMillis(null)).isEqualTo(ChaosExperimentRun.DEFAULT_WAIT);
+ assertThat(ChaosExperimentRun.waitMillis("90s")).isEqualTo(90_000);
+ assertThatThrownBy(() ->
ChaosExperimentRun.waitMillis("11m")).hasMessageContaining("at most 10m");
+ assertThatThrownBy(() ->
ChaosExperimentRun.waitMillis("soon")).hasMessageContaining("not a time");
+ }
+}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
b/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
index a4b2b6dd9b2d..2900be01a02a 100644
--- a/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
+++ b/dsl/camel-jbang/camel-jbang-core/src/test/resources/golden/cli-help.txt
@@ -551,46 +551,54 @@ Performs commands in the running Camel integrations, such
as start/stop route,
or change logging levels. (use cmd --help to see sub commands)
-h, --help Display the help and sub-commands
Commands:
- browse Browse pending messages on endpoints
- chaos-add Add chaos rules to a running integration (camel-chaos, for
- development only)
- chaos-change Change a chaos rule of a running integration (camel-chaos,
- for development only)
- chaos-clear Remove all chaos rules of running integrations
- (camel-chaos, for development only)
- chaos-load Load a chaos scenario into a running integration
- (camel-chaos, for development only)
- chaos-remove Remove a chaos rule of a running integration (camel-chaos,
- for development only)
- chaos-suggest Suggest chaos rules to try in a running integration, per
- route (camel-chaos, for development only)
- disable-processor Disable Camel processor
- enable-processor Enable Camel processor
- gc Trigger Java Memory Garbage Collector
- heap-dump Write a heap dump (.hprof) file for deep memory analysis
- heap-histogram Display class-level heap memory usage in a running Camel
- integration
- memory-leak Diagnose memory leaks in a running Camel integration
- load Loads new source files into an existing Camel
- logger List or change logging levels
- receive Receive and dump messages from remote endpoints
- reload Trigger reloading Camel
- reset-stats Reset performance statistics
- resume-route Resume Camel routes
- route-diagram Display Camel route diagram in the terminal
- route-dump Dump Camel route in XML, YAML, or Java DSL format
- route-structure Dump Camel route structure
- route-topology Display inter-route topology connections
- send Send messages to endpoints
- sql Execute SQL query on a DataSource
- span Display OpenTelemetry spans from running Camel
integrations
- start-group Start Camel route groups
- start-route Start Camel routes
- stop-group Stop Camel route groups
- stop-route Stop Camel routes
- stub Browse stub endpoints
- suspend-route Suspend Camel routes
- thread-dump List threads in a running Camel integration
+ browse Browse pending messages on endpoints
+ chaos-add Add chaos rules to a running integration (camel-chaos,
+ for development only)
+ chaos-change Change a chaos rule of a running integration
+ (camel-chaos, for development only)
+ chaos-clear Remove all chaos rules of running integrations
+ (camel-chaos, for development only)
+ chaos-experiment Run one chaos experiment in the exploration of a
+ running integration: add its rules, send its traffic,
+ wait for the verdict, and end it
+ chaos-explore-finish Finish the chaos exploration of a running integration
+ before its deadline
+ chaos-explore-start Start a chaos exploration in a running integration: a
+ time to try chaos rules in, that camel-chaos enforces
+ chaos-load Load a chaos scenario into a running integration
+ (camel-chaos, for development only)
+ chaos-remove Remove a chaos rule of a running integration
+ (camel-chaos, for development only)
+ chaos-suggest Suggest chaos rules to try in a running integration,
+ per route (camel-chaos, for development only)
+ disable-processor Disable Camel processor
+ enable-processor Enable Camel processor
+ gc Trigger Java Memory Garbage Collector
+ heap-dump Write a heap dump (.hprof) file for deep memory
analysis
+ heap-histogram Display class-level heap memory usage in a running
+ Camel integration
+ memory-leak Diagnose memory leaks in a running Camel integration
+ load Loads new source files into an existing Camel
+ logger List or change logging levels
+ receive Receive and dump messages from remote endpoints
+ reload Trigger reloading Camel
+ reset-stats Reset performance statistics
+ resume-route Resume Camel routes
+ route-diagram Display Camel route diagram in the terminal
+ route-dump Dump Camel route in XML, YAML, or Java DSL format
+ route-structure Dump Camel route structure
+ route-topology Display inter-route topology connections
+ send Send messages to endpoints
+ sql Execute SQL query on a DataSource
+ span Display OpenTelemetry spans from running Camel
+ integrations
+ start-group Start Camel route groups
+ start-route Start Camel routes
+ stop-group Stop Camel route groups
+ stop-route Stop Camel routes
+ stub Browse stub endpoints
+ suspend-route Suspend Camel routes
+ thread-dump List threads in a running Camel integration
== camel cmd browse --help
Usage: camel cmd browse [-h] [--dump] [--fresh-size] [--json] [--mask]
@@ -741,6 +749,129 @@ Examples:
camel cmd chaos-clear myApp
camel cmd chaos-clear myApp --reset
+== camel cmd chaos-experiment --help
+Usage: camel cmd chaos-experiment [-h] [--end] [--body=<body>] [--feed=<feed>]
+ [--hypothesis=<hypothesis>]
+ [--interval=<interval>] [--note=<note>]
+ [--repeat=<repeat>] [--timeout=<timeout>]
+ [--wait=<wait>] [--header=<headers>]...
+ [--rule=<rules>]... [<name>]
+Run one chaos experiment in the exploration of a running integration: add its
+rules, send its traffic, wait for the verdict, and end it
+ [<name>] Name or pid of the running Camel integration (must
+ match one); without it, the only one running, or
+ the only one with chaos switched on
+ -h, --help Display the help and sub-commands
+ --timeout=<timeout> How long to wait for the integration to answer, in
+ millis or as a duration such as 30s
+ Default: 10000
+ --hypothesis=<hypothesis>
+ What the experiment tests, in a few words, such as
+ 'orders survive a payment outage'
+ --rule=<rules> A rule in its one-line text form, as an entry of
+ camel.chaos.rules; can be given more than once,
+ and one of them needs an expectation (expect=)
+ --feed=<feed> The endpoint, or the id of the route, to send the
+ messages of the experiment to
+ --body=<body> The body of the messages: text, file:<path>, or
+ file:<folder> for a message per file
+ --header=<headers> A header of the messages as key=value; can be given
+ more than once
+ --repeat=<repeat> How many messages to send in all (at most 10000)
+ Default: 0
+ --interval=<interval> The pause between two messages, such as 200ms, or a
+ range such as 100..500 to draw each pause at
+ random (at most 60s)
+ --wait=<wait> The longest wait for the verdict after the messages
+ were sent (at most 10m)
+ Default: 2m
+ --note=<note> What to note with the experiment when it ends
+ --end End the open experiment (with --note) instead of
+ running a new one
+
+An experiment is part of an exploration (camel cmd chaos-explore-start). It
+tests a
+hypothesis with rules, one of them with an expectation (expect=) that decides
+it.
+The rules are added together, the messages of --feed are sent (as camel cmd
send
+sends them, with --body, --header, --repeat and --interval), and then it waits
+until
+the expectations are decided, or --wait is up. The experiment is ended then,
its
+rules are removed, and what came of it is printed: the verdict, the
+expectations,
+what the routes did, and the traffic. It stays in the journal of the
exploration
+(camel get chaos --report).
+
+The verdict is FAIL when an expectation is unmet (exit code 1), PASS when all
+are
+met, and INCONCLUSIVE when it ended before an expectation was decided: wait
+longer
+than the within= of the expectation. --feed can be the id of a route, as the
+feed:
+line of camel cmd chaos-suggest has it.
+
+Examples:
+ camel cmd chaos-experiment --hypothesis='a failed refund is read again' \
+ --rule='route=refunds error rate=every:5 expect=refunds:failed<=0
+within=30s' \
+ --feed=refunds --body=file:test/chaos/refunds-fails/ --repeat=20
+--interval=200ms
+ camel cmd chaos-experiment myApp --hypothesis='orders survive a payment
+outage' \
+ --rule='endpoint=http:payment* error status=503 until=1m expect=orders:
+failed<=0 within=70s' \
+ --wait=90s
+ camel cmd chaos-experiment --end --note='stopped by hand'
+
+== camel cmd chaos-explore-finish --help
+Usage: camel cmd chaos-explore-finish [-h] [--timeout=<timeout>] [<name>]
+Finish the chaos exploration of a running integration before its deadline
+ [<name>] Name or pid of the running Camel integration (must
+ match one); without it, the only one running, or
+ the only one with chaos switched on
+ -h, --help Display the help and sub-commands
+ --timeout=<timeout> How long to wait for the integration to answer, in
+ millis or as a duration such as 30s
+ Default: 10000
+
+The rules are removed (an open experiment is ended with what came of it so
+far), and
+the journal stays in the report (camel get chaos --report) until the next
+exploration starts or the report is reset.
+
+Examples:
+ camel cmd chaos-explore-finish
+ camel cmd chaos-explore-finish myApp
+
+== camel cmd chaos-explore-start --help
+Usage: camel cmd chaos-explore-start [-h] [--budget=<budget>] [--focus=<focus>]
+ [--timeout=<timeout>] [<name>]
+Start a chaos exploration in a running integration: a time to try chaos rules
+in, that camel-chaos enforces
+ [<name>] Name or pid of the running Camel integration (must
+ match one); without it, the only one running, or
+ the only one with chaos switched on
+ -h, --help Display the help and sub-commands
+ --timeout=<timeout> How long to wait for the integration to answer, in
+ millis or as a duration such as 30s
+ Default: 10000
+ --budget=<budget> How long the exploration runs, between 10s and 2h
+ Default: 10m
+ --focus=<focus> What the exploration is about, such as a route
+
+While the exploration runs, camel-chaos keeps its journal: the baseline (the
+counts
+per route before its first rule), the rules tried, and the experiments
+(camel cmd chaos-experiment). A rule ends at the deadline at the latest, and at
+the
+deadline the rules are removed and the exploration is finished, whatever
+happens.
+One exploration runs at a time. camel get chaos --report shows it.
+
+Examples:
+ camel cmd chaos-explore-start
+ camel cmd chaos-explore-start myApp --budget=30m --focus=orders
+
== camel cmd chaos-load --help
Usage: camel cmd chaos-load [-h] [--timeout=<timeout>] <file> [<name>]
Load a chaos scenario into a running integration (camel-chaos, for development
