[
https://issues.apache.org/jira/browse/CAMEL-26607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139952#comment-18139952
]
Claus Ibsen commented on CAMEL-26607:
-------------------------------------
Ideas for stage 2, written down so we can walk them one by one. Each item names
where it would go in the code, and gives a rough size (S/M/L). Everything is
based on main as of today (stage 1 plus CAMEL-26674..26679, 26695, 26698, 26699
and 26701).
h3. Decide before 4.24.0 ships
{{SendHook}} and {{StepHook}} are both {{@since 4.24}}, so neither SPI has been
released yet. Two stage 2 items need a change to them, and those changes cost
nothing until the release. After it, they become API changes:
* *the {{duplicate}} fault* (P-1) needs a "send again" result in
{{SendHook.beforeSend}}. {{SendHelper.HookedSend.send()}} already reserves the
negative values other than {{SKIP}} ("negative values other than SKIP are
reserved"), so a {{SendHook.SEND_TWICE = -2}} fits without breaking anything.
* *more than one hook*. {{ChaosManager}} logs "A StepHook is installed already
..., so chaos rules do not fire at steps" and steps aside. Chaos, the proposed
exchange sanitizer and anything else that wants a hook have to be able to run
together (debugger, tracer, sanitizer). Options:
{{ExtendedCamelContext.addStepHook/removeStepHook}} with a list inside, or a
{{CompositeStepHook}} in camel-support that the context holds. The same
question applies to {{SendHook}}.
h3. Stage 2 scope
*S2-1. Errors at the route entry* (size M)
* Syntax: {{route=orders error rate=0.05}}, {{route=orders error
exception=java.net.ConnectException}}. Also {{delay}} and {{drop}} at the
entry: {{route=orders delay duration=2s}}.
* Why: event-driven consumers (Kafka, JMS, platform-http) have no {{at=poll}},
so today their consumer side can't be faulted at all. For platform-http, an
entry error is what the HTTP client sees. A rule at the entry also fires once
per exchange, which is the honest meaning of "fail 5% of the order route".
{{step=* rate=0.05}} would compound the rate over every step.
* Where: {{ChaosRuleParser.offered(ROUTE)}} adds ERROR, DELAY (and maybe DROP)
to {{Hook.ENTRY}}, and {{hint()}} loses its "route= does not reach the sends
and steps" text for errors. {{ChaosRoutePolicy.onExchangeBegin}} already meets
each exchange once per route ({{ChaosExchangeState.enter}}).
* Open question, *where the error happens*:
** (a) Set the exception in {{onExchangeBegin}}. The pipeline stops before the
first step, so the route's {{onException}}, redelivery and DLC do *not* apply.
The consumer sees a failed exchange (a JMS rollback, a file not moved, an HTTP
500).
** (b) Mark the exchange in the policy ({{ChaosExchangeState}}), and let
{{ChaosStepHook.beforeStep}} raise the error at the next step it meets, which
is the first step of the route. The error handler then applies, as for a real
failure inside the route.
** Proposal: (b) by default, because that's what people mean by "the route
fails", and (a) as {{at=consumer}} for testing the consumer's own failure
handling. Both need a test per error handler: default, DLC, noErrorHandler,
transacted.
* Risks: {{delay}} at the entry runs on the consumer thread, so it throttles
the consumer. That's fine, but it should be said in the guide. {{drop}} at the
entry means the route never runs, so it is close to {{filter}}. It may not be
worth offering.
*S2-2. "The sends and steps of route X"* (size S-M)
* Syntax: a second selector, {{in=<route pattern>}}, that combines with the
targets we already have: {{endpoint=http:* in=orders error status=503}},
{{step=* in=orders delay duration=500ms}}, {{component=kafka in=order* drop}}.
* Why: people know route ids, but not node ids ({{to3}}, {{log2}} are
generated), nor every endpoint URI a route calls. {{in=}} keeps one target per
rule and narrows it to a route.
* Where: {{ChaosRule.matchesRoute(String)}}, asked in {{ChaosSendHook.fire}} /
{{afterSend}} and in {{ChaosStepHook.beforeStep}} with
{{ChaosExchangeState.routeId(exchange)}} (the route of the unit of work, which
is the current route in a {{direct:}} call). {{ChaosTargets.check}} should warn
when no route matches {{in=}}, or when the target matches nothing inside those
routes. {{ChaosModel}} gets the new option, and so do the recipes (the TUI form
can prefill {{in=}} from the selected route).
* Open questions: should {{in=}} also match the routes the route calls
({{direct:}}, {{seda:}})? Proposal: no, that's what {{in=order*}} is for. The
poll hook has no route of the exchange, only the route of the consumer, so for
{{at=poll}} {{in=}} should mean the route that consumes.
*S2-3. A condition on the message: {{when=}}* (size M)
* Syntax: {{endpoint=http:stock* error status=503 when="${header.country} ==
'ES'"}}. The simple language only, in quotes. The {{or}} from the ticket's
prompt stays two rules.
* Where: the parser already reserves {{when=}} ({{NOT_SUPPORTED}}). The
predicate is created once when the rule is parsed (the context's simple
language), and checked in the hooks *before* {{ChaosRule.fires(now)}}. That way
{{matches}}, {{rate}} and {{until=<count>}} count only the exchanges that meet
the condition, and a seed gives the same faults for the same matching messages.
* Not allowed where there is no exchange: {{at=poll}}, {{stall}} and
{{lifecycle}}. The parser should say so.
* Risks:
** Reading a stream body in the predicate consumes it. Reset the stream cache
after the check (as {{MessageHelper.resetStreamCache}} does), or refuse a body
condition when stream caching is off.
** A predicate that throws should count as "not matched" and show up in
{{skippedReasons}} (CAMEL-26676), not fail the exchange.
** Performance: one predicate per send in dev mode only. That's acceptable, but
it should be in the doc.
** Rules only come from {{camel.chaos.rules}} and the CLI connector (the local
developer), as stage 1 already decided. The AI path shows the rule for
confirmation (S2-5).
*S2-4. Leave out the integration name when only one runs* (size S)
* {{camel cmd chaos-add --rule='...'}} with no name. The positional parameter
of {{CamelChaosAddAction}} / {{CamelChaosChangeAction}} /
{{CamelChaosRemoveAction}} / {{CamelChaosClearAction}} becomes {{arity =
"0..1"}}. With no name and exactly one integration running, that one is used.
With none, or more than one, the command fails with the list (exit code 2,
usage). The MCP tools already take {{nameOrPid}}, so they follow the same rule.
* Unlike {{camel cmd gc}}, it must never mean "all of them" for add or change.
*S2-5. Loose prompts through AI, with confirmation* (size M)
Most of the pieces are already there: the vocabulary in the catalog
({{CamelCatalog.chaosModelJSon()}}, CAMEL-26689), the confirmation of
{{chaos_add}}/{{chaos_change}} in {{camel ask}} ({{ToolConfirms.chaos}}), and
the warnings for a rule that matches nothing (CAMEL-26676). What's missing:
* *A check without adding*: {{camel cmd chaos-add --dry-run}}, plus an MCP tool
{{camel_runtime_chaos_check}}. Both parse the rule, run {{ChaosTargets.check}}
and return what it *would* hit (the endpoints, routes and steps it matches, not
only the warnings). The AI can then show "this rule fails about 5% of the
exchanges entering route orders, for 5 minutes, and hits: orders" before it
asks. Where: a {{check}} command next to {{add}} in the chaos action of the CLI
connector, and a {{ChaosManager.checkText(String)}} that parses without
registering.
* *The rule in plain words*, in camel-chaos and without an AI: {{ChaosModel}}
or {{ChaosRule}} renders a rule as one sentence. The confirmation, the TUI
table, the report and the dry-run all use it, so a person can check that the
rule is what they meant.
* *An eval set*: 15-20 loose prompts with the rules we expect, the ticket's
example among them. Run them against the MCP tools with a small local model, as
we do for the other tool groups. This is how we find out whether the vocabulary
is enough, or whether the tool descriptions need a "split an or into two rules"
hint.
* TUI: a "describe the chaos you want" input on the chaos screen, which opens
the AI panel with a prompt built like {{ChaosAiPrompt}} (CAMEL-26702).
h3. Parked items: keep or drop
* *P-1. {{duplicate}} fault* ({{endpoint=kafka:orders* duplicate rate=0.05}}).
*Keep, before 4.24.0.* This is the one fault that tests idempotent consumers.
It needs {{SendHook.SEND_TWICE}}: {{SendHelper}} calls the producer a second
time after the first callback, on the same thread, one after the other. That
respects one exchange, one thread at a time. For InOut, the second reply wins,
and the doc says so. The copy-and-resend alternative (a {{ProducerTemplate}} in
{{afterSend}}) gets a new exchange id and bypasses the hooks, so it's worse.
Size M (core plus chaos).
* *P-2. {{hooks: ["send","poll"]}} for {{direct:}}* (the list shows the hooks
the target type offers, not the ones that apply). *Keep, small.* Intersect the
hooks with what {{ChaosTargets}} finds: poll only when a matching route
consumer is a {{ScheduledPollConsumer}}. Size S.
* *P-3. YAML form of the rules.* *Drop for now.* The one-line form works the
same in properties, the CLI, MCP and the TUI, and a second grammar doubles the
parser and docs. If scenario files (W-1) need more structure, they can still be
lines of rules.
* *P-4. Tie attempts of the same message by a key* (file name, JMSMessageID,
Kafka offset). *Keep, but later.* A broker redelivery is a new exchange, so
today the report counts one message that failed twice and then completed as 2
failed + 1 completed. A key per component ({{CamelFileName}}, {{JMSMessageID}},
{{CamelKafkaPartition}}+{{CamelKafkaOffset}}), with an optional {{key=}}
override, would let the report say "lost: 0, recovered after 2 broker
redeliveries". Size M-L.
* *P-5. Fault SPI* (custom faults in a JAR). *Drop.* It was a non-goal in the
design, and nobody has asked for it. Revisit when someone does.
h3. Wider ideas
* *W-1. Scenarios: named, saved, replayable.* A file of rules
({{chaos/outage.chaos}}: one rule per line, plus {{seed=}}), with
{{from=}}/{{until=}} giving the phases. That's enough for a "game day" script,
so no new timing syntax is needed. {{camel cmd chaos-load <file>}}, {{camel run
--chaos=outage.chaos}}, and "save the current rules" in the TUI. Why: an
experiment becomes repeatable and shareable in the repo, not something typed
once. Size M.
* *W-2. CI mode: exit code by verdict.* {{camel run --chaos=<scenario>
--max-seconds=120}} ends with exit 0 for PASS and 1 for FAIL (the exit codes of
CAMEL-25388), and writes the JSON report to a file
({{--chaos-report=target/chaos.json}}). Why: chaos can then gate a pipeline.
{{reportOnShutdown}} only logs today. Size S-M.
* *W-3. Chaos in tests.* A small helper for camel-test-junit5 /
camel-test-main-junit5: rules from an annotation or a method, and
{{ChaosAssertions.assertVerdict(context, PASS)}} / {{assertExpectation(...)}}.
The doc example uses {{camel.chaos.seed}}. Also a Citrus action for {{camel
test run}}. Why: a finding from a hands-on session becomes a regression test.
Size M.
* *W-4. Make a test from a chaos run.* From the report (rule, seed,
expectation, the routes it hit), generate a route test on top of W-3. As a
follow-up step of "Explain with AI" ("make a test of it"), using
{{camel_route_test_scaffold}}. Size M, after W-3.
* *W-5. Reproduce a real error as a rule.* On the Errors tab, take a real error
(exception type, node or endpoint, route) and offer "make a chaos rule of
this": {{step=callStock error exception=<type> rate=1 until=10}}. Why: you fix
the bug and then prove the fix against the same failure, without waiting for it
to happen again. Size S-M (TUI plus the rule text).
* *W-6. What should I try?* Rank recipes per route from the route structure: an
http call without a timeout or circuit breaker → {{timeout}} and
{{http-error}}; a route with a DLC → {{dlq-unreachable}}; an event-driven
consumer → S2-1. This shares the curated rule set we talked about for the About
screen hints (CAMEL-26700), so it should be one rule set, not two. Size M.
* *W-7. What has been tried.* A per-route table in the report and the chaos
screen: which fault classes (downstream error, slow, lost, corrupt, consumer,
lifecycle, DLC) were tried, the outcome, and which weren't tried yet. Why: it
tells you when you are done, and it pairs with W-6. Size S-M.
* *W-8. Expectations on time and loss.* {{expect=orders:meanTime<2s}} (or p99,
if the route statistics have it), and end to end
{{expect=orders->shipping:lost=0}}, which compares the completed counts of two
routes. Why: the delay recipes have nothing to check today, and "nothing was
lost between two routes" is the question people actually ask. Where:
{{ChaosRule.Expectation.Kind}}. Size M.
* *W-9. Timeline layer* (CAMEL-26691). The faults per second of each rule as a
layer of the cause band, with down windows shaded, and key {{g}} on the chaos
screen. The timeline is being built now. Chaos only needs {{faultsPerSecond}},
which the live state (CAMEL-26678) already has.
* *W-10. Spring Boot / Quarkus* (CAMEL-26479, In Progress). Keep it as planned:
the exported project gets the dependency and Camel profile {{dev}}, and the
starter/extension binds {{camel.chaos.*}}. Without it, rules can only come from
the CLI connector.
h3. Recommended order
# Before 4.24.0: the multi-hook decision in camel-core, and P-1 {{duplicate}}
with {{SEND_TWICE}}. Both are SPI changes that are free until the release.
# S2-4 (name optional) and P-2 (hooks listing). Small, and they make the CLI
nicer right away.
# S2-2 {{in=}}, then S2-1 route-entry errors (on top of {{in=}} and the step
hook).
# S2-3 {{when=}}.
# S2-5: dry-run check, the rule in plain words, and the eval set. The eval set
is what tells us whether S2-1..S2-3 were enough.
# Then W-1 scenarios + W-2 CI mode, and W-3 tests (+ W-4).
h3. Questions for Claus
# Multi-hook: a list on {{ExtendedCamelContext}} or a composite in
camel-support, and should it happen before 4.24.0?
# S2-1: by default, should the entry error go through the route's error handler
(b), or fail the consumer directly (a)?
# S2-2: is {{in=}} a good name? Alternatives: {{route=}} as a filter next to
another target, or {{scope=}}.
# S2-3: only the simple language for {{when=}}, or any predicate language?
# P-1: is {{duplicate}} worth an SPI change in 4.24.0, or should it wait for
4.25?
# W-1/W-2: should scenario files go in the project ({{chaos/*.chaos}}), and
should a FAIL verdict give exit code 1?
_Claude Code on behalf of Claus Ibsen_
> camel-chaos - stage 2: more flexible rules and conditions (errors at the
> route entry, a when= condition on the message, loose prompts through AI)
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26607
> URL: https://issues.apache.org/jira/browse/CAMEL-26607
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> Stage 2 of camel-chaos (stage 1 is CAMEL-26330 and its follow-ups, all in
> 4.24.0; design in {{design/chaos.adoc}}).
> Stage 1 puts chaos at a specific place: an endpoint, a step, a node, a poll,
> a route entry or the route lifecycle. Stage 2 makes the rules more flexible,
> so that a person can prompt loosely and the AI ({{camel ask}}, the
> camel-jbang MCP server) works out the rules. For example:
> {quote}the order route should have chaos, by failing 5% or when a message
> origin country is spain{quote}
> Against stage 1 this prompt hits two gaps:
> # *An error at the route entry.* {{route=orders error}} is rejected today:
> the route entry offers only {{corrupt}}, and {{route=}} matches the route
> entry and the lifecycle only, not the sends and steps of the route.
> # *A content condition.* A rule fires by {{rate}} (a probability or
> {{every:N}}) only. There is no condition on the message, such as
> {{when=<predicate>}} on a header or the body (origin country is Spain).
> The "or" in the prompt can stay two rules (one with {{rate=0.05}}, one with
> {{when=...}} and no rate).
> Scope (to decide after the hands-on round):
> * Errors (and possibly other faults) at the route entry.
> * A filter for "the sends or steps of route X", so that a rule can target
> everything a route does.
> * A {{when=}} condition on the message, evaluated only in dev mode as the
> rest of chaos.
> * Omit the integration name in {{chaos-add}}/{{chaos-change}} when only one
> integration runs.
> * Check that the AI tools can turn loose prompts like the one above into
> rules, and show the rules for confirmation before adding them (as {{camel
> ask}} does today).
> Other candidates from the design, not committed: a YAML form of the rules,
> tying attempts of the same message by a key in the report (file name,
> JMSMessageID, Kafka offset), custom fault types through a plugin SPI.
> Before this starts: try stage 1 hands-on from the CLI, an AI and the terminal
> UI, and collect what is intuitive and what is not. Those findings feed into
> this ticket.
> Chaos must keep working together with the other dev features (debugger,
> tracer, the proposed exchange sanitizer and so on), see the single
> {{StepHook}} slot of CAMEL-26338.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
