[
https://issues.apache.org/jira/browse/CAMEL-26667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen reassigned CAMEL-26667:
-----------------------------------
Assignee: Claus Ibsen
> camel-chaos - findings from the hands-on round of stage 1: report accuracy,
> feedback on rules, live rule state, new faults
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26667
> URL: https://issues.apache.org/jira/browse/CAMEL-26667
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> h2. Hands-on round 1 of camel-chaos (stage 1): findings and ideas
> Stage 1 of camel-chaos (CAMEL-26330) was tried for real with {{camel run
> --chaos}}, driven from the Camel CLI ({{camel cmd chaos-*}}, {{camel get
> chaos}}) and from the MCP tools ({{camel_runtime_chaos_*}}), against a small
> app: a timer route {{orders}} with redelivery (2 attempts, 100ms) and a dead
> letter channel to {{seda:dlq}}, an HTTP call to a local {{platform-http}}
> route {{stock}}, a {{direct}} route with named steps, a {{seda}} hop to
> {{shipping}} and a file poller.
> Every hook works as documented: HTTP {{status=503}} faults with redelivery
> and the DLC, {{step=}} error and delay, {{at=poll drop}}, {{route=}}
> {{corrupt}} at the entry, {{stall}} and {{lifecycle action=stop}},
> expectations, the shutdown report, and the parser refuses wrong rules with
> clear reasons that say what to use instead. This issue collects what felt
> wrong or missing while using it, as input before stage 2 (CAMEL-26607) and
> the chaos screen (CAMEL-26608).
> h3. Usability problems
> # *An expectation can be met while every message went to the DLQ.*
> {{expect=orders:20}} with {{endpoint=http://...stock* error status=503}}
> (every send fails) is *met*: a handled exchange counts as completed (as
> documented). The honest check was {{expect=shipping:20}} on the downstream
> route (FAIL: "route shipping completed 0 of 20 exchanges within 8s: 20
> lost"). Users and AI agents will write the first form.
> # *No expectation on the error handler itself.* "N messages should end in the
> DLQ" works only when a route consumes the DLQ ({{expect=dlq:5}} with
> {{seda:dlq}} -> route {{dlq}}). It cannot be said for a DLC to {{log:}},
> {{mock:}} or {{kafka:dlq}}, and the most common resilience check, "no message
> reaches the DLQ" (at most 0), cannot be said at all. And when such an
> expectation is unmet, the report says "5 lost", while the 5 messages were in
> fact recovered by redelivery: the wording is the opposite of what happened.
> The per-route {{handled}} count is already in the report, so see idea 1.
> # *A message lost inside the dead letter channel is not reported.*
> {{component=seda error until=6}} also failed the send to {{seda:dlq}}; with
> {{deadLetterHandleNewException}} (the default) that exchange counts as
> handled, but never reached the DLQ ({{orders handled 11}}, {{dlq completed
> 10}}, a WARN from DeadLetterChannel). The report says {{lost: 0}} for it. The
> docs warn about {{endpoint=*}} and {{component=}} hitting the DLC, but chaos
> knows the send is the DLC send (the failure endpoint) and could report it
> ("r6 failed the send to the dead letter channel of route orders: 1 message
> gone"), or skip DLC sends unless asked for ({{at=dlq}}).
> # *A rule that matches nothing gives no feedback.*
> {{endpoint=htp://localhost:8765/stock*}} (typo) and {{step=callStok}} were
> accepted with only an id, and fired 0 times. The expectation
> {{expect=shiping:10}} (no such route) logs a WARN in the application log, but
> the CLI and the MCP result only return {{\{"id":"r3"\}}}, so an AI agent
> never sees it. Suggest: {{chaos-add}} / {{chaos-change}} return warnings:
> "matches no endpoint / step / route of the context now", "expects route
> shiping, which does not exist", and the MCP tools pass them through.
> # *{{chaos-clear}} does not start a fresh experiment.* After clear the old
> FAIL verdict, {{lost}} and the history stay until the application restarts.
> Suggest {{chaos-clear --reset}} (or {{chaos-reset}}) that also resets the
> report, the route counts and the verdict (a new "session", optionally a new
> seed).
> # *A removed rule with a pending expectation keeps the report not final for
> good.* Removing the typo rule {{r3}} (pending expectation) made the report
> "FAIL (not final: an expectation is pending)" for the rest of the run. As
> documented, but in practice a removed rule's expectation should become
> {{cancelled}} (not counted, not pending).
> # *"skipped" has no reason.* {{route=shipping corrupt mode=charset}} was
> skipped 79 of 79 times: the body after the HTTP call is a stream cache, which
> {{charset}} does not change ({{mode=truncate}} worked, the body became
> {{in-s}}). Neither the CLI table, the JSON, nor the dev console say why, so
> it looks like it worked. Suggest a count per skip reason ({{skipped:
> \{"stream body (charset)": 79\}}}), and a hint in the table.
> # *Minor:* the JSON report lists the versions of one id as 3, 1, 2; a rule on
> {{direct:}} lists {{hooks: ["send","poll"]}} although direct has no polls
> (the hooks the target offers, not what applies).
> h3. Live rule state (for the TUI chaos screen, CAMEL-26608)
> Today a running rule shows {{matches}}, {{fired}} (per hook in JSON),
> {{skipped}}, {{ends in}}, and a pending expectation {{completed}} / {{time
> left}}. For "fail 5%" that is {{117 matches, 6 fired}}. What a person (and
> the chaos screen) needs to see while it runs:
> * the observed rate next to the target ({{5.1% of 5%}}), and for {{every:N}}
> the next match that fires;
> * when it last fired, and a short ring buffer of the last faults: time,
> exchange id, route, endpoint/step, fault (so a fault can be matched to a DLQ
> entry, a trace or the message history);
> * the effect of the rule, not only its faults: of the exchanges it faulted,
> how many were recovered by redelivery, handled by the DLC, failed, still in
> flight. In the run, {{r7}} (stop {{stock}} for 3s) sent 5 orders to the DLQ,
> because 2 x 100ms redelivery is too short for a 3s outage: today the report
> shows {{orders handled +5}} but not that r7 caused it;
> * a rate over time (faults per second over the last minute) for a sparkline;
> * mark the exchange: a faulted exchange gets a property (e.g.
> {{CamelChaosRule=r1}}) so the tracer, message history, the DLQ route and
> {{camel get history}} show "faulted by chaos rule r1".
> h3. Ideas for new rules and features
> # *Expectations on the error handler:* {{expect=orders:handled<=0}}
> (redelivery must recover everything), {{expect=orders:handled=5}},
> {{expect=orders:redeliveries<=30}}, {{expect=orders:failed=0}}; the count
> kinds already exist per route.
> # *Error after the send* ({{at=reply}} or {{after=true}}): the producer is
> called, then the send fails, as a timeout after the remote side did the work;
> the classic duplicate test for idempotent consumers and retries (a payment
> that is retried after it went through).
> # *Duplicate:* {{endpoint=kafka:orders* duplicate}} sends the message twice
> (at-least-once delivery), to test the idempotent consumer.
> # *Burst / outage windows:* {{rate=burst:5s/30s}} fails everything for 5s
> every 30s, closer to a real outage than an even rate, and what circuit
> breakers react to.
> # *Slow then fail:* {{delay duration=30s}} combined with an error ({{error
> after=30s}}), a timeout that happens after a long wait, which tests the
> timeouts of the route.
> # *Reply corrupt* at the send: change the reply of the endpoint ({{corrupt
> at=reply mode=truncate}}), not only the request.
> h3. From reading the code (each traced end to end; 2 and 3 also confirmed on
> the running app)
> # *A route is counted once per pass, not once per exchange.*
> {{ChaosRoutePolicyFactory}} counts in {{onExchangeDone}}, and the route
> policy advice runs each time the route processor runs. An exchange that
> enters a {{direct}} route 3 times (a {{loop(3).to("direct:sub")}}, or a
> caller that redelivers {{to("direct:sub")}} while {{sub}} has
> {{noErrorHandler()}}) counts 3 completed or failed exchanges in {{sub}},
> completes {{expect=sub:30}} with 10 messages, and makes a {{route=sub
> corrupt}} rule fire and use up its {{until=}} count 3 times. chaos.adoc says
> "each counted once when the route is done with it". Fix: count once per
> exchange and route (a marker on the exchange), or document "per pass".
> # *{{exception=java.lang.InterruptedException}} bypasses all error handling.*
> {{AbstractExchange.setException}} marks the exchange interrupted, so Camel
> stops routing it: no {{onException}}, no redelivery, no DLC. Confirmed:
> {{step=enrich error exception=java.lang.InterruptedException until=3}} gave 3
> failed exchanges in {{orders}}, 0 redeliveries, nothing in the DLQ, only a
> WARN of the timer consumer. Fix: reject it (and its subclasses) in the
> parser, or wrap it.
> # *More rules that can never fire are accepted silently* (adds to problem 4):
> {{component=http:localhost}} (a {{component=}} pattern is matched against the
> scheme only, so a {{:}} never matches; confirmed, 0 matches), {{rate=every:10
> until=5}} (fires 0 times), and {{message=}} with an exception class that has
> only a no-argument constructor (the message is dropped). Fix: reject the
> first two, reject or warn on the third.
> # *Rules from {{camel.chaos.rules}} start their clock when the context
> initializes, not when it has started.* {{addedAt}} is set in
> {{onContextInitializing}}, so {{until=}}, {{within=}} and the lifecycle times
> include the startup. {{expect=orders:100 within=1m}} with routes that take
> 20s to start (a slow broker connect, the supervising route controller) leaves
> 40s. Fix: re-base the times of the startup rules when the context has started.
> # *The report stays not final after a change too* (adds to problem 6): a
> frozen expectation of a changed or replaced rule, or one frozen by a context
> restart, reads as pending, and {{final}} includes the history, so one
> {{chaos-change}} of a rule with an undecided {{expect=}} keeps
> {{final=false}} for the rest of the process. An MCP agent loop that waits for
> {{final}} never ends. Fix: leave frozen expectations out of {{final}}.
> # *The CLI text hides the lifecycle state, and an empty result says nothing.*
> {{camel get chaos}} and {{--report}} do not show the {{lifecycle}} object
> (down, back in 12s, skipped: route was stopped, left, cancelled): a {{stall}}
> row shows only {{active 1 1 0}}, and the reason only appears with {{--json}}.
> With no rules, or without camel-chaos on the classpath, {{camel get chaos}}
> prints nothing and exits 0. Fix: a LIFECYCLE column (state plus back-in or
> reason), and a "no chaos rules" / "camel-chaos is not on the classpath" line.
> # *Doc nit:* chaos.adoc says {{\"}}, {{\'}} and {{\\}} are escapes inside
> quotes, but only the quote that opened the value and {{\\}} are:
> {{message="it\'s"}} keeps the backslash.
> More ideas from the code read, on top of the list above: a jittered delay
> from the seeded sequence ({{step=enrich delay duration=100ms..2s rate=0.3}});
> a start offset ({{endpoint=kafka:orders* error from=+30s until=1m}}, "the
> broker dies after warm-up", which also avoids the startup clock problem); a
> flapping route ({{route=poller stall duration=10s repeat=3 every=1m}}, for
> reconnect, back-off and supervision); more corrupt modes ({{mode=empty}},
> {{mode=remove-header header=JMSCorrelationID}}, {{truncate}} for String and
> byte[] bodies, which today are skipped). {{SendHook.afterSend}} is already
> there and unused, so the "error after the send" and "corrupt the reply" ideas
> fit the SPI as it is; {{duplicate}} needs a "send again" result in the
> SendHook SPI, and "slow then fail" needs a delay and an error to combine
> (today the first rule that fires decides).
> h3. Not in this issue
> Route-entry errors, a "sends/steps of route X" filter, {{when=}} conditions
> and loose AI prompts are stage 2 (CAMEL-26607). Claus's test prompt from the
> design (2026-10-27) "the order route should have chaos, by failing 5% or when
> a message origin country is spain" was tried through MCP: {{route=orders
> error rate=0.05}} is refused (pointing to endpoint=/step=), the 5% works as
> {{endpoint=direct:inventory error rate=0.05}} (6 of 117), and
> {{when="$\{header.country\} == 'ES'"}} is refused with "not supported yet".
> So the AI needed to look up the first send of the route itself: that is the
> case for the stage 2 filter.
> _Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
