davsclaus opened a new pull request, #29501: URL: https://github.com/apache/camel/pull/29501
[CAMEL-26787](https://issues.apache.org/jira/browse/CAMEL-26787), step 6 of "let there be chaos" ([CAMEL-26760](https://issues.apache.org/jira/browse/CAMEL-26760)). **Stacked on #29499**, which is stacked on #29498. It shows only its own commit, and gets retargeted to main once those merge. The user says "let there be chaos", and the AI explores by itself for the budget and reports at the end. **What changes** - `ChaosExplorePlaybook`: one playbook text for both clients, each with its own tool names: 1. read the report and the suggestions; 2. run one experiment per call, with a hypothesis, a short `until=`, and an expectation that decides it; 3. read the verdict (on FAIL, read the errors); 4. go on while `timeLeft` allows; 5. finish and report: findings with evidence and a hint, what held, and what was not tried. - The AI asks nothing on the way and never changes the routes or files. - It expects what should hold despite the fault (nothing lost, handled, redelivered), not that nothing fails. - Hints are about the route's error handling, not endpoint options it is not sure exist. - MCP prompt `camel_chaos_explore` (`budget`, `focus`, `name`). - `camel cmd chaos-explore [name] --budget --focus` with the LLM options of `camel ask`: 1. starts the exploration itself, so the budget is the user's; 2. runs camel ask's agent loop with the playbook. The start tool is hidden from the model there. 3. always finishes the exploration; 4. prints the AI's report and the journal of experiments. Exit code 1 when an experiment failed. - One consent: starting the command is the consent (decided with Claus on #29499). - Docs: camel-jbang-managing, camel-jbang-mcp (prompts table), and the chaos doc (copied to the catalog). Generated command page and golden help. **Live run.** Integration: a timer route `orders` hands off over `seda` to `shipping`, plus a `direct` route `refunds`. Model: local `qwen3.6:35b-mlx`. Budget: 3m. ``` Let there be chaos: ... 3m0s [tool] chaos_suggest({}) [tool] chaos_experiment({"hypothesis":"messages handed off to shipping are not lost when shipping faults", "rule":"route=shipping error rate=0.2 until=30s expect=orders->shipping:lost<=0 within=45s","wait":"60s",...}) ... **FAIL: Orders → Shipping hand-off loses messages when shipping faults** - Hint: Add a dead letter channel or retry on the `to:seda:shipping` step ... Chaos exploration finished at its deadline after 3m0s: 3 experiments, 1 failed 1 FAIL messages handed off to shipping are not lost when shipping faults (r5, ended after 46s) expect orders->shipping:lost<=0: unmet, 11 lost (88 completed in orders, 77 in shipping) ... exit=1 ``` - Both runs found the real weakness on their own: the seda hand-off loses messages when the next route fails. - The first run showed three things I fixed here: - the model tried to start a second exploration (now the tool is hidden); - an expectation failed by design (the playbook now says what to expect); - one hint invented `seda` options (the playbook now asks for error handling instead). - What is still weak is down to the model, and an eval will measure it (follow-up): - some expectations are loose; - one experiment was run without a feed; - it planned too close to the deadline. **Limits of the first release** - One agent and one integration. The multi-agent idea (N isolated copies, combined report) is kept open for later. - A route fed over HTTP or from files cannot be fed by an experiment yet. The playbook says so. **Tests** - `ChaosExplorePlaybookTest`: the names per client; the tools it names exist; the expectation forms are the ones camel-chaos reads. - `ChaosExploreTest`: camel ask gets the playbook, the options and the hidden start tool; the journal lines. - `PromptDefinitionsTest`: the prompt. The existing check that every tool a prompt names exists covers it too. - camel-jbang-core: 2732 + 318 green. camel-jbang-mcp: 543 green. _Claude Code on behalf of davsclaus_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012nJsFsjqqFiwroeHRWaKbS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
