This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch blog/let-there-be-chaos in repository https://gitbox.apache.org/repos/asf/camel-website.git
commit ad12db78d9706f723db5574c61419f9a1d5dcb13 Author: Claus Ibsen <[email protected]> AuthorDate: Mon Nov 9 22:22:45 2026 +0100 Blog: chaos in Camel 4.24, part 2 - let there be chaos An AI explores chaos on a running integration by itself for a time budget (camel cmd chaos-explore), with the run of a local model on the shop of the chaos example: a figure of the ten minutes, the journal, and what it found. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../11/let-there-be-chaos/chaos-exploration.svg | 317 +++++++++++++++++++++ .../blog/2026/11/let-there-be-chaos/featured.jpg | Bin 0 -> 122556 bytes content/blog/2026/11/let-there-be-chaos/index.md | 165 +++++++++++ .../11/let-there-be-chaos/let-there-be-chaos.gif | Bin 0 -> 166434 bytes 4 files changed, 482 insertions(+) diff --git a/content/blog/2026/11/let-there-be-chaos/chaos-exploration.svg b/content/blog/2026/11/let-there-be-chaos/chaos-exploration.svg new file mode 100644 index 00000000..834e3c61 --- /dev/null +++ b/content/blog/2026/11/let-there-be-chaos/chaos-exploration.svg @@ -0,0 +1,317 @@ +<svg xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 1200 652" width="1200" height="652" font-family="-apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif" role="img" aria-label="Timeline of a ten minute chaos exploration of the shop integration: ten experiments, one failed; refunds lost 5 messages on the same payment fault that orders survived"> +<rect width="1200" height="652" fill="#fcfcfb"/> +<text x="24.0" y="34.0" font-size="19" fill="#0b0b0b" font-weight="600" text-anchor="start">Ten minutes of chaos: what the AI tried on the shop, and what each route did</text> +<text x="24.0" y="56.0" font-size="13" fill="#52514e" font-weight="normal" text-anchor="start">10 experiments chosen by a local model (qwen3.6:35b-mlx), one at a time, within a 10 minute budget</text> +<line x1="96" y1="114" x2="1176" y2="114" stroke="#e4e3df" stroke-width="1"/> +<line x1="96.0" y1="114" x2="96.0" y2="118" stroke="#8a8984"/> +<text x="96.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">0 min</text> +<line x1="204.0" y1="114" x2="204.0" y2="118" stroke="#8a8984"/> +<text x="204.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">1 min</text> +<line x1="312.0" y1="114" x2="312.0" y2="118" stroke="#8a8984"/> +<text x="312.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">2 min</text> +<line x1="420.0" y1="114" x2="420.0" y2="118" stroke="#8a8984"/> +<text x="420.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">3 min</text> +<line x1="528.0" y1="114" x2="528.0" y2="118" stroke="#8a8984"/> +<text x="528.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">4 min</text> +<line x1="636.0" y1="114" x2="636.0" y2="118" stroke="#8a8984"/> +<text x="636.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">5 min</text> +<line x1="744.0" y1="114" x2="744.0" y2="118" stroke="#8a8984"/> +<text x="744.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">6 min</text> +<line x1="852.0" y1="114" x2="852.0" y2="118" stroke="#8a8984"/> +<text x="852.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">7 min</text> +<line x1="960.0" y1="114" x2="960.0" y2="118" stroke="#8a8984"/> +<text x="960.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">8 min</text> +<line x1="1068.0" y1="114" x2="1068.0" y2="118" stroke="#8a8984"/> +<text x="1068.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">9 min</text> +<line x1="1176.0" y1="114" x2="1176.0" y2="118" stroke="#8a8984"/> +<text x="1176.0" y="131.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">10 min</text> +<text x="84.0" y="107.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="end">when</text> +<rect x="142.3" y="96" width="74.2" height="16" rx="3" fill="#0ca30c"><title>experiment 1: PASS, 0:26 to 1:07</title></rect> +<path d="M179.4,136 C179.4,150 150.0,144 150.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="223.9" y="96" width="74.2" height="16" rx="3" fill="#0ca30c"><title>experiment 2: PASS, 1:11 to 1:52</title></rect> +<path d="M261.0,136 C261.0,150 258.0,144 258.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="307.3" y="96" width="11.1" height="16" rx="3" fill="#d03b3b"><title>experiment 3: FAIL, 1:57 to 2:04</title></rect> +<path d="M312.9,136 C312.9,150 366.0,144 366.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="348.1" y="96" width="74.2" height="16" rx="3" fill="#0ca30c"><title>experiment 4: PASS, 2:20 to 3:01</title></rect> +<path d="M385.2,136 C385.2,150 474.0,144 474.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="431.5" y="96" width="74.2" height="16" rx="3" fill="#0ca30c"><title>experiment 5: PASS, 3:06 to 3:48</title></rect> +<path d="M468.6,136 C468.6,150 582.0,144 582.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="515.0" y="96" width="163.4" height="16" rx="3" fill="#a3a29c"><title>experiment 6: INCONCLUSIVE, 3:53 to 5:24</title></rect> +<path d="M596.7,136 C596.7,150 690.0,144 690.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="687.6" y="96" width="165.2" height="16" rx="3" fill="#0ca30c"><title>experiment 7: PASS, 5:29 to 7:00</title></rect> +<path d="M770.2,136 C770.2,150 798.0,144 798.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="863.9" y="96" width="74.3" height="16" rx="3" fill="#0ca30c"><title>experiment 8: PASS, 7:07 to 7:48</title></rect> +<path d="M901.0,136 C901.0,150 906.0,144 906.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="949.3" y="96" width="74.2" height="16" rx="3" fill="#0ca30c"><title>experiment 9: PASS, 7:54 to 8:35</title></rect> +<path d="M986.4,136 C986.4,150 1014.0,144 1014.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<rect x="1032.7" y="96" width="74.3" height="16" rx="3" fill="#0ca30c"><title>experiment 10: PASS, 8:40 to 9:22</title></rect> +<path d="M1069.9,136 C1069.9,150 1122.0,144 1122.0,158" fill="none" stroke="#e4e3df" stroke-width="1.5"/> +<line x1="1112.6" y1="92" x2="1112.6" y2="114" stroke="#52514e" stroke-dasharray="2 2"/> +<text x="1108.6" y="90.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="end">report</text> +<circle cx="120.0" cy="174" r="11" fill="#0ca30c"/> +<text x="120.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="136.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#1</text> +<text x="136.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="150.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment answers</text> +<text x="150.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">503, 25%</text> +<circle cx="228.0" cy="174" r="11" fill="#0ca30c"/> +<text x="228.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="244.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#2</text> +<text x="244.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="258.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">shipping route</text> +<text x="258.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">fails, 20%</text> +<circle cx="336.0" cy="174" r="11" fill="#d03b3b"/> +<text x="336.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✗</text> +<text x="352.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#3</text> +<text x="352.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">FAIL</text> +<text x="366.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment answers</text> +<text x="366.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">503, 25%</text> +<circle cx="444.0" cy="174" r="11" fill="#0ca30c"/> +<text x="444.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="460.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#4</text> +<text x="460.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="474.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment times</text> +<text x="474.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">out (5s), 10%</text> +<circle cx="552.0" cy="174" r="11" fill="#0ca30c"/> +<text x="552.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="568.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#5</text> +<text x="568.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="582.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment gets</text> +<text x="582.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">duplicates, 10%</text> +<circle cx="660.0" cy="174" r="11" fill="#a3a29c"/> +<text x="660.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">?</text> +<text x="676.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#6</text> +<text x="676.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">UNDECIDED</text> +<text x="690.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment down,</text> +<text x="690.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">dead letter fails</text> +<circle cx="768.0" cy="174" r="11" fill="#0ca30c"/> +<text x="768.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="784.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#7</text> +<text x="784.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="818.4" y="185.0" font-size="11" fill="#d03b3b" font-weight="700" text-anchor="start">b</text> +<text x="798.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">the same,</text> +<text x="798.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">waits longer</text> +<circle cx="876.0" cy="174" r="11" fill="#0ca30c"/> +<text x="876.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="892.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#8</text> +<text x="892.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="906.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">parked file</text> +<text x="906.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">write fails, 25%</text> +<circle cx="984.0" cy="174" r="11" fill="#0ca30c"/> +<text x="984.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="1000.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#9</text> +<text x="1000.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="1034.4" y="185.0" font-size="11" fill="#d03b3b" font-weight="700" text-anchor="start">a</text> +<text x="1014.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment times</text> +<text x="1014.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">out (5s), 10%</text> +<circle cx="1092.0" cy="174" r="11" fill="#0ca30c"/> +<text x="1092.0" y="178.5" font-size="13" fill="#ffffff" font-weight="700" text-anchor="middle">✓</text> +<text x="1108.0" y="172.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">#10</text> +<text x="1108.0" y="185.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">PASS</text> +<text x="1142.4" y="185.0" font-size="11" fill="#d03b3b" font-weight="700" text-anchor="start">c</text> +<text x="1122.0" y="206.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">payment route</text> +<text x="1122.0" y="220.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="middle">fails, 5%</text> +<text x="84.0" y="261.0" font-size="13" fill="#0b0b0b" font-weight="600" text-anchor="end">orders</text> +<g><title>experiment 1, orders: 41 completed, 0 failed, 11 redeliveries</title> +<rect x="98.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="150.0" y="255.0" font-size="12" fill="#0b0b0b" font-weight="normal" text-anchor="middle">41 ok</text> +<text x="150.0" y="269.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">11 retried</text> +</g> +<g><title>experiment 2, orders: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="206.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="258.0" y="261.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 3, orders: 6 completed, 0 failed, 3 redeliveries</title> +<rect x="314.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="366.0" y="255.0" font-size="12" fill="#0b0b0b" font-weight="normal" text-anchor="middle">6 ok</text> +<text x="366.0" y="269.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">3 retried</text> +</g> +<g><title>experiment 4, orders: 32 completed, 0 failed, 2 redeliveries</title> +<rect x="422.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="474.0" y="255.0" font-size="12" fill="#0b0b0b" font-weight="normal" text-anchor="middle">32 ok</text> +<text x="474.0" y="269.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">2 retried</text> +</g> +<g><title>experiment 5, orders: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="530.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="582.0" y="261.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 6, orders: 90 completed, 0 failed, 118 redeliveries, 58 lost at the dead letter channel</title> +<rect x="638.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<rect x="638.0" y="236" width="4" height="42" rx="2" fill="#ec835a"/> +<text x="692.0" y="255.0" font-size="13" fill="#0b0b0b" font-weight="700" text-anchor="middle">58 lost</text> +<text x="692.0" y="269.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">at dead letter</text> +</g> +<g><title>experiment 7, orders: 90 completed, 0 failed, 117 redeliveries, 58 lost at the dead letter channel</title> +<rect x="746.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<rect x="746.0" y="236" width="4" height="42" rx="2" fill="#ec835a"/> +<text x="800.0" y="255.0" font-size="13" fill="#0b0b0b" font-weight="700" text-anchor="middle">58 lost</text> +<text x="800.0" y="269.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">at dead letter</text> +</g> +<g><title>experiment 8, orders: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="854.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="906.0" y="261.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 9, orders: 36 completed, 0 failed, 1 redeliveries</title> +<rect x="962.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="1014.0" y="255.0" font-size="12" fill="#0b0b0b" font-weight="normal" text-anchor="middle">36 ok</text> +<text x="1014.0" y="269.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">1 retried</text> +</g> +<g><title>experiment 10, orders: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="1070.0" y="236" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="1122.0" y="261.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<text x="84.0" y="307.0" font-size="13" fill="#0b0b0b" font-weight="600" text-anchor="end">refunds</text> +<rect x="98.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="150.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="206.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="258.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<g><title>experiment 3, refunds: 15 completed, 5 failed, 0 redeliveries</title> +<rect x="314.0" y="282" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<rect x="314.0" y="282" width="4" height="42" rx="2" fill="#d03b3b"/> +<text x="368.0" y="301.0" font-size="13" fill="#0b0b0b" font-weight="700" text-anchor="middle">5 failed</text> +<text x="368.0" y="315.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">gone</text> +</g> +<rect x="422.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="474.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<g><title>experiment 5, refunds: 20 completed, 0 failed, 0 redeliveries</title> +<rect x="530.0" y="282" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="582.0" y="307.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">20 ok</text> +</g> +<rect x="638.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="690.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="746.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="798.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="854.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="906.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<g><title>experiment 9, refunds: 18 completed, 2 failed, 0 redeliveries</title> +<rect x="962.0" y="282" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<rect x="962.0" y="282" width="4" height="42" rx="2" fill="#d03b3b"/> +<text x="1016.0" y="301.0" font-size="13" fill="#0b0b0b" font-weight="700" text-anchor="middle">2 failed</text> +<text x="1016.0" y="315.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">gone</text> +</g> +<rect x="1070.0" y="282" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="1122.0" y="307.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<text x="84.0" y="353.0" font-size="13" fill="#0b0b0b" font-weight="600" text-anchor="end">shipping</text> +<g><title>experiment 1, shipping: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="98.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="150.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 2, shipping: 41 completed, 0 failed, 5 redeliveries</title> +<rect x="206.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="258.0" y="347.0" font-size="12" fill="#0b0b0b" font-weight="normal" text-anchor="middle">41 ok</text> +<text x="258.0" y="361.0" font-size="10" fill="#52514e" font-weight="normal" text-anchor="middle">5 retried</text> +</g> +<g><title>experiment 3, shipping: 6 completed, 0 failed, 0 redeliveries</title> +<rect x="314.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="366.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">6 ok</text> +</g> +<g><title>experiment 4, shipping: 32 completed, 0 failed, 0 redeliveries</title> +<rect x="422.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="474.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">32 ok</text> +</g> +<g><title>experiment 5, shipping: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="530.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="582.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 6, shipping: 32 completed, 0 failed, 0 redeliveries</title> +<rect x="638.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="690.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">32 ok</text> +</g> +<g><title>experiment 7, shipping: 32 completed, 0 failed, 0 redeliveries</title> +<rect x="746.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="798.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">32 ok</text> +</g> +<g><title>experiment 8, shipping: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="854.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="906.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 9, shipping: 36 completed, 0 failed, 0 redeliveries</title> +<rect x="962.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="1014.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">36 ok</text> +</g> +<g><title>experiment 10, shipping: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="1070.0" y="328" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="1122.0" y="353.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<text x="84.0" y="399.0" font-size="13" fill="#0b0b0b" font-weight="600" text-anchor="end">parked</text> +<rect x="98.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="150.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="206.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="258.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="314.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="366.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="422.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="474.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="530.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="582.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="638.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="690.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="746.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="798.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<g><title>experiment 8, parked: 20 completed, 0 failed, 0 redeliveries</title> +<rect x="854.0" y="374" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="906.0" y="399.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">20 ok</text> +</g> +<rect x="962.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="1014.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<rect x="1070.0" y="374" width="104.0" height="42" rx="4" fill="none" stroke="#e4e3df" stroke-dasharray="3 3"/> +<text x="1122.0" y="399.0" font-size="10" fill="#8a8984" font-weight="normal" text-anchor="middle">no traffic</text> +<text x="84.0" y="445.0" font-size="13" fill="#0b0b0b" font-weight="600" text-anchor="end">payment</text> +<g><title>experiment 1, payment: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="98.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="150.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 2, payment: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="206.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="258.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 3, payment: 21 completed, 0 failed, 0 redeliveries</title> +<rect x="314.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="366.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">21 ok</text> +</g> +<g><title>experiment 4, payment: 32 completed, 0 failed, 0 redeliveries</title> +<rect x="422.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="474.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">32 ok</text> +</g> +<g><title>experiment 5, payment: 69 completed, 0 failed, 0 redeliveries</title> +<rect x="530.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="582.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">69 ok</text> +</g> +<g><title>experiment 6, payment: 32 completed, 0 failed, 0 redeliveries</title> +<rect x="638.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="690.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">32 ok</text> +</g> +<g><title>experiment 7, payment: 32 completed, 0 failed, 0 redeliveries</title> +<rect x="746.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="798.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">32 ok</text> +</g> +<g><title>experiment 8, payment: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="854.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="906.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<g><title>experiment 9, payment: 54 completed, 0 failed, 0 redeliveries</title> +<rect x="962.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="1014.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">54 ok</text> +</g> +<g><title>experiment 10, payment: 41 completed, 0 failed, 0 redeliveries</title> +<rect x="1070.0" y="420" width="104.0" height="42" rx="4" fill="#f3f2ef"/> +<text x="1122.0" y="445.0" font-size="11" fill="#8a8984" font-weight="normal" text-anchor="middle">41 ok</text> +</g> +<rect x="96.0" y="158" width="108.0" height="308" rx="7" fill="none" stroke="#0b0b0b" stroke-width="1.5"/> +<rect x="312.0" y="158" width="108.0" height="308" rx="7" fill="none" stroke="#0b0b0b" stroke-width="1.5"/> +<path d="M150.0,466 V482 H366.0 V466" fill="none" stroke="#0b0b0b" stroke-width="1.5"/> +<text x="24.0" y="504.0" font-size="14" fill="#0b0b0b" font-weight="700" text-anchor="start">Same fault, two routes.</text> +<text x="212.0" y="504.0" font-size="14" fill="#0b0b0b" font-weight="normal" text-anchor="start">Orders retries through its dead letter channel and loses none; refunds has no error handling and loses 5 of 20.</text> +<rect x="24" y="524" width="4" height="13" rx="2" fill="#d03b3b"/> +<text x="34.0" y="535.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">failed: the message is gone</text> +<rect x="222.7" y="524" width="4" height="13" rx="2" fill="#ec835a"/> +<text x="232.7" y="535.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">lost at the dead letter channel</text> +<text x="445.8" y="535.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">ok: completed (retried: redelivered and then completed)</text> +<text x="24.0" y="562.0" font-size="11" fill="#0b0b0b" font-weight="600" text-anchor="start">Read afterwards:</text> +<text x="24.0" y="580.0" font-size="11" fill="#d03b3b" font-weight="700" text-anchor="start">a</text> +<text x="38.0" y="580.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">The expectation measured time, so the 2 failed refunds did not fail the experiment (the AI saw them, but put them down to an earlier experiment).</text> +<text x="24.0" y="596.0" font-size="11" fill="#d03b3b" font-weight="700" text-anchor="start">b</text> +<text x="38.0" y="596.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">Passed in this run; it fails since CAMEL-26789, as an experiment now counts what the dead letter channel lost.</text> +<text x="24.0" y="612.0" font-size="11" fill="#d03b3b" font-weight="700" text-anchor="start">c</text> +<text x="38.0" y="612.0" font-size="11" fill="#52514e" font-weight="normal" text-anchor="start">The fault never fired: the rule matched nothing, so the PASS says nothing.</text> +</svg> \ No newline at end of file diff --git a/content/blog/2026/11/let-there-be-chaos/featured.jpg b/content/blog/2026/11/let-there-be-chaos/featured.jpg new file mode 100644 index 00000000..832ae4a2 Binary files /dev/null and b/content/blog/2026/11/let-there-be-chaos/featured.jpg differ diff --git a/content/blog/2026/11/let-there-be-chaos/index.md b/content/blog/2026/11/let-there-be-chaos/index.md new file mode 100644 index 00000000..c552e62d --- /dev/null +++ b/content/blog/2026/11/let-there-be-chaos/index.md @@ -0,0 +1,165 @@ +--- +title: "Chaos in Camel 4.24, part 2: let there be chaos, an AI that breaks your routes on purpose" +date: 2026-11-15 +draft: true +authors: [davsclaus] +categories: ["Camel", "Tooling", "AI"] +keywords: ["apache camel", "chaos", "chaos engineering", "ai", "camel cli", "mcp", "local llm", "resilience", "camel 4.24"] +preview: "camel-chaos lets you add faults to a running integration. Now you can hand the whole thing to an AI: give it ten minutes, and it tries faults by itself, measures what held and what broke, and comes back with a report and scenarios you can replay." +--- + +In [part 1](/blog/2026/11/camel-chaos/) you added the faults yourself: "fail a quarter of the payment calls", +"fail the dead letter channel too", and then you read the report. That works, but it means you have to know what to +try. With Camel 4.24 you can hand that part to an AI as well: + +```shell +camel cmd chaos-explore shop --budget=10m +``` + +Let there be chaos. For ten minutes the AI of `camel ask` explores how the integration copes with faults, by +itself. It reads what the routes do, picks a fault to try, says what should hold, runs the experiment, reads the +verdict, and goes on. At the end it writes a report: what held, what broke and why, and a hint for each finding. The +integration keeps running; the AI never changes your routes. + + + +## The shop + +The shop is the one from the [chaos example](https://github.com/apache/camel-jbang-examples/tree/main/fail-well/chaos): +an order every second, charged at a payment service over HTTP, handed to shipping, with a dead letter channel that +parks what could not be charged. We added one route: refunds arrive on Kafka and are paid back at the payment +service. The refunds route has no error handling of its own. That is the kind of gap you want found before +production. Kafka is stubbed (`camel run --stub=kafka`), so no broker is needed and nothing real is touched. + +```shell +camel run * --chaos --stub=kafka --name=shop +camel cmd chaos-explore shop --budget=10m +``` + +The model is qwen3.6 (35B), running locally on a laptop. No cloud, no API key. + +## Ten minutes, in one picture + +Here is the whole run. Along the top, when each experiment ran within the budget; below, one column per experiment, +with what it did to each route: + + + +Look at columns 1 and 3. It is the same fault both times: the payment service answers 503 to a quarter of the calls. +The orders retried 11 calls through their dead letter channel, and every order got through. The refunds have no +retry: 5 of the 20 failed, and they are gone. + +The AI did not stop at the verdict. It looked up the errors of the integration by itself and explained what it saw: + +> The refunds route failed with `HttpOperationFailedException` - it has no error handling for the HTTP call to +> payment. [...] This is a real issue: the refunds route needs better error handling (like an onException with +> redelivery or a dead letter channel). + +That is the finding, the evidence and the hint, in the AI's own words, without anybody telling it where to look. + +In another run it went for the nastier case: the payment service down, and the dead letter channel failing too. +The dead letter channel then handles the orders it could not park, so they neither fail nor complete: they are just +gone. That run reported "when payment returned 503 and the dead letter channel also failed, 2 messages were lost at +the dead letter channel" (columns 6 and 7 above are the same experiment, before Camel counted those losses; more on +that below). + +## The journal + +Every experiment goes into the journal of the exploration, with the hypothesis the AI wrote for it and the evidence +for the verdict. These are the ten experiments of the run, as the AI phrased them: + +| # | Hypothesis | Expected | Verdict | Evidence | +|---|---|---|---|---| +| 1 | When payment returns 503, orders should retry and not lose messages | orders: failed ≤ 0 | PASS | 0 failed, 11 retried | +| 2 | When shipping fails, orders should not lose the message handed off via seda | orders→shipping: lost ≤ 0 | PASS | 0 lost (39 in orders, 39 in shipping) | +| 3 | When payment returns 503, refunds should handle errors gracefully | refunds: failed ≤ 0 | **FAIL** | **5 failed** | +| 4 | When payment times out, orders should retry and complete within reasonable time | orders: max time ≤ 30s | PASS | max 5.5s of 31 orders | +| 5 | When payment receives duplicates, refunds should handle them safely | refunds: failed ≤ 0 | PASS | 0 failed | +| 6 | When payment fails and the dead letter channel also fails, orders should not lose messages entirely | orders: failed ≤ 0 | undecided | the window was longer than the wait | +| 7 | The same, again with a shorter window | orders: failed ≤ 0 | PASS | 0 failed (58 lost at the dead letter channel) | +| 8 | When the file write fails, parked should handle the error gracefully | parked: failed ≤ 0 | PASS | 0 failed, 7 handled | +| 9 | When payment times out, refunds should complete within reasonable time | refunds: max time ≤ 30s | PASS | max 5s (2 failed) | +| 10 | When the payment route fails, HTTP clients should see proper error responses | payment: failed ≤ 0 | PASS | the fault never fired | + +The AI kept to its budget: with 38 seconds left it said that was not enough for another experiment, and finished. + +## One experiment per call + +The trick that makes this work with a small local model is that each experiment is a single tool call. The model +does not juggle rules, traffic and timers; it says what it wants to know: + +```json +{ + "hypothesis": "When payment returns 503, refunds should handle errors gracefully", + "rule": "endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=30s expect=refunds:failed<=0 within=40s", + "feed": "refunds", "body": "{\"refundId\": \"R-1\"}", "repeat": 20, "interval": "100..300" +} +``` + +Camel adds the rule, sends the 20 refunds to the stubbed Kafka topic, waits until the expectation is decided, removes +the rule, and answers with the verdict, the counts per route and the traffic. You can do exactly the same by hand, +without any AI: + +```shell +camel cmd chaos-explore-start --budget=10m +camel cmd chaos-experiment --hypothesis='a failed refund is retried, not lost' \ + --rule='endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=30s expect=refunds:failed<=0 within=40s' \ + --feed=refunds --body='{"refundId": "R-1"}' --repeat=20 --interval=100..300 +camel cmd chaos-explore-finish +``` + +## Safe by construction + +You say yes once, by starting the exploration, and Camel holds the line from there, whatever the model does: + +- Chaos only runs in development (profile `dev`), never with `prod`. +- The exploration has a deadline (10 minutes by default, 2 hours at most). Every rule ends at the deadline at the + latest, and at the deadline all rules are removed, even if the AI has crashed or lost its context. +- Remote systems are stubbed, so the experiments do not reach your real Kafka or payment provider. +- The AI only adds faults and sends test messages to the routes; it does not edit your project. + +## What you keep + +Everything ends up in `test/chaos`, which `camel run` and `camel export` leave alone: + +- `test/chaos/reports/explore-<time>.md`: the report of the AI and the journal of all experiments, and a `.json` next + to it for tools. The picture above is drawn from that `.json`. +- `test/chaos/explore-<time>-<n>.chaos`: each experiment that failed, as a scenario: + +```text +# From the chaos exploration of shop on 2026-11-09 18:14, experiment 3: FAIL +# hypothesis: When payment returns 503, refunds should handle errors gracefully +# expect refunds:failed<=0: unmet, 5 failed +seed=-7420423075154098588 +endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=30s expect=refunds:failed<=0 within=40s +``` + +Load it again (`camel cmd chaos-load`, or `camel run * --chaos=...`), send some refunds, and the verdict is FAIL +again, until you fix the route. Then keep it as a test that gates your pipeline. + +## Honest about the model + +A small local model is not a chaos engineer, and the picture shows where it slipped (the red letters): + +- **a**: In experiment 9 it measured time, not failures, so 2 failed refunds passed. It saw them, and put them down + to an earlier experiment. +- **b**: Experiments 6 and 7 passed while orders were lost at the dead letter channel. That one was on Camel, not + on the model: the experiment did not count those losses yet. The run showed it, and it is fixed: such an experiment + now fails. +- **c**: In experiment 10 the fault never fired, so its PASS says nothing. + +Our earlier runs were rougher: thresholds such as `failed<=5` that hide what was lost, messages fed to an endpoint +no route reads from, rules that matched nothing. Each of those made an experiment that said nothing, so we made +Camel catch them: an experiment refuses a feed that reaches no route (and lists the routes it can feed), warns when +a rule matches nothing, waits until its expectations can decide, and counts what the dead letter channel lost. The +model reads why and corrects itself. A bigger model makes better choices, and the journal always carries the +evidence, so you can judge each finding yourself. + +## Any agent + +The same playbook is an MCP prompt, `camel_chaos_explore`, so Claude Code, or any agent with the Camel MCP server, +can run the same loop with the `camel_runtime_chaos_*` tools. And because each exploration is bound to one running +integration, you can imagine where this goes: several agents, each with its own copy of the integration, exploring +in parallel for an hour, and one agent that combines their reports. + +Try it with Camel 4.24, and tell us what it found in your routes. diff --git a/content/blog/2026/11/let-there-be-chaos/let-there-be-chaos.gif b/content/blog/2026/11/let-there-be-chaos/let-there-be-chaos.gif new file mode 100644 index 00000000..cb7fa736 Binary files /dev/null and b/content/blog/2026/11/let-there-be-chaos/let-there-be-chaos.gif differ
