[
https://issues.apache.org/jira/browse/CAMEL-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25738:
--------------------------------
Attachment: camel-test-dst-sqs-world.patch
> camel-test-dst - deterministic simulation testing for Camel consumers (stage
> 1)
> -------------------------------------------------------------------------------
>
> Key: CAMEL-25738
> URL: https://issues.apache.org/jira/browse/CAMEL-25738
> Project: Camel
> Issue Type: New Feature
> Components: camel-test
> Reporter: Claus Ibsen
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
> Attachments: camel-test-dst-sqs-world.patch
>
>
> A new {{camel-test-dst}} module for deterministic simulation testing of Camel
> consumers: one seeded scheduler runs every Camel thread pool on the test
> thread, time is virtual, a fake of the external system injects faults chosen
> by the seed, invariants are checked after each run, and a failing seed
> replays with the same trace.
> The prototype is on the branch {{camel-test-dst-prototype}} of
> github.com/allthingssecurity/camel
> ({{components/camel-test/camel-test-dst}}). This issue is the scope of stage
> 1: making the harness trustworthy enough that a fake world can be written for
> a component without running into the harness. The chaos proposal waits until
> this is done.
> h3. Evidence so far
> A third world was written for camel-aws2-sqs (the visibility timeout
> extender, a fake SQS queue written from the AWS API reference, 458 lines):
> * main: 0 of 5000 seeds fail; {{Sqs2Consumer}} before CAMEL-25700: 1383 of
> 5000 fail, only with {{maxMessagesPerPoll >= 2}} and {{concurrentConsumers >=
> 2}}, as that fix describes; a failing seed replays with the same SHA-256.
> * it found CAMEL-25737 ({{visibilityTimeout=1}}), because the seed also picks
> the endpoint configuration.
> * it needed a harness fix first (item 1): without it, 271 of 500 seeds on
> main were false failures.
> h3. Stage 1 scope
> # *Time read outside the instrumented package.* Byte code is rewritten only
> in the package of {{@DstInstrument}}, and {{java.time}} was not rewritten at
> all ({{Instant.now()}} in the SQS extender). Core clocks still read real
> time: {{MonotonicClock}} (camel-support), {{StopWatch}} (camel-util),
> {{ScheduledPollConsumer}}. Decide between rewriting the core packages too and
> a small time-source hook in core that DST replaces (the README suggests the
> hook; it touches core, so it needs a dev@ discussion).
> # *Real sleeps on the loop thread.* {{RedeliveryPolicy.sleep()}}
> ({{Thread.sleep}}, used by the default synchronous redelivery,
> {{asyncDelayedRedelivery=false}}), {{ForegroundTask}} (used by 10 component
> source files), {{DefaultShutdownStrategy}} (waits for in-flight exchanges
> with {{Thread.sleep}}; the Slack world copies the suspend sequence by hand
> because of it). A virtual sleep that becomes {{dst.block()}} on the loop
> thread.
> # *Timed waits never time out.* {{DstTask.get(timeout)}} ignores the timeout
> and {{awaitTermination}} returns true, so timeout paths (circuit breaker,
> shutdown, {{Future.get(timeout)}}) cannot be explored. Timeouts must fire in
> virtual time.
> # *Waits on something that is not a DST task deadlock.*
> {{CompletableFuture.get()}}/{{join()}} or {{CountDownLatch.await()}} on the
> loop thread hang (the SQS consumer with more than 10 messages per poll). At
> least fail fast with a clear message; better, rewrite them to run the loop
> until the future completes.
> # *Escapes fail the seed, not only the report.* The {{Instant.now()}} gap
> showed up only as a line in {{escapes.txt}}; a mode where reaching an
> uncontrolled escape at runtime fails the seed makes the cause obvious.
> # *Module layout.* camel-test-dst holds the harness only; the demos
> (azure-eventhubs, slack, and the sqs world) move to the tests of their
> component modules as passing regression tests. Refresh the README numbers:
> the fixes of #27922 and #27923 are on main.
> # *A guide for AI agents (and humans) on writing a fake world*, in the module
> doc page with a pointer in {{AGENTS.md}}, using the SQS world as the worked
> example:
> #* find where the client is swapped (an endpoint option such as
> {{amazonSQSClient}}, else a static mock of the factory);
> #* run the escape scan before writing anything, and fix, avoid or document
> every escape;
> #* write the fake from the service's documentation, quoting the rule for each
> behaviour and marking assumptions;
> #* the one-stack timing rule: a blocked task resumes only after the tasks
> that blocked after it, so reads take effect at the end of their latency,
> writes at the start, and background tasks (extenders, timers) do not block;
> #* let the seed pick the endpoint configuration as well as the timing;
> #* seeded faults, switched off for a drain phase, with a record of which
> injected fault excuses which violation;
> #* the invariant from the component's contract, starting from the reusable
> {{Invariants}};
> #* the self-check: fail on the code before a past fix ({{git log}}), pass on
> main, and the same SHA-256 when a seed is replayed;
> #* triage before reporting: group failing seeds by configuration, read one
> trace, tell harness artifact, fake mistake and real bug apart.
> Each harness fix comes with a self-test, like {{DstSchedulerTest}}.
> h3. Later stages (not in this issue)
> * The one-stack model itself (a real fix may need virtual threads with a
> custom scheduler); stage 1 documents it.
> * Async clients on Reactor's own schedulers (the Azure SDKs).
> * Several runs in parallel in one JVM (the active run is a static field).
> * Sharing the fault rules with the chaos proposal.
> _Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
