Claus Ibsen created CAMEL-25738:
-----------------------------------
Summary: camel-test-dst - deterministic simulation testing for
Camel consumers (stage 1)
Key: CAMEL-25738
URL: https://issues.apache.org/jira/browse/CAMEL-25738
Project: Camel
Issue Type: New Feature
Components: camel-test
Reporter: Claus Ibsen
Assignee: shashank
Fix For: 4.24.0
A new {{camel-test-dst}} module for deterministic simulation testing of Camel
consumers: one seeded scheduler runs every Camel thread pool on the test
thread, time is virtual, a fake of the external system injects faults chosen by
the seed, invariants are checked after each run, and a failing seed replays
with the same trace.
The prototype is on the branch {{camel-test-dst-prototype}} of
github.com/allthingssecurity/camel ({{components/camel-test/camel-test-dst}}).
This issue is the scope of stage 1: making the harness trustworthy enough that
a fake world can be written for a component without running into the harness.
The chaos proposal waits until this is done.
h3. Evidence so far
A third world was written for camel-aws2-sqs (the visibility timeout extender,
a fake SQS queue written from the AWS API reference, 458 lines):
* main: 0 of 5000 seeds fail; {{Sqs2Consumer}} before CAMEL-25700: 1383 of 5000
fail, only with {{maxMessagesPerPoll >= 2}} and {{concurrentConsumers >= 2}},
as that fix describes; a failing seed replays with the same SHA-256.
* it found CAMEL-25737 ({{visibilityTimeout=1}}), because the seed also picks
the endpoint configuration.
* it needed a harness fix first (item 1): without it, 271 of 500 seeds on main
were false failures.
h3. Stage 1 scope
# *Time read outside the instrumented package.* Byte code is rewritten only in
the package of {{@DstInstrument}}, and {{java.time}} was not rewritten at all
({{Instant.now()}} in the SQS extender). Core clocks still read real time:
{{MonotonicClock}} (camel-support), {{StopWatch}} (camel-util),
{{ScheduledPollConsumer}}. Decide between rewriting the core packages too and a
small time-source hook in core that DST replaces (the README suggests the hook;
it touches core, so it needs a dev@ discussion).
# *Real sleeps on the loop thread.* {{RedeliveryPolicy.sleep()}}
({{Thread.sleep}}, used by the default synchronous redelivery,
{{asyncDelayedRedelivery=false}}), {{ForegroundTask}} (used by 10 component
source files), {{DefaultShutdownStrategy}} (waits for in-flight exchanges with
{{Thread.sleep}}; the Slack world copies the suspend sequence by hand because
of it). A virtual sleep that becomes {{dst.block()}} on the loop thread.
# *Timed waits never time out.* {{DstTask.get(timeout)}} ignores the timeout
and {{awaitTermination}} returns true, so timeout paths (circuit breaker,
shutdown, {{Future.get(timeout)}}) cannot be explored. Timeouts must fire in
virtual time.
# *Waits on something that is not a DST task deadlock.*
{{CompletableFuture.get()}}/{{join()}} or {{CountDownLatch.await()}} on the
loop thread hang (the SQS consumer with more than 10 messages per poll). At
least fail fast with a clear message; better, rewrite them to run the loop
until the future completes.
# *Escapes fail the seed, not only the report.* The {{Instant.now()}} gap
showed up only as a line in {{escapes.txt}}; a mode where reaching an
uncontrolled escape at runtime fails the seed makes the cause obvious.
# *Module layout.* camel-test-dst holds the harness only; the demos
(azure-eventhubs, slack, and the sqs world) move to the tests of their
component modules as passing regression tests. Refresh the README numbers: the
fixes of #27922 and #27923 are on main.
# *A guide for AI agents (and humans) on writing a fake world*, in the module
doc page with a pointer in {{AGENTS.md}}, using the SQS world as the worked
example:
#* find where the client is swapped (an endpoint option such as
{{amazonSQSClient}}, else a static mock of the factory);
#* run the escape scan before writing anything, and fix, avoid or document
every escape;
#* write the fake from the service's documentation, quoting the rule for each
behaviour and marking assumptions;
#* the one-stack timing rule: a blocked task resumes only after the tasks that
blocked after it, so reads take effect at the end of their latency, writes at
the start, and background tasks (extenders, timers) do not block;
#* let the seed pick the endpoint configuration as well as the timing;
#* seeded faults, switched off for a drain phase, with a record of which
injected fault excuses which violation;
#* the invariant from the component's contract, starting from the reusable
{{Invariants}};
#* the self-check: fail on the code before a past fix ({{git log}}), pass on
main, and the same SHA-256 when a seed is replayed;
#* triage before reporting: group failing seeds by configuration, read one
trace, tell harness artifact, fake mistake and real bug apart.
Each harness fix comes with a self-test, like {{DstSchedulerTest}}.
h3. Later stages (not in this issue)
* The one-stack model itself (a real fix may need virtual threads with a custom
scheduler); stage 1 documents it.
* Async clients on Reactor's own schedulers (the Azure SDKs).
* Several runs in parallel in one JVM (the active run is a static field).
* Sharing the fault rules with the chaos proposal.
_Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
