allthingssecurity opened a new pull request, #28067:
URL: https://github.com/apache/camel/pull/28067
CAMEL-25738: camel-test-dst - deterministic simulation testing for Camel
consumers (stage 1)
## What
A new test module, `camel-test-dst` (preview), and three regression tests
that use it. The real consumer of a component
runs under one seeded scheduler with virtual time on the test thread; only
the external system is faked. A test runs
thousands of seeds in seconds, and a failing seed replays with the same step
trace (same SHA-256). Nothing in it runs
in production and no Camel component or core class changes (outside tests,
the PR touches only poms, the catalog/docs registration and one line of
`MojoHelper`).
Stage 1 of CAMEL-25738 makes the harness trustworthy enough to write a fake
world for a component without running into
the harness. Following the direction on Zulip, this is DST only; the chaos
proposal comes later.
## Commits
The first four are the prototype (as on the `camel-test-dst-prototype`
branch). The fifth is Claus Ibsen's SQS world
(his CAMEL-25738 patch, kept as he wrote it). Then one commit per stage 1
item (the pitfalls subsection of item 7 is a
commit of its own), and one per follow-up (9 to 12):
0. **Seed scrambling.** `new Random(seed)` drew the same first values for
consecutive seeds (for seeds 1 to 1000 the
first `nextInt(2)` is always 1), so the configuration a world picked
first barely changed. The seed is scrambled
with SplitMix64; replay stays exact. `DstSeedScrambleTest`.
1. **Time read outside the instrumented package.** The rewrite now covers
`new Date()`, `Clock.system*()` and
`now()` / `now(ZoneId)` of the `java.time` types.
`@DstInstrument(alsoRewrite = ...)` lists core classes or packages
to rewrite too (`StopWatch`, `MonotonicClock`, ...), documented as a
stop-gap: Camel core is not changed here, a
time source in core needs a dev@ thread first. `DstInstrumentationTest`,
`DstAlsoRewriteTest`.
2. **Real sleeps on the loop thread.** `Thread.sleep` / `TimeUnit.sleep`
become virtual waits (`dst.block`). With
`alsoRewrite` this covers `RedeliveryPolicy.sleep` and the synchronous
redelivery loop, `ForegroundTask`, and the
wait loop of `DefaultShutdownStrategy` (a route can be stopped with an
exchange in flight). `DstSleepTest`,
`DstCoreWaitsTest`.
3. **Timed waits.** `DstTask.get(timeout)` and `awaitTermination` time out
in virtual time; `shutdown()` drops periodic
tasks as `ScheduledThreadPoolExecutor` does. `DstTimedWaitTest`.
4. **Waits on something that is not a DST task.**
`CompletableFuture.get/join`, `Future.get`, `CountDownLatch.await`,
`Semaphore.acquire`, `DelayQueue`/`BlockingQueue` `take/poll` in the code
under test run the loop until the wait can
end, or fail the seed at once with the call site. A watchdog fails a seed
whose loop thread waits in code that is not
rewritten (10 s real time), instead of the surefire timeout.
`DstBlockingWaitTest`.
5. **Strict mode**, the default: reaching an uncontrolled escape at runtime
fails the seed. `DstStrictModeTest`.
6. **Module layout.** The package of the test is its world and is not
rewritten. The worlds move to the tests of their
components (`...dst` packages) as passing regression tests with a test
scoped dependency on camel-test-dst; Claus's
SQS world moves in a commit of its own. The module keeps the harness only.
7. **The guide for writing a fake world** (for AI agents and humans), on the
module doc page `test-dst.adoc`, with
the SQS world as the worked example and a pointer in `AGENTS.md`. It
covers each point of CAMEL-25738: find where
the client is swapped; run the escape scan before writing anything; write
the fake from the service's documentation,
quoting the rule for each behaviour and marking assumptions; the
one-stack timing rule; let the seed pick the endpoint
configuration as well as the timing; seeded faults with a drain phase and
a record of which fault excuses which
violation; invariants from the component's contract (`Invariants`); the
self-check (fail on the code before a past
fix, pass on `main`, same SHA-256 on replay); triage before reporting. A
subsection "Pitfalls seen when agents wrote
worlds" lists what went wrong when agents wrote nine worlds with the
guide (fake timestamps at 0 instead of virtual
wall clock time, no unwind for a consumer that loops forever, a fake that
blocks under the consumer's lock, missing
fault-excuse records, old traces in `target/dst`, a test log that filled
the disk, a fixed endpoint configuration),
with matching checklist items.
8. **Registration.** `MojoHelper` lists the module under `camel-test`; the
catalog copies, the BOM entry, the docs nav
entry and the doc symlinks are committed. The module's surefire config
pins `camel.surefire.parallel=false`.
9. **One module doc.** `README.md` and `DEMO.md` of the module are removed
(they repeated the doc page); the run
commands, the limits and the table of the classes move to `test-dst.adoc`.
10. **`runFor` returns when two periodic tasks block each other.** A
blocking task moves the clock past the target of
`runFor`; two periodic tasks that block each other kept each other due,
and `runFor` never returned (100 of 459 seeds
of a Kinesis world ran into the step budget). `runFor` now returns once
the clock passed its target.
`DstSchedulerTest`.
11. **A truncated trace keeps its end.** The trace kept only its first 200
000 lines, so the end of a long run, where it
failed, was lost; it now keeps the first half and a rolling tail.
`DstSchedulerTest`.
12. **Parallel streams.** `parallelStream()` of the `java.util` collections
in the code under test ran on the common
fork-join pool and was not in the escape scan; it is rewritten to
`stream()` and listed by the scan.
`DstInstrumentationTest`.
## Results
Each world is now a regression test in its component, 1000 seeds by default,
2 to 3 ms per seed, strict mode.
| World | `main`, 5000 seeds | fix reverted locally, 5000 seeds |
|---|---|---|
| camel-azure-eventhubs checkpointing (CAMEL-25683, #27922) | 1 failing
(seed 3042, see below) | 2064 |
| camel-slack suspend mid-poll (CAMEL-25684, #27923) | 0 | 2652 |
| camel-aws2-sqs visibility extender (CAMEL-25700, #27985; world by Claus
Ibsen) | 0 | 1397, all with `maxMessagesPerPoll >= 2` and `concurrentConsumers
>= 2` |
`main` with `visibilityTimeout=1` only: 843 of 2000 seeds fail (CAMEL-25737,
still open; left out of the default set).
Event Hubs seed 3042 on `main`: an event completes in the same millisecond
in which the timer of its window ran;
`EventHubsCheckpointUpdaterTask.isExpired()` compares with `>`, so the
window is reused and the event is checkpointed
only with the next event of the partition (if the partition stays quiet, it
is delivered again after a restart). In
real time the window is narrow but not only that millisecond: the timer task
does not take the consumer's lock, so it
can also run between the `isExpired()` check and the count of the event. Low
severity; documented in the test, not
fixed in this PR. Replayed in two JVMs, it gives the same trace (SHA-256
`1137160c...`).
Other worlds written with the guide (not part of this PR, kept outside the
tree): fake worlds for the code before
CAMEL-25709 (aggregate), CAMEL-25679 (mongodb tailable), CAMEL-25712
(google-pubsub synchronous pull) and CAMEL-25711
(mongodb-gridfs) fail in 427, 3019, 1316 and 2578 of 5000 seeds and pass on
`main`. A fake DynamoDB Streams world
written from the AWS API reference found a new defect on `main`, CAMEL-25741
(aws2-ddbstream leaves a closed shard
before reading it to its end; 1253 of 3000 seeds), which has its own fix and
plain JUnit test.
## Not in this PR (later stages, documented on the doc page)
- **A time source in Camel core** (stage 1 item 1, the decision between a
hook in core and rewriting byte code): the
proposal is the option from the TIP on CAMEL-25738, a small static time
source in camel-util (`currentTimeMillis`,
`nanoTime`, `instant`, `sleep`) used only by the 25 to 30 call sites that
change behaviour (the waits and the
consumer timing), with byte code rewriting kept for the component under
test and pointed at it. It touches core, so
it goes to dev@ first with the call-site list and a JMH benchmark. Until
then `@DstInstrument(alsoRewrite)` is a
stop-gap; core still reads `System` directly.
- **The one-stack model**: a blocked task resumes only after the tasks that
blocked after it; a timed wait notices its
timeout only when the tasks above it return; a route stop cannot see a
synchronously blocked exchange below it finish
(the Slack world keeps its hand-made suspend sequence for that reason).
Stage 1 documents it; a fix may need virtual
threads with a custom scheduler.
- **Reactor and other third-party schedulers** (the Azure SDKs): outside the
harness; strict mode fails the seed when
one is reached, so the worlds fake the clients above them.
- **Several runs in parallel in one JVM**: the active run is a static field,
so it is one run per JVM at a time.
- Sharing the fault rules with the chaos proposal.
## Testing
```
./mvnw -Papache-snapshots test -pl
components/camel-test/camel-test-dst,components/camel-aws/camel-aws2-sqs,\
components/camel-azure/camel-azure-eventhubs,components/camel-slack
-Dtest='Dst*Test,*DstTest' \
-Dsurefire.failIfNoSpecifiedTests=false
```
43 harness self-tests pass, and the whole test suites of camel-aws2-sqs
(1077 tests), camel-azure-eventhubs (1036)
and camel-slack (1019) pass with the worlds in them (1000 seeds each), so
the rewriting of the component classes in
the same surefire JVM does not disturb the other tests (whole suites run
before items 9 to 12, which change only DST
runs; the three components do not call `parallelStream`). The counts in the
results table were re-run on this branch
(5000 seeds each, with and without the fixes), after the harness fixes 10 to
12 as well; the seed 3042 trace
hash is unchanged. `formatter:format impsort:sort` applied.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
