[
https://issues.apache.org/jira/browse/CAMEL-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128211#comment-18128211
]
Claus Ibsen commented on CAMEL-25738:
-------------------------------------
Attached {{camel-test-dst-sqs-world.patch}}: a third world, for the visibility
timeout extender of camel-aws2-sqs, written by an AI coding agent (Claude Code)
as evidence that a fake world can be written for another component, and to find
the harness gaps listed above. It applies on top of the
{{camel-test-dst-prototype}} branch ({{git am}}) and touches only the
camel-test-dst module.
* {{FakeSqs}} (254 lines): a fake SQS standard queue written from the SQS API
reference, not from the consumer; each rule quotes the page it comes from, and
the two assumptions are marked.
* {{SqsWorld}} (166 lines): a producer sending bursts, a route with slow and
failing processing, faults on every SQS call; the seed also picks
{{visibilityTimeout}}, {{maxMessagesPerPoll}} and {{concurrentConsumers}}.
Invariant: no message is delivered again while the consumer holds it (plus
every message completes and the queue drains).
* {{DstClock}}/{{DstInstrumentation}}: {{Instant.now()}} reads virtual time
(item 1 of the scope).
Results, about 10 s per 5000 seeds:
||Code||Seeds||Failing||
|main ({{-Ddst.sqs.visibilityTimeouts=2,3,5,10}})|5000|0|
|{{Sqs2Consumer}} before CAMEL-25700 ({{d09593b4a584^}})|5000|1383, only with
{{maxMessagesPerPoll >= 2}} and {{concurrentConsumers >= 2}}|
|main without the {{Instant.now()}} fix|500|271 (false failures)|
|main, {{visibilityTimeout=1}} only|2000|876 (CAMEL-25737)|
Seed 8 on the code before CAMEL-25700 replays twice with the same trace SHA-256
({{f075f2c9...}}, 560 lines).
{code}
mvn -pl
components/camel-aws/camel-aws2-sqs,components/camel-test/camel-test-dst test
-Dtest=SqsVisibilityDstTest \
-Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0
\
-Ddst.demo=true -Ddst.seeds=5000 -Ddst.maxFailures=100000
-Ddst.sqs.visibilityTimeouts=2,3,5,10
{code}
What it took: the fake itself was mostly mechanical; the work was in the
harness. The escape scan flagged {{Instant.now()}} only in {{escapes.txt}}
(item 5), a poll of more than 10 messages would block forever on
{{CompletableFuture.get()}} (item 4, avoided by the configuration), and because
of the one-stack model the fake's visibility change does not block (documented
in {{FakeSqs}}; part of the guide, item 7). Like the other demos, it should
move to the tests of camel-aws2-sqs (item 6).
_Claude Code on behalf of Claus Ibsen_
> camel-test-dst - deterministic simulation testing for Camel consumers (stage
> 1)
> -------------------------------------------------------------------------------
>
> Key: CAMEL-25738
> URL: https://issues.apache.org/jira/browse/CAMEL-25738
> Project: Camel
> Issue Type: New Feature
> Components: camel-test
> Reporter: Claus Ibsen
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
> Attachments: camel-test-dst-sqs-world.patch
>
>
> A new {{camel-test-dst}} module for deterministic simulation testing of Camel
> consumers: one seeded scheduler runs every Camel thread pool on the test
> thread, time is virtual, a fake of the external system injects faults chosen
> by the seed, invariants are checked after each run, and a failing seed
> replays with the same trace.
> The prototype is on the branch {{camel-test-dst-prototype}} of
> github.com/allthingssecurity/camel
> ({{components/camel-test/camel-test-dst}}). This issue is the scope of stage
> 1: making the harness trustworthy enough that a fake world can be written for
> a component without running into the harness. The chaos proposal waits until
> this is done.
> h3. Evidence so far
> A third world was written for camel-aws2-sqs (the visibility timeout
> extender, a fake SQS queue written from the AWS API reference, 458 lines):
> * main: 0 of 5000 seeds fail; {{Sqs2Consumer}} before CAMEL-25700: 1383 of
> 5000 fail, only with {{maxMessagesPerPoll >= 2}} and {{concurrentConsumers >=
> 2}}, as that fix describes; a failing seed replays with the same SHA-256.
> * it found CAMEL-25737 ({{visibilityTimeout=1}}), because the seed also picks
> the endpoint configuration.
> * it needed a harness fix first (item 1): without it, 271 of 500 seeds on
> main were false failures.
> h3. Stage 1 scope
> # *Time read outside the instrumented package.* Byte code is rewritten only
> in the package of {{@DstInstrument}}, and {{java.time}} was not rewritten at
> all ({{Instant.now()}} in the SQS extender). Core clocks still read real
> time: {{MonotonicClock}} (camel-support), {{StopWatch}} (camel-util),
> {{ScheduledPollConsumer}}. Decide between rewriting the core packages too and
> a small time-source hook in core that DST replaces (the README suggests the
> hook; it touches core, so it needs a dev@ discussion).
> # *Real sleeps on the loop thread.* {{RedeliveryPolicy.sleep()}}
> ({{Thread.sleep}}, used by the default synchronous redelivery,
> {{asyncDelayedRedelivery=false}}), {{ForegroundTask}} (used by 10 component
> source files), {{DefaultShutdownStrategy}} (waits for in-flight exchanges
> with {{Thread.sleep}}; the Slack world copies the suspend sequence by hand
> because of it). A virtual sleep that becomes {{dst.block()}} on the loop
> thread.
> # *Timed waits never time out.* {{DstTask.get(timeout)}} ignores the timeout
> and {{awaitTermination}} returns true, so timeout paths (circuit breaker,
> shutdown, {{Future.get(timeout)}}) cannot be explored. Timeouts must fire in
> virtual time.
> # *Waits on something that is not a DST task deadlock.*
> {{CompletableFuture.get()}}/{{join()}} or {{CountDownLatch.await()}} on the
> loop thread hang (the SQS consumer with more than 10 messages per poll). At
> least fail fast with a clear message; better, rewrite them to run the loop
> until the future completes.
> # *Escapes fail the seed, not only the report.* The {{Instant.now()}} gap
> showed up only as a line in {{escapes.txt}}; a mode where reaching an
> uncontrolled escape at runtime fails the seed makes the cause obvious.
> # *Module layout.* camel-test-dst holds the harness only; the demos
> (azure-eventhubs, slack, and the sqs world) move to the tests of their
> component modules as passing regression tests. Refresh the README numbers:
> the fixes of #27922 and #27923 are on main.
> # *A guide for AI agents (and humans) on writing a fake world*, in the module
> doc page with a pointer in {{AGENTS.md}}, using the SQS world as the worked
> example:
> #* find where the client is swapped (an endpoint option such as
> {{amazonSQSClient}}, else a static mock of the factory);
> #* run the escape scan before writing anything, and fix, avoid or document
> every escape;
> #* write the fake from the service's documentation, quoting the rule for each
> behaviour and marking assumptions;
> #* the one-stack timing rule: a blocked task resumes only after the tasks
> that blocked after it, so reads take effect at the end of their latency,
> writes at the start, and background tasks (extenders, timers) do not block;
> #* let the seed pick the endpoint configuration as well as the timing;
> #* seeded faults, switched off for a drain phase, with a record of which
> injected fault excuses which violation;
> #* the invariant from the component's contract, starting from the reusable
> {{Invariants}};
> #* the self-check: fail on the code before a past fix ({{git log}}), pass on
> main, and the same SHA-256 when a seed is replayed;
> #* triage before reporting: group failing seeds by configuration, read one
> trace, tell harness artifact, fake mistake and real bug apart.
> Each harness fix comes with a self-test, like {{DstSchedulerTest}}.
> h3. Later stages (not in this issue)
> * The one-stack model itself (a real fix may need virtual threads with a
> custom scheduler); stage 1 documents it.
> * Async clients on Reactor's own schedulers (the Azure SDKs).
> * Several runs in parallel in one JVM (the active run is a static field).
> * Sharing the fault rules with the chaos proposal.
> _Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
