Hi JB,
Thanks for the +1 and good to hear from you :-) On the follow-ups: 1. Forbidden calls: you're right, there's no check in the build yet, and Camel doesn't use the forbidden-apis plugin today. To start small, I'd add a test in camel-util that scans the sources of the moved classes and fails on System.nanoTime(), System.currentTimeMillis() and Thread.sleep(), with an allow-list for the TimeSource default itself. If people would rather have the forbidden-apis plugin for this, I can do that instead. 2. API or not: I'd keep it internal, as the javadoc says. Components get virtual time through the core classes they already use (the task classes, the delayer, the throttlers, the redelivery), and camel-chaos stays the one exception. My proposal said components would adopt it when touched; I'll correct that, so we don't end up with a public API in org.apache.camel.util by accident. If a component later needs it directly, we can discuss that case on its own. 3. Static provider: agreed. I'll log a line at INFO when a non-default provider is installed, and when the default one is put back, naming the provider class, so it shows up if it ever leaks out of a test harness in a shared runtime like Karaf. 4. camel-performance-tests: the numbers I posted came from a local JMH harness; the benchmark isn't in camel-performance-tests yet. I'll add it there, so anyone can rerun it on x86 or a new JDK. I'll open a JIRA for 1-3 and the PRs over the next days. The next worlds (IMAP, NATS JetStream, Azure Service Bus and Storage Queue, JMS) will come with their components' fixes. Good to hear about Karaf and Camel Karaf. Is there a JIRA or a branch I can look at? If they get a similar time source, it would be nice to line them up, so a test can drive one virtual clock across Karaf, Camel Karaf and Camel. Happy to help with that. Thanks, Shashank On Sun, 8 Nov 2026 at 16:21, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Shashank > > Sorry for the late reply. A late +1 from me as well. Thanks for the clear > write-up and posting the numbers. > > For anyone reading the archive later, this has landed: > - CAMEL-25911 (#28382): TimeSource in camel-util, plus the call sites > listed in this thread (redelivery, delayer, throttlers, the task classes, > the shutdown wait loop, StopWatch, MonotonicClock). > - CAMEL-26103 (#28782): the agent is now loaded with -javaagent instead of > a runtime attach. > - The send and step hooks (CAMEL-26331, CAMEL-26338) and camel-chaos also > read time through TimeSource, so chaos delays run in virtual time too. > > I have a few follow-ups now that it's in: > > 1. Your proposal included a forbidden-apis rule, but I don't see one in the > build yet. Without it, someone adding a System.nanoTime() or Thread.sleep() > to one of these classes would take that wait out of virtual time, and > nothing would catch it. A check limited to the classes that were moved > would be enough to start with. > 2. The javadoc says TimeSource is not an API for components (camel-chaos > excepted). The proposal said components would adopt it when they are > touched anyway. We should decide which one we mean. If components are > expected to use it, it becomes a public API in org.apache.camel.util in > practice, and we should treat its compatibility that way. > 3. The provider is static, which is fine for tests. In a shared runtime > like Karaf, though, one camel-util serves every CamelContext, so an > installed provider affects all of them. Logging a line when a non-default > provider is installed would make it easy to spot if this ever leaks outside > a test harness. > 4. +1 to putting it in camel-performance-tests, if that hasn't happened > yet. Then anyone can rerun it on x86 or on a new JDK. > > Looking forward to the next worlds (IMAP, NATS JetStream, Azure Service Bus > and Storage Queue, JMS). > > FYI, there's something similar in Karaf and Camel Karaf coming. > > Regards > JB > > On Tue, Oct 20, 2026 at 3:08 AM Shashank Jain <[email protected]> > wrote: > > > Hi Claus, > > > > Thanks for the +1. > > > > Unless someone has concerns, I will leave the thread open a few more days > > and then open a JIRA for it as stage 2 of > > CAMEL-25738: the TimeSource in camel-util, the ~30 call sites in core > that > > change behaviour, and camel-test-dst > > installing it, so the rewriting is only needed for the component under > > test. > > > > On more fake worlds: I have worlds for IMAP, NATS JetStream, Azure > Service > > Bus, Azure Storage Queue and JMS that > > found or confirmed bugs this week, and I plan to add them to their > > components together with the fixes. > > > > Looking forward to hearing your other ideas for controlling time once DST > > and chaos are in place. > > > > Thanks, > > Shashank > > > > On Mon, 19 Oct 2026 at 22:53, Claus Ibsen <[email protected]> wrote: > > > > > Hi Shashank > > > > > > Thanks for doing these benchmarks so we can see what the affect would > be. > > > > > > Surely the feature of allowing to "control the time" to make > > deterministic > > > testing in all the complex camel components possible, > > > to ensure that we can have a really really well robust component that > > > handles all kind of edge cases is worth this feature in the core. > > > > > > So its a +1 from me. > > > > > > PS: I have other ideas where this "control the time" can be useful, but > > > lets get the DST and the Chaos feature built first. And more fake > worlds > > in > > > various components so we can make Camel very roboust. > > > > > > > > > > > > On Mon, Oct 19, 2026 at 1:43 PM Shashank Jain < > [email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > Here are the JMH numbers I promised. JMH 1.37, JDK 21.0.11, Apple M4 > > Pro > > > > laptop (8P+4E cores), camel 4.24.0-SNAPSHOT; > > > > 3 forks, 5 x 1 s warmup, 10 x 1 s measurement, average time in > ns/op. A > > > > laptop result, so indicative only. > > > > > > > > 1 thread 8 threads > > > > System.nanoTime() 8.92 +- 0.01 356.1 +- 3.3 > > > > TimeSource.nanoTime(), volatile 9.10 +- 0.06 354.9 +- 3.2 > > > > TimeSource.nanoTime(), plain 9.15 +- 0.08 358.0 +- 3.3 > > > > TimeSource.nanoTime(), bimorphic 9.22 +- 0.13 355.7 +- 2.8 > > > > System.currentTimeMillis() 13.59 +- 0.09 186.5 +- 7.8 > > > > TimeSource.currentTimeMillis() 13.62 +- 0.02 185.2 +- 9.5 > > > > new StopWatch().taken(), today 19.96 +- 0.04 709.6 +- 7.6 > > > > the same with TimeSource 20.52 +- 0.12 710.8 +- 7.8 > > > > > > > > direct:a -> log:x?level=OFF, route statistics on (today's core, for > > > > scale): > > > > 2.43M msg/s 10.6M msg/s > (8 > > > > threads) > > > > > > > > "bimorphic" means a second provider class was installed and used once > > in > > > > setup, then SYSTEM was put back, so the > > > > call site has seen two receiver types. With 8 threads the clock calls > > are > > > > much slower on this machine for every > > > > variant, System.nanoTime() included; I have not dug into why, but the > > > > variants are the same there. > > > > > > > > What it means: on one thread the indirection costs about 0.2 ns per > > > > nanoTime() call and about 0.5 ns per StopWatch, > > > > nothing measurable for currentTimeMillis(), and nothing measurable > > with 8 > > > > threads. The volatile field is not > > > > measurably slower than a plain static field here (9.10 vs 9.15 ns), > so > > I > > > > would keep volatile, which also keeps > > > > install() safe without extra rules. Against about 410 ns per message > > for > > > > the simplest direct -> log route, a few > > > > tenths of a nanosecond per clock read is well below noise. On x86 a > > > > volatile read is also a plain load, but > > > > I have only measured on AArch64; numbers from an x86 box would be > > > welcome. > > > > > > > > I can put the benchmark in camel-performance-tests if useful. > > > > > > > > Thanks, > > > > Shashank > > > > > > > > On Mon, 19 Oct 2026 at 12:44, Shashank Jain < > [email protected]> > > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > camel-test-dst (CAMEL-25738, PR > > > > https://github.com/apache/camel/pull/28067) > > > > > runs the real consumer code of a > > > > > component under one seeded scheduler with virtual time, so a test > can > > > try > > > > > thousands of interleavings and faults in > > > > > seconds and replay a failing seed exactly. Worlds written with it > > found > > > > or > > > > > confirmed several timing bugs, for > > > > > example CAMEL-25683, CAMEL-25737, CAMEL-25741, CAMEL-25756 and > > > > CAMEL-25758. > > > > > > > > > > The harness controls Camel's thread pools through the existing > > > > > ThreadPoolFactory SPI. Time is the gap: core reads > > > > > System directly, so today a test either rewrites the byte code of > > core > > > > > classes or lives with real time and real > > > > > sleeps in core. Claus suggested a small time source in camel-util, > > and > > > I > > > > > would like to discuss that here before > > > > > anything goes into core. > > > > > > > > > > Where core reads the clock or sleeps today (src/main/java on main, > > > > > 2026-10-17): > > > > > > > > > > camel-util 4 currentTimeMillis, 5 nanoTime, 3 > > Thread.sleep > > > > > camel-support 24 currentTimeMillis, 6 nanoTime, 6 > > > Thread.sleep, > > > > 5 > > > > > Instant.now > > > > > camel-base 43 currentTimeMillis (nearly all event > > > timestamps) > > > > > camel-base-engine 9 currentTimeMillis, 2 Thread.sleep, 1 > > > > > Instant.now, 1 new Date() > > > > > camel-core-processor 6 currentTimeMillis, 7 nanoTime, 5 > > Thread.sleep > > > > > camel-management 9 currentTimeMillis, 1 Thread.sleep, 2 new > > > Date() > > > > > others (api, languages, console, health, main): a handful > > > > > > > > > > About 30 of these change behaviour, and only those need to be > > > controlled > > > > > in a test: > > > > > - waits: RedeliveryPolicy.sleep and the sleep loop of > > > > > RedeliveryErrorHandler, ForegroundTask, the wait loop of > > > > > DefaultShutdownStrategy, DelayProcessorSupport, > MulticastProcessor, > > > > > OptimisticLockRetryPolicy, > > > > > DefaultStartupConditionStrategy; > > > > > - task timing: StopWatch, MonotonicClock, AbstractTask, > > BackgroundTask, > > > > > TimeBoundedBudget, BackOffTimerTask, > > > > > DefaultTimeoutMap; > > > > > - throttling: TotalRequestsThrottler, ConcurrentRequestsThrottler, > > > > > SamplingThrottler, > > > > > ThrottlingExceptionRoutePolicy. > > > > > > > > > > Event timestamps, the debugger, the tracer, management and the > > consoles > > > > > can stay on System. > > > > > > > > > > Options I see: > > > > > > > > > > 1. Byte code rewriting only (what stage 1 does). No core change, > but > > > > > runtime agent attach is being phased out in the > > > > > JDK, it only covers the calls it is told about, and rewritten > core > > > > > classes stay rewritten for the whole test JVM. > > > > > > > > > > 2. A time source for all of core: about 120 call sites, many on hot > > > > paths, > > > > > for no gain in tests. > > > > > > > > > > 3. In between, which is what I would propose: > > > > > - a small static, internal TimeSource in camel-util: > > > > > currentTimeMillis(), nanoTime(), instant(), sleep(long). > > > > > The default delegates to System and Thread.sleep; a test > harness > > > can > > > > > install another provider. Static rather > > > > > than per CamelContext, because StopWatch and MonotonicClock > have > > > no > > > > > context; > > > > > - move only the ~30 call sites above; > > > > > - keep byte code rewriting for the component under test, pointed > > at > > > > > TimeSource, with the agent loaded by > > > > > -javaagent instead of attached at runtime; > > > > > - later, components use it when they are touched anyway, and a > > > > > forbidden-apis rule keeps new > > > > > Instant.now() / Thread.sleep() out of consumers. > > > > > > > > > > Rough shape: > > > > > > > > > > public final class TimeSource { > > > > > public interface Provider { > > > > > long currentTimeMillis(); > > > > > long nanoTime(); > > > > > void sleep(long millis) throws InterruptedException; > > > > > } > > > > > private static volatile Provider provider = SYSTEM; > > > > > public static long nanoTime() { return provider.nanoTime(); } > > > > > ... > > > > > public static Provider install(Provider p) { ... } // test > > > > > harnesses only > > > > > } > > > > > > > > > > Open points: package and name; a volatile field or a plain static > > field > > > > > set once before the context starts; and > > > > > whether sleep belongs there or in a separate wait helper. > > > > > > > > > > On cost: I will run a JMH benchmark (direct System.nanoTime vs. > > > > TimeSource > > > > > with a volatile and a plain field, > > > > > StopWatch, and a direct -> log route with statistics on) and post > the > > > > > numbers in this thread. My expectation is > > > > > that the default path inlines to the same call plus one ordinary > > load, > > > so > > > > > it should be within noise. > > > > > > > > > > Thoughts? > > > > > > > > > > Thanks, > > > > > Shashank > > > > > > > > > > > > > > > > > > -- > > > Claus Ibsen > > > > > >
