Hi serviceability-dev and hotspot-jfr-dev, I am looking into CPU profiling support for Virtual Threads and Kotlin coroutines. Below are our conclusions regarding heap access, our prototyping experience with RequestStackTrace, and the path forward in GoogleJDK. I would like to learn opinions and options from the OpenJDK community.
1. Background & Technical Motivation ----------------------------------- Profiling asynchronous and concurrent execution models—specifically Kotlin coroutines and Virtual Threads—presents a challenge for ASGCT: - Kotlin Coroutines: Implemented via Continuation-Passing Style (CPS). Suspended coroutines save their logical callers as a linked list of Continuation objects (not the same as Virtual Thread's Continuation) directly on the Java heap. - Virtual Threads: Unmounted frames and continuation states reside in heap-allocated StackChunk objects. 2. The Verdict: Unviability of Accessing Heap Objects from ASGCT ---------------------------------------------------------------- We explored whether the established profiling model (walking frames inside a SIGPROF signal handler via an extended ASGCT) could be adapted to traverse heap-allocated continuation objects. Our conclusion is that there is fundamentally no safe way to dereference or traverse Java heap objects from ASGCT: - Concurrent Relocation & Load Barriers (ZGC & Shenandoah): Modern collectors relocate heap objects concurrently while mutator threads execute. Accessing heap pointers from ASGCT bypasses load barriers or invokes barriers in prohibited contexts (e.g., during active relocation or page allocation), leading to immediate VM crashes or memory corruption. - STW Evacuation Races on Java Threads in Native (_thread_in_native): When a JavaThread executes native code via JNI/FFM or system calls, its state is _thread_in_native and it is considered safepoint-safe; STW collectors (such as G1) do not wait for it before starting evacuation pauses. If SIGPROF interrupts a JavaThread while in native, GC worker threads may be concurrently evacuating objects on the Java heap. Dereferencing heap pointers from ASGCT races directly against object copying and mark word updates. - Forwarding Pointers & Compact Object Headers: During evacuation, GC installs forwarding pointers into the object's markWord. With Compact Object Headers (-XX:+UseCompactObjectHeaders), decoding oop->klass() from a forwarding pointer results in an invalid pointer dereference and a fatal SIGSEGV. - Lack of JIT OopMaps at Arbitrary Signal PCs (Cannot Inspect Locals or Arguments): JIT compilers (C1/C2) only generate OopMap records at designated safepoint PCs (e.g. calls, backedges, safepoint polls). Because SIGPROF interrupts execution at an arbitrary instruction PC, HotSpot has no OopMap to decode which CPU registers or stack slots hold live object references (oops) versus primitive values. Consequently, a stack walker in ASGCT cannot inspect method arguments or local variables (such as the 1st argument Continuation passed to BaseContinuationImpl.resumeWith) to locate the active coroutine root on the stack without intrusive mechanisms (like compiler intrinsics caching references on JavaThread). Based on my estimation, patching G1 GC to be ASGCT-aware would be a substantial and invasive change and would still be forced to drop samples whenever race conditions occur. Therefore, we decided against extending ASGCT. 3. A New Path — RequestStackTrace --------------------------------- Summary of JVMTI RequestStackTrace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roman Kennke's PR (#29067: "8373578: JVMTI replacement for AsyncGetCallTrace", https://github.com/openjdk/jdk/pull/29067) refactors HotSpot's JFR CPU-time thread sampler infrastructure and exposes JFR cooperative sampling (JEP-518) as a JVMTI extension function: RequestStackTrace (or RequestJFRStackTrace). The new API is async-signal-safe: instead of walking stacks synchronously in an arbitrary signal context, it enqueues a request into a thread-local queue to be processed cooperatively when the target thread hits a safepoint poll in a valid VM state. Prototype & Experience ~~~~~~~~~~~~~~~~~~~~~~ We have experimented with RequestStackTrace in Google's JDK. We collected empirical profiling data across microbenchmarks and multi-threaded DaCapo suites, and the results show that it is very promising to replace ASGCT: - Safe for G1 GC and Concurrent GCs (ZGC). - Produces clean, well-formed Java traces without the broken or unrooted stacks frequently observed under ASGCT. - Across DaCapo benchmarks, the number of samples collected by RequestStackTrace is comparable to ASGCT (~90% of ASGCT samples), and the remaining gaps are known and addressable. Missing Features to Address ~~~~~~~~~~~~~~~~~~~~~~~~~~~ However, our evaluation highlighted a few missing features needed for production parity and for our goals with Kotlin: - Support _thread_in_vm in StackWalker::request_stack_trace: StackWalker currently rejects requests when jt->thread_state() == _thread_in_vm (check_state() returns false), silently dropping requests. In real-world workloads, mutator threads spend significant CPU time in _thread_in_vm (e.g., JNI entries, runtime stubs, object allocation slow paths, monitor contention, and class verification), leading to 30%–50%+ sample loss in some workloads. We need a way to safely enqueue biased requests up to the Last Java Frame (LJF) and drain them when transitioning back to Java or native. - Support Mixed Stacktrace (Native Frames): Currently, profilers built on ASGCT plus native unwinding (e.g., async-profiler) can produce mixed stacktraces today. In RequestStackTrace, we need support for mixed traces across two main scenarios: * Native frames before/below the Last Java Frame (LJF): Native code calling into Java via JNI or FFM upcalls. These native caller frames remain on the physical execution stack when StackWalker runs, but are currently skipped because StackWalker only walks Java frames. * Native frames called by the Last Java Frame (LJF): Java code calling out to native C/C++ via JNI or FFM downcalls (e.g., burning CPU in native functions). Because cooperative sampling defers stack walking to safepoints and unwinds starting from the LJF, native frames above the LJF need to be retained. - Be ASGCT-compatible: The stack walker produces ASGCT_CallTrace rather than a JFR event. - Extend the StackWalker to Generate Logical Callers for Kotlin Coroutines: While Project Loom continuations (StackChunk) are handled at the VM level, Kotlin coroutines maintain their logical caller hierarchy on the heap. We prototyped an extension in the cooperative stack walker to traverse the Continuation chain and interleave logical coroutine caller frames into the stack trace. Questions & Discussion Points for OpenJDK ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Status of PR #29067 & Plans for a Public Stacktrace API: Has OpenJDK reached a consensus on providing a new public stacktrace interface (such as JVMTI's RequestStackTrace)? Given Roman's shift to PR #31059 (https://github.com/openjdk/jdk/pull/31059), is JDK-8380294 intended to succeed RequestStackTrace? The updated PR offers less flexibility. Because ASGCT_CallTrace's struct and ABI are widely integrated across our infrastructure, GoogleJDK plans to adhere to it; if JFR events are produced instead, we would need to transform them. - Community Attitude toward Features: What is OpenJDK's / Serviceability's attitude toward those features? In particular, is the JFR CPU profiler team interested in feature 1 (_thread_in_vm) and feature 2 (mixed stack traces)? If the JFR CPU profiler would take features 1 and 2 down the road, it would be well worth refactoring and sharing the underlying data structures and utility classes between the JFR profiler and the JVMTI extension. If not, I think it’s easier to manage our patches if RequestStackTrace just doesn’t depend on JFR CPU profiler. Thanks, --lx
