I got my prototype ThreadNode cleanup code working again. It seems to be
doing the job of keeping the list down to a minimal size, with usually
at most a few ThreadNodes on the list. However, that didn't help
performance. The reason is because the debug agent normally does not
need to traverse the list. It maps jthread to ThreadNode by using JVMTI
thread local data, not by searching the list, and the list is doubly
linked, so ThreadNodes can be removed quickly.
After the above results I talked with Serguei and he confirmed that
JVMTI also has a linked list, and there are some occasions where it
needs to be traversed, so that likely the bottleneck here.
Regarding my suggestion that we may be able to disable
JVMTI_EVENT_VIRTUAL_THREAD_START/END, I did experiment with that and
when disabled there are no performance issues, so I will look into
getting this to work properly. The events need to be enable if there are
any ThreadStartRequests or ThreadDeathRequests that do not have the
PlatformThreadOnly filter enabled.
https://docs.oracle.com/en/java/javase/24/docs/api/jdk.jdi/com/sun/jdi/request/ThreadStartRequest.html#addPlatformThreadsOnlyFilter()
Jdb by default uses this filter, and I suspect that IDEA does also. If
not, it would get a flood of ThreadStart and ThreadDeath events for all
the virtual threads.
Chris
On 4/3/25 2:15 AM, Serguei Spitsyn wrote:
Hi Egor,
Thank you for reporting this scalability issue when debugger is enabled.
It looks like a JVMTI problem and we have some guesses but need some
investigation to identify it better.
Thanks,
Serguei
*From: *serviceability-dev <serviceability-dev-r...@openjdk.org> on
behalf of Egor Ushakov <egor.usha...@jetbrains.com>
*Date: *Wednesday, April 2, 2025 at 3:32 AM
*To: *Chris Plummer <chris.plum...@oracle.com>, serviceability-dev
<serviceability-...@openjdk.java.net>
*Subject: *Re: Debugger overhead for virtual threads creation
Thanks Chris!
I've made the bug https://youtrack.jetbrains.com/issue/IDEA-365900
visible, there's a reproducer there.
Thanks,
Egor
On 02.04.2025 01:39, Chris Plummer wrote:
> The short answer is yes. The debug agent needs to deal with
> JVMTI_EVENT_VIRTUAL_THREAD_START/END events for every virtual thread.
> What makes it worse is when there are a large number of virtual
> threads that are currently alive. They are tracked on a list of
> ThreadNodes that starts to slow down debug agent performance when it
> gets too long. I have a work in progress that proactively purges these
> ThreadNodes so the list does not get too big. I've been meaning to
> revive this project for quite some time. If you have a test case I'd
> be willing to experiment with these changes some more. I could not
> access to the IDEA-365900 link you provided.
>
> Note I think after the work is done to purge ThreadNodes proactively
> it might not be that hard of step to move to not needing
> JVMTI_EVENT_VIRTUAL_THREAD_START/END events enabled, which will help
> performance a lot more.
>
> Chris
>
> On 4/1/25 10:14 AM, Egor Ushakov wrote:
>> Hi everyone!
>>
>> Is it expected that with the debugger attached creating virtual
>> threads is much slower?
>> We're getting bugs like:
>> https://youtrack.jetbrains.com/issue/IDEA-365900
>> And I can reproduce it easily with jdb...
>> Just attaching the debugger immediately slows down virtual threads
>> creation significantly.
>>
>> >java
>> -agentlib:jdwp=transport=dt_shmem,server=y,suspend=n,address=8000 app
>> ...
>> 6808805 (1.2046688E7 threads per second)
>> ...
>> after >jdb -attach 8000
>> ...
>> 30215 (95986.055 threads per second)
>> ...
>>
>> Thanks,
>> Egor
