Rainer,
On 3/20/24 13:33, Rainer Jung wrote:
Am 20.03.24 um 18:17 schrieb Christopher Schultz:
Rainer,
Thanks for writing this up.
On 3/20/24 07:22, Rainer Jung wrote:
I wanted to share an observation and I hope the things are correct
how I am describing them. Maybe things have already improved and I am
not aware of it, hints welcome.
Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
handle observability of virtual threads from inside the JVM and tooling.
The final outcome is, that virtual threads are not included in the
typical JVM APIs which one can use to observe threads, like
enumerating them or accessing the stacks of individual threads. As a
consequence, also jstack and "kill -QUIT" do not show virtual threads
at all, not even when they are attached to a native thread and
executing code.
O_O
There is one single method to help with observability of virtual threads
https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
which dumps a full thread dump to a file. Of course that is by no
means appropriate, if you want to do a fine grained observation. At
least you can choose to write a json structure instead of a hard to
parse text format, but that's it.
Boy, that's a real miss from the JVM team. I'm surprised that they
have overlooked this. I don't see a real reason that a Virtual Thread
would be treated any differently than a regular thread.
The JEP hints at "since thre could be millions of virtual threads it
makes no sense to support them in the existing JMX APIs". I don't by
this extremely simplistic reasoning.
:/
Furthermore, the "dump to file" method they provided does not contain
any locking information you usually expect in a thread dump.
+1
For instance I am often using a tool, that inspects our
RequestProcessors to find long running requests and then retrieves
the list of Java threads as ThreadInfo objects to find the one
executing the request and finally retrieves the stack of that request
from the JVM. Such an approach is no longer possible. Almost all of
JMX does not show any info about virtual threads.
It also seems Tomcat no longer uses a pool when a connector is
configured to use virtual threads. So there are no metrics like
currentThreadCount or currentThreadsBusy.
When using virtual threads, the number of requests is the same as the
number of threads, since each request -> new Virtual Thread.... though
I think the request-count is only incremented when the request has
completed, so maybe there are threads running that can't be counted
(yet).
But I understand that you were hoping to get some information about
these threads and though maybe Tomcat's metrics could help. I guess
not really.
But thread-busy count is the same as in-flight-request count. And
current thread count is the same as in-flight-request count as well.
I guess this also limits the use of some manager features and
probably also the StuckThreadsDetectionValve when combined with
virtual threads.
Most likely. I'm fairly sure that uses the "usual Thread-walker
things" which you are reporting do not work any more with VTs.
What JVM are you using for your investigations?
Zulu 21, but I expect the other OpenJDK variants to behave the same way.
It mostly comes out of the JEP.
Of course Tomcat has solved most of the problems, that virtual
threads want to solve, by doing the hard work of using the NIO and
NIO2 APIs. So virtual threads are probably not that important for us.
But we should be aware of the observability deficiencies whenever we
would discuss whether we could switch from NIO or NIO2 to virtual
threads to simplify connector code in some distant future.
+1
I've been enthusiastically talking with markt about dropping all that
nasty NIO stuff and "going back to BIO with virtual threads" which, at
this point, is mostly a threat and not a promise. But if VTs really
deliver everything they are claiming to deliver, then it may be
possible to go back to BIO as long as servlet-async is retired. And
I'm not holding my breath on that one.
There was an interesting talk at FOSDEM about the current limitations of
the virtual thread implementation. They have several situations where
they switch over to blocking. I think for instance file I/O and also
some types of locks they can not yet handle well.
Yes, these are known issues. Specifically, synchronized blocks and
virtual threads don't play well together, which is why Tomcat recently
converted many synchronized { ... } to use ReentrantLocks, which DO work
with (today's) VTs.
They have some improvements in the pipeline, but I think one would at
least wait for those. And hopefully they improve the observability in
the meantime.
+100
I wish I had known this type of problem at FOSDEM so I could have
directly talked to the OpenJDK devs there.
There's always next year :)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
