Mark,
On 9/5/23 15:55, Mark Thomas wrote:
On 05/09/2023 20:38, Christopher Schultz wrote:
All,
I have some questions about Virtual Threads and their use within
Tomcat. Note that only Tomcat 11 currently has support for Virtual
Threads when running on a version 19 or later JVM.
Not quite. All current versions support virtual threads when running on
Java 21 or later.
Thanks for the correction. I just did a quick docs[1] search for
"virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads,
so I assumed it wasn't in there. Maybe we need some documentation
back-ports?
My (admittedly limited) understanding is that the use of Virtual
Threads, specifically within Tomcat, will allow Tomcat to use fewer
platform threads (e.g. native OS threads) to support a larger number
of JVM/application threads for request-processing. The justification
for this improvement is that request-processing threads spend a lot of
time in an I/O-blocked state which, when using Virtual Threads, would
no longer consume a platform thread.
There appears to only be one setting to enable Virtual Threads in Tomcat:
<Connector ... useVirtualThreads="true|false" />
or
<Executor ... useVirtualThreads="true|false" />
In both cases, there is only one setting to affect the number of
"threads" (by any description) which the Executor will ultimately use
(Connectors without an explicit Executor will create a non-shared
Executor to be used internally). That setting is "maxThreads" which
limits the total number of "threads" that the Executor will create.
The implementation of the VirtualThreadExecutor does not seem to have
any upper-bound on the number of Virtual Threads that will be created
at all. I believe this means that a large number of incoming requests
(i.e. a burst) will result in the creation of a large number of
Virtual Threads. Without an upper-bound, we are expecting that the
JVM's (virtual) thread scheduler will schedule each application thread
to be mounted to a platform thread in the order it was added to the
queue (essentially in request-order).
The virtual thread scheduler uses a work stealing queue so it isn't
quite FIFO.
That sounds like nit-picking to me. I suppose if each new Virtual Thread
is assigned to a platform thread when it's initially queued, things can
be executed in an not-strictly-FIFO-manner, but one has to assume work
loads are equal (which they likely are not) and work is spread evenly
across all the platform threads (which is likely) to have a general
conversation about all this. I think the work-stealing ForkJoinPool is
about as close to FIFO as you can get without introducing more expensive
contention in order to enforce strict FIFO while not gaining much in the
way of meaningful "fairness".
But it's probably worth mentioning that the queue might not be "strictly
fair". What *is* fair, though -- and maybe more fair than in the past?
-- is that pipelined requests have to get back in line instead of
jumping the queue. I think the old blocking JIO connector would allow
pipelined requests to skip the queue, but all NIO-based connectors are
"more fair" than that.
Before Virtual Threads were introduced, the Executor would use a queue
of requests to dispatch to available (non-Virtual) threads which would
use that thread until the request was complete, then return the thread
to the pool. With Virtual Threads, the same thing is essentially still
happening except that (a) Tomcat no longer manages the thread pool (a
JVM-defined one is being used instead) and (b) requests are
immediately handed a (Virtual) thread to carry their execution.
I believe there are some subtle differences in how Tomcat will behave,
now. As an example, if I have two applications running, say, the
Tomcat Manager application and the Tomcat Examples application,
without using Virtual Threads, each application's thread pools should
be "fair" within the context of each application: requests are
processed in the order they are received in Manager and Examples,
separately. If all requests are equally "expensive" and everything is
"fair", then requests to the Examples application are scheduled
alongside those to the Manager application, and they can both execute
simultaneously as well as separately-manage the order in which the
requests are processed.
The above assumes each application has a separate thread pool (which
implies a separate Connector).
Yes, I'm sorry, I completely skipped over the "fact" that a separate
<Connector> would be used for such a "priority" application such as the
Manager. My example was assuming that each application had the
possibility to use a separate <Executor>.
Once Virtual Threads are introduced, the requests are filed into a
single JVM-wide thread-scheduling system where activity in one
application can affect the other.
Correct.
Let's replace Examples with RealWorldApp, an application that is
expected to be used by users. Maybe a LOT of users. Without Virtual
Threads, a high number of requests to RealWorldApp will not cause
starvation of requests to (maybe the more important, at least for
admins) the Manager. Once Virtual Threads are introduced, a limitless
number of requests can be queued in front of a request to Manager,
which can experience starvation.
While Tomcat did not previously implement any specific
priority-queuing of requests, the use of separate Executors for each
application effectively provided that kind of thing. Yes, each
Executor can be configured separately to either use Virtual Threads or
not, and so Manager can be configured to NOT use Virtual Threads while
RealWorldApp can be configured to use Virtual Threads and the balance
is "restored". But it is no longer possible to have RealWorldApp and
RealWorldApp2 and Manager all with equally probable request-scheduling
when using Virtual Threads. You can pick some subset of applications
to get (essentially) priority by NOT using Virtual Threads, but the
whole set of applications running in a single JVM will share a single
Executor with FIFO behavior. If an application creates Virtual Threads
(hey, why not?! they are cheaper!) then they will be scheduled
alongside the request-processing threads as well.
Do I understand things correctly? Is my scenario of request-starvation
for a little-used but potentially critical application (e.g. Manager)
a real concern, or am I missing something fundamental about the way
Virtual Threads are scheduled relative to other Virtual Threads, and
relative to non-virtual threads?
I think you have summed things up pretty well. I don't see a way with
the current API to specify multiple virtual thread schedulers (which is
what I think you would need to address this).
Yeah, in all of the coverage I've seen online, there are no examples
where Virtual Threads are used with an "executor" other than one that
(a) accepts Runnable tasks and (b) just generates a Virtual Thread from
that task which appears to be "bound" to the build-in default JVM-wide
Virtual Thread executor.
It would be potentially interesting to see an API which would allow
different Executors to be used with Virtual Threads, even though the
whole point of the entire thing is to avoid the (application-level)
complexity of a thread pool/executor/etc. I think it does make sense,
however, to be able to prioritize your threads a little bit. I haven't
read anything about Virtual Threads and their priorities, so I assume
it's just not part of anything user-facing at this point.
Thanks,
-chris
[1] https://tomcat.apache.org/tomcat-10.1-doc/config/executor.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
