All,
I have some questions about Virtual Threads and their use within Tomcat.
Note that only Tomcat 11 currently has support for Virtual Threads when
running on a version 19 or later JVM.
My (admittedly limited) understanding is that the use of Virtual
Threads, specifically within Tomcat, will allow Tomcat to use fewer
platform threads (e.g. native OS threads) to support a larger number of
JVM/application threads for request-processing. The justification for
this improvement is that request-processing threads spend a lot of time
in an I/O-blocked state which, when using Virtual Threads, would no
longer consume a platform thread.
There appears to only be one setting to enable Virtual Threads in Tomcat:
<Connector ... useVirtualThreads="true|false" />
or
<Executor ... useVirtualThreads="true|false" />
In both cases, there is only one setting to affect the number of
"threads" (by any description) which the Executor will ultimately use
(Connectors without an explicit Executor will create a non-shared
Executor to be used internally). That setting is "maxThreads" which
limits the total number of "threads" that the Executor will create.
The implementation of the VirtualThreadExecutor does not seem to have
any upper-bound on the number of Virtual Threads that will be created at
all. I believe this means that a large number of incoming requests (i.e.
a burst) will result in the creation of a large number of Virtual
Threads. Without an upper-bound, we are expecting that the JVM's
(virtual) thread scheduler will schedule each application thread to be
mounted to a platform thread in the order it was added to the queue
(essentially in request-order).
Before Virtual Threads were introduced, the Executor would use a queue
of requests to dispatch to available (non-Virtual) threads which would
use that thread until the request was complete, then return the thread
to the pool. With Virtual Threads, the same thing is essentially still
happening except that (a) Tomcat no longer manages the thread pool (a
JVM-defined one is being used instead) and (b) requests are immediately
handed a (Virtual) thread to carry their execution.
I believe there are some subtle differences in how Tomcat will behave,
now. As an example, if I have two applications running, say, the Tomcat
Manager application and the Tomcat Examples application, without using
Virtual Threads, each application's thread pools should be "fair" within
the context of each application: requests are processed in the order
they are received in Manager and Examples, separately. If all requests
are equally "expensive" and everything is "fair", then requests to the
Examples application are scheduled alongside those to the Manager
application, and they can both execute simultaneously as well as
separately-manage the order in which the requests are processed.
Once Virtual Threads are introduced, the requests are filed into a
single JVM-wide thread-scheduling system where activity in one
application can affect the other.
Let's replace Examples with RealWorldApp, an application that is
expected to be used by users. Maybe a LOT of users. Without Virtual
Threads, a high number of requests to RealWorldApp will not cause
starvation of requests to (maybe the more important, at least for
admins) the Manager. Once Virtual Threads are introduced, a limitless
number of requests can be queued in front of a request to Manager, which
can experience starvation.
While Tomcat did not previously implement any specific priority-queuing
of requests, the use of separate Executors for each application
effectively provided that kind of thing. Yes, each Executor can be
configured separately to either use Virtual Threads or not, and so
Manager can be configured to NOT use Virtual Threads while RealWorldApp
can be configured to use Virtual Threads and the balance is "restored".
But it is no longer possible to have RealWorldApp and RealWorldApp2 and
Manager all with equally probable request-scheduling when using Virtual
Threads. You can pick some subset of applications to get (essentially)
priority by NOT using Virtual Threads, but the whole set of applications
running in a single JVM will share a single Executor with FIFO behavior.
If an application creates Virtual Threads (hey, why not?! they are
cheaper!) then they will be scheduled alongside the request-processing
threads as well.
Do I understand things correctly? Is my scenario of request-starvation
for a little-used but potentially critical application (e.g. Manager) a
real concern, or am I missing something fundamental about the way
Virtual Threads are scheduled relative to other Virtual Threads, and
relative to non-virtual threads?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
