Hi,
I wanted to get some feedback. (MINA 2.0.0 M6).
We're currently doing inrush testing of clients against our mina based
application. We're able to have 1024 clients connect in roughly simultaneously
(30 second window), however we're not able to get our system to stabilize when
doing 2048 or 4096 simultaneous clients.
On connection, there is an exchange of messages between the client and server,
and a couple database calls that take place per client. We also have
application level keep-alives that we need to send out, otherwise the device
will disconnect. Our keep-alive filter basically implements the sessionIdle
method, and when it expires for the server, we write a keep-alive, and when it
expires for the device, we disconnect the device.
Based on a network trace, we are not seeing keep-alives sent out when they
should be under this load. Our presumption is that this is due to mina's
worker threads being too busy with database processing and such.
acceptor = new NioSocketAcceptor(Runtime.getRuntime().availableProcessors() +
1);
acceptor.getFilterChain().addLast("protocol", new ProtocolCodecFilter(new
XXCodecFactory()));
acceptor.getFilterChain().addLast("threadPool", new
ExecutorFilter(workerThreadPool));
acceptor.getFilterChain().addLast("keepAlives", newXXKeepAliveFilter(new
MTKeepAliveMessageFactory());
acceptor.setHandler(devHandler);
During these tests we were using the default constructor for the
ExecutorFilter, limiting us to 16 worker threads. We have since changed this
to be configurable and will bump the number of threads up quite a bit. We're
expecting this to improve the situation quite a bit. However, it seems that if
for whatever reason this constant number of threads get tied up doing work, our
keep alive processing will halt, and clients will disconnect which eventually
could cause a cyclic destabilizing of the system.
What would be nice is to be able to have a set of threads that are dedicated to
handling keep-alives. One thing I thought of is to add another ExecutorFilter
dedicated to handling SESSION_IDLE events, similar to what is shown in the
javadocs for WRITE IoEvents in the javadocs. This way, a thread should still
pass on the SESSION_IDLE events to the keep alive filter we have. Some
questions about this though. If from the idle event in the keep alive filter I
attempt to write the keep alive, will this end up being blocked due to the fact
that there are no extra threads from the main executor available? (And hence,
I actually need two extra executor filters...one for SESSION_IDLE and one for
WRITE). And if I decide to add these extra executor filters, where should they
go in the filter chain in relation to the other executor filter? It just seems
that if an executor is out of threads, it would block the filter chain
regardless of whether there is an
executor earlier on in the chain that has contributed a thread. I would have
to assume this is handled somehow though.
Also, is there any benefit to increasing the number of IO processor threads to
more than the number of cores + 1? I read that as the suggested starting
point, but in the somewhat outdated documentation on the threading model,
(http://mina.apache.org/configuring-thread-model.html), it is stated the a
newCachedThreadPool is suggested for IoServices.
Thanks for any question or comments,
Chris
