Christopher Popp a écrit :
Hi,
Hi,
comments inline
I wanted to get some feedback. (MINA 2.0.0 M6).
We're currently doing inrush testing of clients against our mina based
application. We're able to have 1024 clients connect in roughly simultaneously
(30 second window), however we're not able to get our system to stabilize when
doing 2048 or 4096 simultaneous clients.
On connection, there is an exchange of messages between the client and server, and a couple database calls that take place per client.
Blind guess : a couple of database calls will most probably be the
contention point in your system. From some tests I conducted with
Oracle, a request/response loopback with a DB costs a few ms. Even f it
sounds a small number, this is a clear limitation when you don't have
tens of CPU...
But this is just a blind guess. My advice here would be to use a
profiler to check what's going on exactly. Another option would be to
mockup the DB and see how MINA will behave when you have moved aside the
potential costly operations.
We also have application level keep-alives that we need to send out, otherwise
the device will disconnect. Our keep-alive filter basically implements the
sessionIdle method, and when it expires for the server, we write a keep-alive,
and when it expires for the device, we disconnect the device.
Based on a network trace, we are not seeing keep-alives sent out when they
should be under this load. Our presumption is that this is due to mina's
worker threads being too busy with database processing and such.
acceptor = new NioSocketAcceptor(Runtime.getRuntime().availableProcessors() +
1);
acceptor.getFilterChain().addLast("protocol", new ProtocolCodecFilter(new
XXCodecFactory()));
acceptor.getFilterChain().addLast("threadPool", new
ExecutorFilter(workerThreadPool));
acceptor.getFilterChain().addLast("keepAlives", newXXKeepAliveFilter(new
MTKeepAliveMessageFactory());
acceptor.setHandler(devHandler);
During these tests we were using the default constructor for the ExecutorFilter, limiting us to 16 worker threads. We have since changed this to be configurable and will bump the number of threads up quite a bit. We're expecting this to improve the situation quite a bit. However, it seems that if for whatever reason this constant number of threads get tied up doing work, our keep alive processing will halt, and clients will disconnect which eventually could cause a cyclic destabilizing of the system.
Again, profiling the application could help here.
What would be nice is to be able to have a set of threads that are dedicated to
handling keep-alives. One thing I thought of is to add another ExecutorFilter
dedicated to handling SESSION_IDLE events, similar to what is shown in the
javadocs for WRITE IoEvents in the javadocs. This way, a thread should still
pass on the SESSION_IDLE events to the keep alive filter we have. Some
questions about this though. If from the idle event in the keep alive filter I
attempt to write the keep alive, will this end up being blocked due to the fact
that there are no extra threads from the main executor available? (And hence,
I actually need two extra executor filters...one for SESSION_IDLE and one for
WRITE). And if I decide to add these extra executor filters, where should they
go in the filter chain in relation to the other executor filter? It just seems
that if an executor is out of threads, it would block the filter chain
regardless of whether there is an
executor earlier on in the chain that has contributed a thread. I would have
to assume this is handled somehow though.
Also, is there any benefit to increasing the number of IO processor threads to more than the number of cores + 1?
I don't think so, as soon as you have an executor in the chain. The
IoProcessor is the class which manage the selector, and I'm not sure
that having many selectors can help. However, I have no metric on that ...
I read that as the suggested starting point, but in the somewhat outdated
documentation on the threading model,
(http://mina.apache.org/configuring-thread-model.html), it is stated the a
newCachedThreadPool is suggested for IoServices.
yeah, it's totally outdated and need to be improved A LOT. In fact, we
need to conduct some extensive tests and see what could be the best
configuration for a different set of use cases...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
