Christopher Popp wrote:
Hi,I wanted to get some feedback. (MINA 2.0.0 M6). We're currently doing inrush testing of clients against our mina based application. We're able to have 1024 clients connect in roughly simultaneously (30 second window), however we're not able to get our system to stabilize when doing 2048 or 4096 simultaneous clients. On connection, there is an exchange of messages between the client and server, and a couple database calls that take place per client. We also have application level keep-alives that we need to send out, otherwise the device will disconnect. Our keep-alive filter basically implements the sessionIdle method, and when it expires for the server, we write a keep-alive, and when it expires for the device, we disconnect the device. Based on a network trace, we are not seeing keep-alives sent out when they should be under this load. Our presumption is that this is due to mina's worker threads being too busy with database processing and such. acceptor = new NioSocketAcceptor(Runtime.getRuntime().availableProcessors() + 1); acceptor.getFilterChain().addLast("protocol", new ProtocolCodecFilter(new XXCodecFactory())); acceptor.getFilterChain().addLast("threadPool", new ExecutorFilter(workerThreadPool)); acceptor.getFilterChain().addLast("keepAlives", newXXKeepAliveFilter(new MTKeepAliveMessageFactory()); acceptor.setHandler(devHandler); During these tests we were using the default constructor for the ExecutorFilter, limiting us to 16 worker threads. We have since changed this to be configurable and will bump the number of threads up quite a bit. We're expecting this to improve the situation quite a bit. However, it seems that if for whatever reason this constant number of threads get tied up doing work, our keep alive processing will halt, and clients will disconnect which eventually could cause a cyclic destabilizing of the system. What would be nice is to be able to have a set of threads that are dedicated to handling keep-alives. One thing I thought of is to add another ExecutorFilter dedicated to handling SESSION_IDLE events, similar to what is shown in the javadocs for WRITE IoEvents in the javadocs. This way, a thread should still pass on the SESSION_IDLE events to the keep alive filter we have. Some questions about this though. If from the idle event in the keep alive filter I attempt to write the keep alive, will this end up being blocked due to the fact that there are no extra threads from the main executor available? (And hence, I actually need two extra executor filters...one for SESSION_IDLE and one for WRITE). And if I decide to add these extra executor filters, where should they go in the filter chain in relation to the other executor filter? It just seems that if an executor is out of threads, it would block the filter chain regardless of whether there is an executor earlier on in the chain that has contributed a thread. I would have to assume this is handled somehow though. Also, is there any benefit to increasing the number of IO processor threads to more than the number of cores + 1? I read that as the suggested starting point, but in the somewhat outdated documentation on the threading model, (http://mina.apache.org/configuring-thread-model.html), it is stated the a newCachedThreadPool is suggested for IoServices. Thanks for any question or comments, Chris
Hi Chris, If your server is truly busy doing IO processing, then increasing the number of threads in theory would just increase contention and stack usage. The rationale at least from what I've read is that putting one more than the number of cores is the underlying operating system scheduler and by doing so, you always have a thread waiting to have work done. If the Mina framework for some reason is not scheduling the application keep alive in a timely manner, then you could probably just obtain a handle to a ScheduledExecutorService from java.util.concurrent.Executors and schedule tasks to send keep alives from there. It would also help isolate where the problem is occurring (e.g: if the keep alives are operating as expected, then that would narrow the problem further to your application (including framework) before the change, and if not, well you still have to keep digging at what is eating up the processor time) Best, Johnny
