Christopher Popp wrote:
Thanks Johnny and Emmanuel for the replies.
I totally expect that the time is being taken up by the database calls, and it
is not anything Mina related that is actually taking the time. Also, my
questions regarding the number of IoProcessors was mainly a curiosity, because
it was a previous developer that had put in the # of Cores + 1, which seemed
reasonable enough but I didn't fully understand it. I don't expect that the
system is overloaded at all due to the IO processing.
In short, I was intending to say that I believe my problem is that every thread
available to the ExecutorFilter (16 by default) eventually gets tied up
handling database calls, so none are available to handle the SESSION_IDLE event
with sufficient speed in my keep alive filter to ensure that clients receive
their expected keep-alives and don't disconnect. With regard to this Johnny,
you're suggestion to try sending keep-alives out of band through my own
ScheduledExecutor is a good one. The only thing I might wonder about
that....since a write has to go all the way through my filter chain, does that
mean even if I use an executor outside of Mina, that if all of the threads of
the executor are busy waiting on a database call, that the write would block as
well?
There is actually another developer who is doing the load testing, so
unfortunately there is a bit of a turn-around between me implementing some of
the configuration parameters for him, and getting some results. It may very
well turn out that upping the number of the threads for the executor filter
from 16 to some higher value will solve the problem.
We're having a lot of good news as well with our testing. We had 12k clients
connected to two Mina systems, and then we have a front end machine that
distributes web service requests through our cluster to the appropriate client.
In this case we were able to service 600 web service requests a second, which
consist of routing through the cluster to the particular Mina system, to the
client, and the the reply flowing back up to the front end machine with the
reply being sent as the body of the http response. Pretty fun stuff. :-)
Thanks,
Chris
Your welcome -- this problem is actually fairly interesting because of
the high concurrency involved.
Since you believe its probably related to the database (you can find out
definitively with a profiler as Emmanuel stated), then you probably have
a couple of choices at your disposal:
1) Improve the specifications of the database machine.
2) Evict the database operations to a separate thread pool so that it
doesn't consume one of the Mina threads. The classes in
java.util.concurrent probably provides the granularity you need to check
the state and retrieve the task results, etc.
As for the out of band task, my thinking was that at some point you
could get the actual NIO handle from IOSession. So you would
effectively be bypassing Mina at that point... as to how you would do it
is not really clear and maybe not a good thing to do.
You can schedule as many tasks as you want. If it does block or hasn't
run yet, you would see from the status of the Future object that the
task is running. You could use the result object to give you timing
information to see if ran on time or was delayed.
Hope that helps,
Johnny
