Hi guys, some thoughts about the TCP server. feel free to comment.
TCP server MINA 3
-----------------
As we are reworking the server part of MINA, we can review the current
architecture. There are a few problems we can address.
1) One vs Many selectors
In MINA 2, we do have at least 2 selectors used :
- one for the OP_ACCEPT event
- One or many for the OP_READ/OP_WRITE
I don't think that it makes a lot of sense to force such a choice. IMO,
we could perfectly start with one single selector to handle all the
events, assuming we are fast enough to process them. Otherwise, we can
also configure the server to use more selectors, if needed.
Typically, the acceptor selector will just deal with incoming new
connections, and the created channel will be registred on an IoProcessor
selector on MINA 2. We could register this channel on the acceptor selector.
In any case, if we do create more than one selector, I suggest that the
first one would always handle OP_ACCEPT events, and that's all.
The general algorithm will look likes :
signal started
while true
nbSelect = select( delay )
if nbSelect != 0
for each selectionKey do
case isAccept // Only if the selector is the first one, otherwise
we don't need to heck this case
create session
case isRead
process read
case isWrite
process write
done
if dispose // In order to stop the server
process dispose
break
done
The key is to start all the selector workers *before* accepting any
connection, otherwise we may lose some messages. One more thing : each
selector should signal that there have started before entering in the
loop, so that the caller don't have to wait a random period of time for
the selectors to be started.
2) Events processing
Now, in order to limit the number of selectors to use, we need to limit
the time it takes to process the read/write/accept events. But even if
we have many selectors, we should try to minimize the risk that one
selector is blocked by a single session blocked somewhere while doing
some heavy prcoessing, as it will block all the other sessions.
Having more than a selector is one way to mitigate this issue : as we
have many threads (one per selector), we spread the loads on as many
threads.
Another solution would be to use an executor in charge of processing the
events, with a queue between the selector and the executor, queue that
is used to process the events as fast as possible on the selector (this
is really important for UDP, as we don't want to lose messages simply
because the OS buffer is full).
The problem is that we are just not solving the problem of a rogue
service that block a thread for a long time (if we use a limited size
executor), or we may end with so many threads that it may kill the
server. But anyway, it sounds like a better solution, as incoming events
that won't require a long processing will be favored in the long term.
3) Write processing
This is a complex issue too : we may not be able to push all the data we
want into the socket, if it becoms full (or was already full). In this
case, we will have to store the data in a queue. The following algorithm
describe this situation and a proposal to solve it
if there are some data in the writeQueue
then
// We need to enqueue the data, and write the head of the queue
enqueue data
// Now, we shoudl try to write as much of the queue as we can
while ( queue not empty)
do
poll the data from the queue
nbWritten = channel.write( remaining data ) // We may have
already written some part of the head data
if nbWritten < data.remaining
then
// the socket is already full, set its OP_WRITE
interest, and don't touch the queue
selectionKey.ops = OP_WRITE
break // We can stop, the socket is full anyway
else
pop the data from the queue // We just remove the head,
and continue with the next data
done
if queue is empry
then
selectionKeys.ops = ! OP_WRITE // We remoe this flag,
it's ueless now
else
nbWritten = channel.write( remaining data ) // We may have
already written some part of the head data
if nbWritten < data.remaining
then
// the socket is already full, set its OP_WRITE interest,
and add the data in the queue
selectionKey.ops = OP_WRITE
writeQueue.add( remaining data )
4) Read/Write order
MINA 2 was processing all the reads first, then all the writes. This is
not necessarilly a good idea. We may rather process read and write on a
per session basis. It's perfectly possible that a session has to process
some reads and some write at the same time. Waiting for all the reads to
be processed create some memory load, as we will have to wait until all
the reads are done, storing all the data to be written in the mean time,
until we are done with the last session.
5) MINA events
Atm, we are processing the following events :
- messageReceived (when we have read some data from the channel)
- messageSent (when a user message has been sentcompletely)
- exceptionCaugth (when an exception has been caught)
- sessionIdle (if the session has not received or sent anything for a
period of time)
- sessionCreated (when a new session is created)
- sessionOpened
- sessionClosed (when the session has been closed)
- filterWrite
- filterClose
The sessionOpened, filterWrite and filterClose are a bit mysterious. I
don't see why we need a special event SessionOpened when we alreayd have
the sessionCreated event. That may seems we *may* create a session, but
not accept any incoming or outgoing messages for this session, until
it's initialized. I'm not sure this is necessary.
The two other events are probably some YAGNY iplementation...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
