Hello, I would like to share some thoughts and adventures about tcp
and nio transports to your consideration, hopefully waiting for some
feedback.
We are using a 4.1 activemq compiled from the 4.1 svn branch. For
some time we didn't run into any important problem, but lately, we
were suffering some issue regarding tcp transport.
The problem arises when the tcp buffer gets full during a
TcpBufferedOutputStream.flush(). When this happens, and probably
when all the consumers/producers are sharing the same connection, we
run into a deadlock situation, since the socket OutputStream writes
in locking mode. Meanwhile, no reader that could extract some data
from the socket to ease the situation is allowed to do its work,
since it shares the same connection locked in the write attempt. Do
you agree with this analysis and the chance that it could happen?
As a solution, nio and its non-blocking socket management, selectors
and friends, seemed the way to go. Unfortunately, the nio transport
is not available in the 4.1 branch, but it was easily backported
from the trunk. Trying to use it, some issues arised:
- Connection attempts were temporized, and the whole system worked
randomly and unresponsible. There were no deadlocks, but one symptom
was that transport.nio.SelectorSelection spent a lot of time waiting
for the socketChannel.register call to complete, in the
SelectorSelection constructor.
I don't know the exact reason, but it seems that
SelectorWorker.run() monopolizes the access to the selector doing:
while (isRunning()) {
int count = selector.select(10);
if (count == 0) {
continue;
}
I didn't have the chance to check if this thread has greater
priority than the one running the SelectorSelection constructor.
Anyway, as a workaround I changed the previous code with:
int count = selector.select(10);
if (count == 0) {
+ Thread.yield();
continue;
}
and mostly everything started to work as expected. I was able to
connect consistently to the broker, using a nio:// transport.
- The remaining problem I found is that a java test client (connect,
sends a message, and closes the connection) didn't close itself
correctly, and it did so using the tcp:// transport. I found two
possible sources for this problem:
a). NIOTransport doesn't close the selection on doStop. I think
this is needed to allow the SelectorWorker thread to finalize.
b). Even after doing that, and since the
SelectorManager.selectorExecutor is the result of calling
Executors.newCachedThreadPool, the idle threads are not destroyed
inmediatly, but after 60 seconds. Since these threads are created as
non-daemon threads, the VM waits for them to finish. As a
workaround, I changed the instantiation of
SelectorManager.selectorExecutor to:
private Executor selectorExecutor =
Executors.newCachedThreadPool(new ThreadFactory() {
public Thread newThread(Runnable r) {
Thread rc = new Thread(r);
rc.setName("NIO Transport Thread");
+ rc.setDaemon(true);
return rc;
}
});
Hence, avoiding them to be created as non-daemon threads. However, I
suppose this could be dangerous, and something could remain
inconsistent. Another solution could be not to use a
cachedThreadPool, but this could hit the performance. What would be
the best way to avoid the client shutdown delay?
Currently, changing to 5.1 or 5.2 is not an option for us, since we
run into problems in our previous attempts to switch. We need to
remain (at least while we don't have time enough to run a complete
validation of 5.1 or the upcoming 5.2) with 4.1 and the needed
patches to make it work properly.
Also, if you want 4.1 to feature NIO support, I could open a JIRA
issue attaching the patch. Anyway, any idea, comment or proposal
about the problems we run into and the exposed solutions will be
very welcome.
Best regards.
Manuel.
