Hi all,
I encountered an interesting problem today, in which MINA didn't accept any
incoming connections any longer, because we had too many open files on the
server machine.
The application in question uses MINA version 2.0.0-M1 with 3 different
IoAcceptor services sharing the same Executor and IoProcessor thread pool.
The first two services are designed to have typically 5-10 permanent
IoSessions connected per service. These two services don't have an
ExecutionFilter in their filter chain and are driven by the IoProcessor
theads. The third service is a HTTP Request/Response service based on
AsyncWeb, using an ExecutorFilter with a dedicated HTTP thread pool.
After running for several weeks without any problems, the app stooped
responding to new connections and the log file was spammed with the
Exception Monitor log entries:
WARN [NioSocketAcceptor-3] org.apache.mina.common.DefaultExceptionMonitor:
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSocketAcceptor.java:153)
at
org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSocketAcceptor.java:47)
at
org.apache.mina.common.AbstractPollingIoAcceptor$Worker.processHandles(AbstractPollingIoAcceptor.java:305)
at
org.apache.mina.common.AbstractPollingIoAcceptor$Worker.run(AbstractPollingIoAcceptor.java:243)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
After further investigation using "lsof -p", I really found approx. 1000
sockets in CLOSE_WAIT state that didn't get released. The sockets were
mainly bound to the HTTP service. Note that the already open sessions on the
ohter two services worked without any problems (messages were processed
normally). Also interesting: The app has a Connection Limit by IP address
enforced by an IoFilter. The filter prints the managed session count for the
IoService in its DEBUG log messages when sessionClosed() is invoked. The
last DEBUG entry of this kind close before the max file limit was reached
said that the managed sessions count of the HTTP service was 0. I couldn't
detect a missing coseSession() event in the Logs either.
What circumstances could lead to the effect that MINA doesn't close/release
sockets that have been colsed remotely? The processors seemed to work (the
other sessions were still active), the acceptors spammed the exceptions. Any
ideas?
Some enviromental information:
Linux 2.6.16.21-0.25-smp #1 SMP Tue Sep 19 07:26:15 UTC 2006 x86_64 x86_64
x86_64 GNU/Linux
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_11-b03, mixed mode)
Thanks for your help.
Cheers,
Tom.
