I've deployed a Mina-based SSL Gateway that uses a remote client IoAcceptor
with the SSLFilter, called the source channel, linked to an IoConnector that
does not include the SSLFilter, called the destination channel, to TCP
communicate to a back-end Application Server. The IoSession of the source
and destination channels are held in a PipedSession where the source can
push packets to the destination and vice-versa. We're coming across a
synchronization issue, where the Gateway will run for over an hour and than
will lock up. We incorporated the ExecutorFilter for both the source and
destination to reduce the likelihood of the lock up, but it eventually will
lock up the threads in the source and destination thread pools.
We've taken a thread dump of the Gateway in the bad state and the following
is a typical scenario, where the client might be pipelining. In pipelining,
the client could send packets without waiting for the response of each
packet. What we see is the source decrypting the request packet and the
destination attempting to get a lock on the SSLHandler to encrypt the
response back to the client. The issue is that the source is waiting on a
lock in the SSLEngineImpl.unwrap of the Sun JDK 6.0 U2. The thread dump
does not show who has the SSLEngineImpl.unwrap lock, but I believe that
whoever has the lock can't release it due to a deadlock scenario. An
example of the thread dump entries of the scenario are included. I'm
reviewing the SSLFilter and SSLHandler, I'm not sure what code path could
cause such a scenario.
To note, we're running Mina 1.1.0 with
mina-filter-ssl-1.1.1-20070611.145558-5.jar, since using the
mina-filter-1.1.0.jar caused lockup issues with the SSL handshake. We
deployed mina-filter-ssl-1.1.1-20070611.145558-5.jar to address the SSL
handshake induced lockup.
Is anyone else have similar SSL synchronization/deadlock issues? Any ideas
or suggestions is greatly appreciated.
The thread dump does not include what thread has the lock for < 0x94f12f98
>.
"pool-2-thread-9" prio=10 tid=0x083f6c00 nid=0x2b7a runnable
[0x8f445000..0x8f4461b0]
java.lang.Thread.State: RUNNABLE
at
com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:667)
- waiting to lock <0x94f12f98> (a java.lang.Object)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.apache.mina.filter.support.SSLHandler.unwrap(SSLHandler.java:659)
at
org.apache.mina.filter.support.SSLHandler.decrypt(SSLHandler.java:465)
at
org.apache.mina.filter.support.SSLHandler.messageReceived(SSLHandler.java:334)
at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:411)
- locked <0x94f12c10> (a org.apache.mina.filter.support.SSLHandler)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:83)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:247)
at
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:307)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-4-thread-33" prio=10 tid=0x083c3800 nid=0x2bef waiting for monitor
entry [0x8e056000..0x8e0570b0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java:478)
- waiting to lock <0x94f12c10> (a
org.apache.mina.filter.support.SSLHandler)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:824)
at
com.verisign.mina.filter.SetSessionIdInLogFilter.filterWrite(SetSessionIdInLogFilter.java:102)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:824)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:227)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:824)
at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite(AbstractIoFilterChain.java:727)
at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:445)
at
org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(AbstractIoFilterChain.java:436)
at
org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSessionImpl.java:196)
at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:149)
at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:135)
at
com.verisign.cag.protocol.common.PipedSession.write(PipedSession.java:78)
at
com.verisign.cag.protocol.common.SourceSession.write(SourceSession.java:371)
at
com.verisign.cag.protocol.common.SourceSession.messageReceived(SourceSession.java:245)
at
com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(SRSEPPDestinationHandler.java:68)
at
com.verisign.cag.protocol.srsepp.SRSEPPDestinationHandler.messageReceived(SRSEPPDestinationHandler.java:48)
at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:703)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:62)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:200)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
com.verisign.mina.filter.SetSessionIdInLogFilter.messageReceived(SetSessionIdInLogFilter.java:84)
at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:54)
at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
at
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:247)
at
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:307)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
--
View this message in context:
http://www.nabble.com/Deadlock-with-Mina-based-SSL-Gateway-tf4159854s16868.html#a11835745
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
