*Hi Emmanuel,*
*
*
*Deadlock is happening in the 3rd party jars,not within the code,which
is out of the scope of the business logic. When large data is being
transferred between two entities with parallel TCP ip connections
**over SSL connection.** This is happening *
*Could you please provide a work around this problem,we are kind of
stuck now in the production.*
*Currently we are using mina version 2.2.1.*
*
*
*StateMachine.java:138*
*public* *void* handle(Event
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/event/Event.html#Event> event) {
136
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L136> StateContext <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/context/StateContext.html#StateContext> context = event.getContext();
137
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L137> 138 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L138> *synchronized (context) {*
139
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L139> LinkedList<Event> eventQueue = eventQueueThreadLocal.get();
140
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L140> eventQueue.addLast(event);
141
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L141> 142 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L142> *if* (processingThreadLocal.get()) {
143
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L143> //*/
144
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L144> /* This thread is already processing an event. Queue this /
145
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L145> /* event./
146
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L146> /*//
147
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L147> *if* (LOGGER.isDebugEnabled()) {
148
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L148> LOGGER.debug("State machine called recursively. Queuing event k{} for later processing.", event);
149
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L149> }
150
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L150> }*else* {
151
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L151> processingThreadLocal.set(*true*);
152
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L152> 153 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L153> *try* {
154
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L154> *if* (context.getCurrentState() ==*null*) {
155
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L155> context.setCurrentState(startState);
156
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L156> }
157
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L157> 158 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L158> processEvents(eventQueue);
159
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L159> }*finally* {
160
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L160> processingThreadLocal.set(false);
161
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L161> }
162
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L162> }
163
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L163> }
164
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L164> }
*
*
* at
org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)*
*
*
*SSLHandlerG0.java:*
synchronized public void write(NextFilter next, WriteRequest request)
throws SSLException, WriteRejectedException {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("{} write() - source {}", toString(), request);
}*
*
........................
..........................
}
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Fri, 22 Dec 2023 at 20:10, Kishore Mokkarala <kishore....@gmail.com
<mailto:kishore....@gmail.com>> wrote:
1.
Is the below issue fixed ?
1. DIRMINA-604
<https://issues.apache.org/jira/browse/DIRMINA-604>
Deadlock occurs when implementing two mina StateMachine
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Fri, 22 Dec 2023 at 13:04, Emmanuel Lécharny <elecha...@gmail.com
<mailto:elecha...@gmail.com>> wrote:
Hi,
you have a deadlock on 0x00007f5c9f001408, which is in
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext.
On 22/12/2023 08:10, Kishore Mokkarala wrote:
> Hi Emmanuel,
>
> We are facing deadlock in production with mina 2.2.1
version,earlier it
> was on 2.0.21, After upgrade only we are seeing this
issue,kindly please
> provide an update any similar kind issue is fixed in 2.2.3 or
not ?
>
>
> Thread pool-123-thread-3 is in deadlock with thread
NioProcessor-37
>
>
> pool-123-thread-3
>
> PRIORITY : 5
>
> THREAD ID : 0X00007F5978002870
>
> NATIVE ID : 0X7C15
>
> NATIVE ID (DECIMAL) : 31765
>
> STATE : BLOCKED
>
>
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)
> - waiting to lock <0x00007f5c9f001168> (a
> org.apache.mina.filter.ssl.SSLHandlerG0)
> at
org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:380)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:332)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.filter.executor.ExecutorFilter.filterWrite(ExecutorFilter.java:595)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.filterWrite(G10GPBMessageIoFilter.java:63)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746)
> at
>
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575)
> at
>
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520)
> at
>
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.verifyAndSendStartMsgs(G10PluginCaptureProcessor.java:2627)
> at
>
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.sessionConnected(G10PluginCaptureProcessor.java:2552)
> at
>
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.connect(G10MinaClient.java:220)
> at
jdk.internal.reflect.GeneratedMethodAccessor144.invoke(Unknown
Source)
> at
>
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
> at
java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
> at
>
org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
> at
>
org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
> at
>
org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
> at
>
org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
> - locked <0x00007f5c9f001408> (a
> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> at
>
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> at jdk.proxy4.$Proxy87.sessionOpened(jdk.proxy4/Unknown
Source)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionOpened(DefaultIoFilterChain.java:940)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:127)
> at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
> at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)
>
>
> NioProcessor-37
>
> PRIORITY : 5
>
> THREAD ID : 0X00007F5A040B5880
>
> NATIVE ID : 0X6D19
>
> NATIVE ID (DECIMAL) : 27929
>
> STATE : BLOCKED
>
>
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:138)
> - waiting to lock <0x00007f5c9f001408> (a
> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> at
>
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> at jdk.proxy4.$Proxy87.event(jdk.proxy4/Unknown Source)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.event(DefaultIoFilterChain.java:1039)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.finish_handshake(SSLHandlerG0.java:589)
> - locked <0x00007f5c9f001168> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:271)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
> at
org.apache.mina.filter.ssl.SSLHandlerG0.receive(SSLHandlerG0.java:162)
> - locked <0x00007f5c9f001168> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
> at
org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:342)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1224)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1213)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> at
>
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
> at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)
>
> Regards,
> ------------------------------------------
> M.V.S.Kishore
> Lead Dev Engineer
> NetScout S/w Pvt. Ltd.
> 91-9886412814
-- *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
elecha...@apache.org <mailto:elecha...@apache.org>