Here is the Jira ID. [DIRMINA-1173] Apache mina 2.2.1 threads blocking on ConnectFuture.awaitUninterruptibly() for ever - ASF JIRA <https://issues.apache.org/jira/browse/DIRMINA-1173>
Thanks in advance. Regards, ------------------------------------------ M.V.S.Kishore 91-9886412814 On Wed, 26 Apr 2023 at 13:05, Emmanuel Lécharny <[email protected]> wrote: > > > On 26/04/2023 09:06, Kishore Mokkarala wrote: > > Hi All, > > I don't know how to create a JIRA, Please share me the link if any. > > First go there, selecting MINA project: > > https://selfserve.apache.org/jira-account.html > > then I'll validate your account creation. We can then have a technical > discussion tehre, with attached thread dump. > > I > > can't share the working code with you. > > At some point, as MINA is just a framework that is used by ten of > thousands of project, there are two possibilities: > - you have been hit by a new bug > - or you are somehow misuing MINA > > TRhe general rule of thumb is to assume (1) to be very unlikely, so (2) > is probably where lies your problem. If we can't access your code to > understand what's wrong, then it will be *very* hartd for us to point > you in any solution. > > I can share the thread dump.PFA > > Thread dump. > > > Our application gets data from Sniffers. For communicating > > with sniffers we use async communication, we use apache mina here. > > This is happening only with one type of sniffer. > > Ok, then we need to know what's the difference between this sniffer and > the others. This might be relevent to analyse what's wrong. > > > It is working fine on > > other sniffers. > > ------------------------------------------ > > M.V.S. Kishore > > 91-9886412814 > > > > > > On Wed, 26 Apr 2023 at 03:42, Emmanuel Lécharny <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Kishire, > > > > the fact it's working with 2.0.21 is pure luck, IMHO. > > > > Two things: > > - we need a full thread dump in order to see if there is a deadlock > > somewhere > > - The info you are providing are super limited. Without the whole > code, > > and a way to reproduce the problem, there is no way we can understand > > what your code is doing (beside the intention you explain, ie > getting a > > list of IPs and testing them all). > > > > So, as Jonathan already said: open a JIRA (you'll have to request an > > account creation that we will validate), and provide a working > exemple > > we can run. > > > > Thanks! > > > > On 25/04/2023 12:35, Kishore Mokkarala wrote: > > > Hi, > > > I have migrated from 2.0.21 to 2.023 for solving CVE, i have seen > > thread > > > blocking issue, So used latest mina verstion 2.2.1 but still > > threads were > > > blocked here is the sample code.Thread hungs at > > awaitUninterruptibly. Once > > > this issue comes in sub sequest launches nothing will work all > > threads > > > will be blocked forever,i have to restart the process to make it > > work. For > > > single thread working fine,if i start 50-100 threads this thread > > blocking > > > issue will surface. > > > > > > *Here is the sample thread dump:* > > > "pool-118-thread-6" #508 prio=5 os_prio=0 cpu=345.84ms > > elapsed=*1929.48s* > > > tid=0x00007ec6fc001800 nid=0x4b5d4 in Object.wait() > > [0x00007ec7792d4000] > > > java.lang.Thread.State: TIMED_WAITING (on object monitor) > > > at java.lang.Object.wait([email protected]/Native > > <http://[email protected]/Native> Method) > > > - waiting on <no object reference available> > > > at > > > > > > org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218) > > > - waiting to re-lock in wait() <0x00007ed90d1b2c40> (a > > > > > > org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest) > > > at > > > > > > > org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148) > > > at > > > > > > > org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149) > > > at > > > > > > > com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622) > > > at > > > > > > > com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67) > > > at > > > > > > > com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156) > > > at > > > > > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283) > > > at > > > > > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268) > > > at > > jdk.internal.reflect.GeneratedMethodAccessor201.invoke(Unknown > > > Source) > > > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke( > > > [email protected]/DelegatingMethodAccessorImpl.java:43 > > <http://[email protected]/DelegatingMethodAccessorImpl.java:43>) > > > at java.lang.reflect.Method.invoke( > > > [email protected]/Method.java:566 > > <http://[email protected]/Method.java:566>) > > > at > > > > > > > org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281) > > > at > > > > > > > org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232) > > > at > > > > > > > org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100) > > > at > > > > > > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183) > > > at > > > > > > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273) > > > at > > > > > > > org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170) > > > at > > > > > > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158) > > > - locked <0x00007ed92a951cd8> (a > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext) > > > at > > > > > > > org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261) > > > at com.sun.proxy.$Proxy85.messageReceived(Unknown Source) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > > org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106) > > > at > org.apache.mina.core.session.IoEvent.run(IoEvent.java:89) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695) > > > at > > java.lang.Thread.run([email protected]/Thread.java:829 > > <http://[email protected]/Thread.java:829>) > > > > > > > > > "pool-116-thread-8" #458 prio=5 os_prio=0 cpu=172.16ms > > elapsed=529.47s > > > tid=0x00007ed6c800e000 nid=0x124b in Object.wait() > > [0x00007ed6d9cda000] > > > java.lang.Thread.State: TIMED_WAITING (on object monitor) > > > at java.lang.Object.wait([email protected]/Native > > <http://[email protected]/Native> Method) > > > - waiting on <no object reference available> > > > at > > > > > > org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218) > > > - waiting to re-lock in wait() <0x00007ee3a971c170> (a > > > > > > org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest) > > > at > > > > > > > org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148) > > > at > > > > > > > org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149) > > > at > > > > > > > com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622) > > > at > > > > > > > com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67) > > > at > > > > > > > com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156) > > > at > > > > > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283) > > > at > > > > > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268) > > > at > > jdk.internal.reflect.GeneratedMethodAccessor281.invoke(Unknown > > > Source) > > > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke( > > > [email protected]/DelegatingMethodAccessorImpl.java:43 > > <http://[email protected]/DelegatingMethodAccessorImpl.java:43>) > > > at java.lang.reflect.Method.invoke( > > > [email protected]/Method.java:566 > > <http://[email protected]/Method.java:566>) > > > at > > > > > > > org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281) > > > at > > > > > > > org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232) > > > at > > > > > > > org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100) > > > at > > > > > > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183) > > > at > > > > > > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273) > > > at > > > > > > > org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170) > > > at > > > > > > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158) > > > - locked <0x00007ee3aaed5d58> (a > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext) > > > at > > > > > > > org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261) > > > at com.sun.proxy.$Proxy85.messageReceived(Unknown Source) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > > org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > > com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) > > > at > > > > > > > org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) > > > at > > > > > > org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106) > > > at > org.apache.mina.core.session.IoEvent.run(IoEvent.java:89) > > > at > > > > > > > org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763) > > > > > > *This code is working fine with 2.0.21.* > > > > > > *Here is the code snippet.* > > > > > > private static final ExecutorFilter executorFilter = *new > > > ExecutorFilter(16,32);* > > > > > > StateMachine stateMachine = > > > StateMachineFactory.getInstance(IoHandlerTransition.class).create( > > > G10MinaClient.CONNECTED, new > > G10MinaClient(processor)); > > > > > > IoHandler ioHandler = new > > > StateMachineProxyBuilder().setStateContextLookup( > > > new IoSessionStateContextLookup(new > > StateContextFactory() { > > > @Override > > > public StateContext create() { > > > final G10StateContext stateContext = new > > > G10StateContext(); > > > stateContext.setStartedTime(new Date()); > > > return stateContext; > > > } > > > })).create(IoHandler.class, stateMachine); > > > > > > NioSocketConnector connector = new NioSocketConnector(); > > > connector.getFilterChain().addLast("LoggingFilter", > > > G10CaptureService.loggingFilter); > > > connector.getFilterChain().addLast("codecFilter", > > > G10CaptureService.probeCodecFilter); > > > connector.getFilterChain().addLast("executorFilter", > > > G10CaptureService.executorFilter); > > > connector.getFilterChain().addLast("gpbMessageFilter", > > > G10CaptureService.gpbMessageFilter); > > > connector.getFilterChain().addLast("keepAliveFilter", > > > G10CaptureService.keepAliveFilter); > > > connector.setHandler(ioHandler); > > > connectionLock.lock(); // connectionLock is private variable. > > > try{ > > > ConnectFuture primaryConnectFuture = > > connector.connect(primaryAddress, > > > initializer); > > > primaryConnectFuture.awaitUninterruptibly(); // no timeout > specified. > > > > > > if (!primaryConnectFuture.isConnected()) > > > { > > > > > > if (handleIOException(searchExpression, > > > captureHandler)) { > > > return; > > > } > > > LOG.info("{} Apache mina connection setup > > time out > > > happend.", > > > handleConnectionFailed(primaryAddress, > > captureHandler, > > > "Primary IP connection timeout"); > > > return; > > > } > > > } > > > finally > > > { > > > connectionLock.unLock(); > > > } > > > > > > ------------------------------------------ > > > M.V.S.Kishore > > > 91-9886412814 > > > > > > > -- > > *Emmanuel Lécharny - CTO* 205 Promenade des Anglais – 06200 NICE > > T. +33 (0)4 89 97 36 50 > > P. +33 (0)6 08 33 32 61 > > [email protected] <mailto:[email protected]> > > https://www.busit.com/ <https://www.busit.com/> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > <mailto:[email protected]> > > For additional commands, e-mail: [email protected] > > <mailto:[email protected]> > > > > -- > *Emmanuel Lécharny - CTO* 205 Promenade des Anglais – 06200 NICE > T. +33 (0)4 89 97 36 50 > P. +33 (0)6 08 33 32 61 > [email protected] https://www.busit.com/ >
