Here is an scenario, I have one gate way IP which i connect initially ,this
> gate way will return list of ips.
> now i will close the previous IOsession and will create a nio connection
> with the first ip in the list.if it did n't work then again i will try to
> connect to next ip etc..
> All these are in the critical section. i.e i will acquire a lock and then
> after successful connection i will release the lock.
> But problem i have noticed is with the awaitUninterruptibly().
>
> I have a state machine too.So connection set up is in one thread and
> response processing is in another thread.
>
> *Thread1:*
>
> private static final ExecutorFilter executorFilter = *new
> ExecutorFilter(16,32);
>
>
>
> StateMachine stateMachine =
> StateMachineFactory.getInstance(IoHandlerTransition.class).create(
> G10MinaClient.CONNECTED, new G10MinaClient(processor));
>
> IoHandler ioHandler = new
> StateMachineProxyBuilder().setStateContextLookup(
> new IoSessionStateContextLookup(new StateContextFactory()
> {
> @Override
> public StateContext create() {
> final G10StateContext stateContext = new
> G10StateContext();
> stateContext.setStartedTime(new Date());
> return stateContext;
> }
> })).create(IoHandler.class, stateMachine);
>
> NioSocketConnector connector = new NioSocketConnector();
> connector.getFilterChain().addLast("LoggingFilter",
> G10CaptureService.loggingFilter);
> connector.getFilterChain().addLast("codecFilter",
> G10CaptureService.probeCodecFilter);
> connector.getFilterChain().addLast("executorFilter",
> G10CaptureService.executorFilter);
> connector.getFilterChain().addLast("gpbMessageFilter",
> G10CaptureService.gpbMessageFilter);
> connector.getFilterChain().addLast("keepAliveFilter",
> G10CaptureService.keepAliveFilter);
> connector.setHandler(ioHandler);
> connectionLock.lock(); // connectionLock is private variable.
> try{
> ConnectFuture primaryConnectFuture = connector.connect(primaryAddress,
> initializer);
> primaryConnectFuture.awaitUninterruptibly(); // no timeout specified.
>
> if (!primaryConnectFuture.isConnected())
> {
>
> if (handleIOException(searchExpression,
> captureHandler)) {
> return;
> }
> LOG.info("{} Apache mina connection setup time out
> happend.",
> handleConnectionFailed(primaryAddress, captureHandler,
> "Primary IP connection timeout");
> return;
> }
> }
> finally
> {
> connectionLock.unLock();
> }
>
> *Thread2:*
>
> public void processGatewaySrQueryResponseSuccess(G10StateContext context,
> IoSession session, GatewaySrQueryResponse response) {
>
//get the *g10CaptureService* from the Thread1 IOsession
>
> if(response.getIpPortCount() > 0) {
> try {
> List<IpAddrPort> msgIpPorts = response.getIpPortList();
> List<InetSocketAddress> probeIpPorts = new
> ArrayList<InetSocketAddress>();
> for (IpAddrPort ipp : msgIpPorts) {
> InetSocketAddress inetSockAddr = new
> InetSocketAddress(InetAddress.getByAddress(ipp.getIp().toByteArray()),
> ipp.getPort());
> probeIpPorts.add(inetSockAddr);
> }
> g10CaptureService.setProbeIps(probeIpPorts); //used for iterate
> through multiple ips once after another
> } catch (Exception ex) {
> LOG.error("{} Exception occured while creating probe
> IP/Port.Exception:{}", getLogStr(session), ex);
> }
>
> context.addGeoBladeGwSession(session);
> g10CaptureService.closeConnectionForSession(session); //close
> connection
> g10CaptureService.startRecordCapture(searchExpression,
> captureHandler, captureId); //connect to the probe again using same thread
> 1 instance.
>
> }
>
> ------------------------------------------
> M.V.S.Kishore
>
> 91-9886412814
>
>
> On Tue, 25 Apr 2023 at 16:36, Jonathan Valliere <[email protected]>
> wrote:
>
>> Please make a Jira. I also need an example to run which reproduces the
>> error.
>>
>> On Tue, Apr 25, 2023 at 6:35 AM Kishore Mokkarala <[email protected]>
>> wrote:
>>
>> > Hi,
>> > I have migrated from 2.0.21 to 2.023 for solving CVE, i have seen thread
>> > blocking issue, So used latest mina verstion 2.2.1 but still threads
>> were
>> > blocked here is the sample code.Thread hungs at awaitUninterruptibly.
>> Once
>> > this issue comes in sub sequest launches nothing will work all threads
>> > will be blocked forever,i have to restart the process to make it work.
>> For
>> > single thread working fine,if i start 50-100 threads this thread
>> blocking
>> > issue will surface.
>> >
>> > *Here is the sample thread dump:*
>> > "pool-118-thread-6" #508 prio=5 os_prio=0 cpu=345.84ms
>> elapsed=*1929.48s*
>> > tid=0x00007ec6fc001800 nid=0x4b5d4 in Object.wait()
>> [0x00007ec7792d4000]
>> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait([email protected]/Native Method)
>> > - waiting on <no object reference available>
>> > at
>> >
>> >
>> org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)
>> > - waiting to re-lock in wait() <0x00007ed90d1b2c40> (a
>> >
>> org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest)
>> > at
>> >
>> >
>> org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148)
>> > at
>> >
>> >
>> org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268)
>> > at
>> jdk.internal.reflect.GeneratedMethodAccessor201.invoke(Unknown
>> > Source)
>> > at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
>> > [email protected]/DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(
>> > [email protected]/Method.java:566)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
>> > at
>> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
>> > at
>> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
>> > at
>> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
>> > - locked <0x00007ed92a951cd8> (a
>> > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
>> > at com.sun.proxy.$Proxy85.messageReceived(Unknown Source)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
>> > at
>> >
>> >
>> org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
>> > at
>> >
>> org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
>> > at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
>> > at
>> >
>> >
>> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
>> > at
>> >
>> >
>> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
>> > at
>> >
>> >
>> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
>> > at java.lang.Thread.run([email protected]/Thread.java:829)
>> >
>> >
>> > "pool-116-thread-8" #458 prio=5 os_prio=0 cpu=172.16ms elapsed=529.47s
>> > tid=0x00007ed6c800e000 nid=0x124b in Object.wait() [0x00007ed6d9cda000]
>> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> > at java.lang.Object.wait([email protected]/Native Method)
>> > - waiting on <no object reference available>
>> > at
>> >
>> >
>> org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)
>> > - waiting to re-lock in wait() <0x00007ee3a971c170> (a
>> >
>> org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest)
>> > at
>> >
>> >
>> org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148)
>> > at
>> >
>> >
>> org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268)
>> > at
>> jdk.internal.reflect.GeneratedMethodAccessor281.invoke(Unknown
>> > Source)
>> > at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
>> > [email protected]/DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(
>> > [email protected]/Method.java:566)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
>> > at
>> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
>> > at
>> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
>> > at
>> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
>> > - locked <0x00007ee3aaed5d58> (a
>> > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
>> > at
>> >
>> >
>> org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
>> > at com.sun.proxy.$Proxy85.messageReceived(Unknown Source)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
>> > at
>> >
>> >
>> org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
>> > at
>> >
>> >
>> com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
>> > at
>> >
>> >
>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
>> > at
>> >
>> org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
>> > at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
>> > at
>> >
>> >
>> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
>> >
>> > *This code is working fine with 2.0.21.*
>> >
>> > *Here is the code snippet.*
>> >
>> > private static final ExecutorFilter executorFilter = *new
>> > ExecutorFilter(16,32);*
>> >
>> > StateMachine stateMachine =
>> > StateMachineFactory.getInstance(IoHandlerTransition.class).create(
>> > G10MinaClient.CONNECTED, new G10MinaClient(processor));
>> >
>> > IoHandler ioHandler = new
>> > StateMachineProxyBuilder().setStateContextLookup(
>> > new IoSessionStateContextLookup(new
>> StateContextFactory() {
>> > @Override
>> > public StateContext create() {
>> > final G10StateContext stateContext = new
>> > G10StateContext();
>> > stateContext.setStartedTime(new Date());
>> > return stateContext;
>> > }
>> > })).create(IoHandler.class, stateMachine);
>> >
>> > NioSocketConnector connector = new NioSocketConnector();
>> > connector.getFilterChain().addLast("LoggingFilter",
>> > G10CaptureService.loggingFilter);
>> > connector.getFilterChain().addLast("codecFilter",
>> > G10CaptureService.probeCodecFilter);
>> > connector.getFilterChain().addLast("executorFilter",
>> > G10CaptureService.executorFilter);
>> > connector.getFilterChain().addLast("gpbMessageFilter",
>> > G10CaptureService.gpbMessageFilter);
>> > connector.getFilterChain().addLast("keepAliveFilter",
>> > G10CaptureService.keepAliveFilter);
>> > connector.setHandler(ioHandler);
>> > connectionLock.lock(); // connectionLock is private variable.
>> > try{
>> > ConnectFuture primaryConnectFuture = connector.connect(primaryAddress,
>> > initializer);
>> > primaryConnectFuture.awaitUninterruptibly(); // no timeout specified.
>> >
>> > if (!primaryConnectFuture.isConnected())
>> > {
>> >
>> > if (handleIOException(searchExpression,
>> > captureHandler)) {
>> > return;
>> > }
>> > LOG.info("{} Apache mina connection setup time out
>> > happend.",
>> > handleConnectionFailed(primaryAddress,
>> captureHandler,
>> > "Primary IP connection timeout");
>> > return;
>> > }
>> > }
>> > finally
>> > {
>> > connectionLock.unLock();
>> > }
>> >
>> > ------------------------------------------
>> > M.V.S.Kishore
>> > 91-9886412814
>> >
>>
>
