Here is an scenario, I have one gate way IP which i connect initially ,this
gate way will return list of ips.
now i will close the previous IOsession and will create a nio connection
with the first ip in the list.if it did n't work then again i will try to
connect to next ip etc..
All these are in the critical section. i.e i will acquire a lock and then
after successful connection i will release the lock.
But problem i have noticed is with the awaitUninterruptibly().
I have a state machine too.So connection set up is in one thread and
response processing is in another thread.
*Thread1:*
private static final ExecutorFilter executorFilter = *new
ExecutorFilter(16,32);
StateMachine stateMachine =
StateMachineFactory.getInstance(IoHandlerTransition.class).create(
G10MinaClient.CONNECTED, new G10MinaClient(processor));
IoHandler ioHandler = new
StateMachineProxyBuilder().setStateContextLookup(
new IoSessionStateContextLookup(new StateContextFactory() {
@Override
public StateContext create() {
final G10StateContext stateContext = new
G10StateContext();
stateContext.setStartedTime(new Date());
return stateContext;
}
})).create(IoHandler.class, stateMachine);
NioSocketConnector connector = new NioSocketConnector();
connector.getFilterChain().addLast("LoggingFilter",
G10CaptureService.loggingFilter);
connector.getFilterChain().addLast("codecFilter",
G10CaptureService.probeCodecFilter);
connector.getFilterChain().addLast("executorFilter",
G10CaptureService.executorFilter);
connector.getFilterChain().addLast("gpbMessageFilter",
G10CaptureService.gpbMessageFilter);
connector.getFilterChain().addLast("keepAliveFilter",
G10CaptureService.keepAliveFilter);
connector.setHandler(ioHandler);
connectionLock.lock(); // connectionLock is private variable.
try{
ConnectFuture primaryConnectFuture = connector.connect(primaryAddress,
initializer);
primaryConnectFuture.awaitUninterruptibly(); // no timeout specified.
if (!primaryConnectFuture.isConnected())
{
if (handleIOException(searchExpression,
captureHandler)) {
return;
}
LOG.info("{} Apache mina connection setup time out
happend.",
handleConnectionFailed(primaryAddress, captureHandler,
"Primary IP connection timeout");
return;
}
}
finally
{
connectionLock.unLock();
}
*Thread2:*
public void processGatewaySrQueryResponseSuccess(G10StateContext context,
IoSession session, GatewaySrQueryResponse response) {
if(response.getIpPortCount() > 0) {
try {
List<IpAddrPort> msgIpPorts = response.getIpPortList();
List<InetSocketAddress> probeIpPorts = new
ArrayList<InetSocketAddress>();
for (IpAddrPort ipp : msgIpPorts) {
InetSocketAddress inetSockAddr = new
InetSocketAddress(InetAddress.getByAddress(ipp.getIp().toByteArray()),
ipp.getPort());
probeIpPorts.add(inetSockAddr);
}
g10CaptureService.setProbeIps(probeIpPorts);
} catch (Exception ex) {
LOG.error("{} Exception occured while creating probe
IP/Port.Exception:{}", getLogStr(session), ex);
}
context.addGeoBladeGwSession(session);
g10CaptureService.closeConnectionForSession(session);
g10CaptureService.setIsGeoBladeIdlSession(true);
g10CaptureService.startRecordCapture(searchExpression,
captureHandler, captureId);
}
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Tue, 25 Apr 2023 at 16:36, Jonathan Valliere <john...@apache.org> wrote:
> Please make a Jira. I also need an example to run which reproduces the
> error.
>
> On Tue, Apr 25, 2023 at 6:35 AM Kishore Mokkarala <kishore....@gmail.com>
> wrote:
>
> > Hi,
> > I have migrated from 2.0.21 to 2.023 for solving CVE, i have seen thread
> > blocking issue, So used latest mina verstion 2.2.1 but still threads were
> > blocked here is the sample code.Thread hungs at awaitUninterruptibly.
> Once
> > this issue comes in sub sequest launches nothing will work all threads
> > will be blocked forever,i have to restart the process to make it work.
> For
> > single thread working fine,if i start 50-100 threads this thread blocking
> > issue will surface.
> >
> > *Here is the sample thread dump:*
> > "pool-118-thread-6" #508 prio=5 os_prio=0 cpu=345.84ms elapsed=*1929.48s*
> > tid=0x00007ec6fc001800 nid=0x4b5d4 in Object.wait() [0x00007ec7792d4000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(java.base@11.0.14.1/Native Method)
> > - waiting on <no object reference available>
> > at
> >
> >
> org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)
> > - waiting to re-lock in wait() <0x00007ed90d1b2c40> (a
> >
> org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest)
> > at
> >
> >
> org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148)
> > at
> >
> >
> org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149)
> > at
> >
> >
> com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622)
> > at
> >
> >
> com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67)
> > at
> >
> >
> com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156)
> > at
> >
> >
> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283)
> > at
> >
> >
> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268)
> > at jdk.internal.reflect.GeneratedMethodAccessor201.invoke(Unknown
> > Source)
> > at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
> > java.base@11.0.14.1/DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(
> > java.base@11.0.14.1/Method.java:566)
> > at
> >
> >
> org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
> > at
> >
> >
> org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
> > at
> >
> >
> org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
> > at
> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
> > at
> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273)
> > at
> >
> >
> org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
> > at
> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
> > - locked <0x00007ed92a951cd8> (a
> > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> > at
> >
> >
> org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> > at com.sun.proxy.$Proxy85.messageReceived(Unknown Source)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> > at
> >
> >
> org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> > at
> >
> >
> com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> > at
> >
> org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
> > at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
> > at
> >
> >
> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
> > at
> >
> >
> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
> > at
> >
> >
> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
> > at java.lang.Thread.run(java.base@11.0.14.1/Thread.java:829)
> >
> >
> > "pool-116-thread-8" #458 prio=5 os_prio=0 cpu=172.16ms elapsed=529.47s
> > tid=0x00007ed6c800e000 nid=0x124b in Object.wait() [0x00007ed6d9cda000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(java.base@11.0.14.1/Native Method)
> > - waiting on <no object reference available>
> > at
> >
> >
> org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)
> > - waiting to re-lock in wait() <0x00007ee3a971c170> (a
> >
> org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest)
> > at
> >
> >
> org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148)
> > at
> >
> >
> org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149)
> > at
> >
> >
> com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622)
> > at
> >
> >
> com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67)
> > at
> >
> >
> com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156)
> > at
> >
> >
> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283)
> > at
> >
> >
> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268)
> > at jdk.internal.reflect.GeneratedMethodAccessor281.invoke(Unknown
> > Source)
> > at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
> > java.base@11.0.14.1/DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(
> > java.base@11.0.14.1/Method.java:566)
> > at
> >
> >
> org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
> > at
> >
> >
> org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
> > at
> >
> >
> org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
> > at
> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
> > at
> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273)
> > at
> >
> >
> org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
> > at
> > org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
> > - locked <0x00007ee3aaed5d58> (a
> > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> > at
> >
> >
> org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> > at com.sun.proxy.$Proxy85.messageReceived(Unknown Source)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> > at
> >
> >
> org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> > at
> >
> >
> com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> > at
> >
> >
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> > at
> >
> org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
> > at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
> > at
> >
> >
> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
> >
> > *This code is working fine with 2.0.21.*
> >
> > *Here is the code snippet.*
> >
> > private static final ExecutorFilter executorFilter = *new
> > ExecutorFilter(16,32);*
> >
> > StateMachine stateMachine =
> > StateMachineFactory.getInstance(IoHandlerTransition.class).create(
> > G10MinaClient.CONNECTED, new G10MinaClient(processor));
> >
> > IoHandler ioHandler = new
> > StateMachineProxyBuilder().setStateContextLookup(
> > new IoSessionStateContextLookup(new
> StateContextFactory() {
> > @Override
> > public StateContext create() {
> > final G10StateContext stateContext = new
> > G10StateContext();
> > stateContext.setStartedTime(new Date());
> > return stateContext;
> > }
> > })).create(IoHandler.class, stateMachine);
> >
> > NioSocketConnector connector = new NioSocketConnector();
> > connector.getFilterChain().addLast("LoggingFilter",
> > G10CaptureService.loggingFilter);
> > connector.getFilterChain().addLast("codecFilter",
> > G10CaptureService.probeCodecFilter);
> > connector.getFilterChain().addLast("executorFilter",
> > G10CaptureService.executorFilter);
> > connector.getFilterChain().addLast("gpbMessageFilter",
> > G10CaptureService.gpbMessageFilter);
> > connector.getFilterChain().addLast("keepAliveFilter",
> > G10CaptureService.keepAliveFilter);
> > connector.setHandler(ioHandler);
> > connectionLock.lock(); // connectionLock is private variable.
> > try{
> > ConnectFuture primaryConnectFuture = connector.connect(primaryAddress,
> > initializer);
> > primaryConnectFuture.awaitUninterruptibly(); // no timeout specified.
> >
> > if (!primaryConnectFuture.isConnected())
> > {
> >
> > if (handleIOException(searchExpression,
> > captureHandler)) {
> > return;
> > }
> > LOG.info("{} Apache mina connection setup time out
> > happend.",
> > handleConnectionFailed(primaryAddress,
> captureHandler,
> > "Primary IP connection timeout");
> > return;
> > }
> > }
> > finally
> > {
> > connectionLock.unLock();
> > }
> >
> > ------------------------------------------
> > M.V.S.Kishore
> > 91-9886412814
> >
>
