Hi,
I have migrated from 2.0.21 to 2.023 for solving CVE, i have seen thread
blocking issue, So used latest mina verstion 2.2.1 but still threads were
blocked here is the sample code.Thread hungs at awaitUninterruptibly. Once
this issue comes in sub sequest launches nothing will work all threads
will be blocked forever,i have to restart the process to make it work. For
single thread working fine,if i start 50-100 threads this thread blocking
issue will surface.
*Here is the sample thread dump:*
"pool-118-thread-6" #508 prio=5 os_prio=0 cpu=345.84ms elapsed=*1929.48s*
tid=0x00007ec6fc001800 nid=0x4b5d4 in Object.wait() [0x00007ec7792d4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.14.1/Native Method)
- waiting on <no object reference available>
at
org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)
- waiting to re-lock in wait() <0x00007ed90d1b2c40> (a
org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest)
at
org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148)
at
org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149)
at
com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622)
at
com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67)
at
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268)
at jdk.internal.reflect.GeneratedMethodAccessor201.invoke(Unknown
Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
java.base@11.0.14.1/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(
java.base@11.0.14.1/Method.java:566)
at
org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
at
org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
at
org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273)
at
org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
- locked <0x00007ed92a951cd8> (a
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
at
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
at com.sun.proxy.$Proxy85.messageReceived(Unknown Source)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
at
org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
at
org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
at
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
at
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
at
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
at java.lang.Thread.run(java.base@11.0.14.1/Thread.java:829)
"pool-116-thread-8" #458 prio=5 os_prio=0 cpu=172.16ms elapsed=529.47s
tid=0x00007ed6c800e000 nid=0x124b in Object.wait() [0x00007ed6d9cda000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.14.1/Native Method)
- waiting on <no object reference available>
at
org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:218)
- waiting to re-lock in wait() <0x00007ee3a971c170> (a
org.apache.mina.core.polling.AbstractPollingIoConnector$ConnectionRequest)
at
org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:148)
at
org.apache.mina.core.future.DefaultConnectFuture.awaitUninterruptibly(DefaultConnectFuture.java:149)
at
com.netscout.nsaapp.geo.g10Plugin.g10.service.G10CaptureService.startRecordCapture(G10CaptureService.java:622)
at
com.netscout.nsaapp.geo.g10Plugin.geoblade.service.GeoBladeCaptureService.startRecordCapture(GeoBladeCaptureService.java:67)
at
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.processGatewaySrQueryResponseSuccess(G10PluginCaptureProcessor.java:2156)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.doHandleGatewaySrQueryResponse(G10MinaClient.java:283)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.handleGatewaySrQueryResponse(G10MinaClient.java:268)
at jdk.internal.reflect.GeneratedMethodAccessor281.invoke(Unknown
Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
java.base@11.0.14.1/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(
java.base@11.0.14.1/Method.java:566)
at
org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
at
org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
at
org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:273)
at
org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
- locked <0x00007ee3aaed5d58> (a
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
at
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
at com.sun.proxy.$Proxy85.messageReceived(Unknown Source)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
at
org.apache.mina.filter.keepalive.KeepAliveFilter.messageReceived(KeepAliveFilter.java:414)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.messageReceived(G10GPBMessageIoFilter.java:100)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
at
org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
at
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
*This code is working fine with 2.0.21.*
*Here is the code snippet.*
private static final ExecutorFilter executorFilter = *new
ExecutorFilter(16,32);*
StateMachine stateMachine =
StateMachineFactory.getInstance(IoHandlerTransition.class).create(
G10MinaClient.CONNECTED, new G10MinaClient(processor));
IoHandler ioHandler = new
StateMachineProxyBuilder().setStateContextLookup(
new IoSessionStateContextLookup(new StateContextFactory() {
@Override
public StateContext create() {
final G10StateContext stateContext = new
G10StateContext();
stateContext.setStartedTime(new Date());
return stateContext;
}
})).create(IoHandler.class, stateMachine);
NioSocketConnector connector = new NioSocketConnector();
connector.getFilterChain().addLast("LoggingFilter",
G10CaptureService.loggingFilter);
connector.getFilterChain().addLast("codecFilter",
G10CaptureService.probeCodecFilter);
connector.getFilterChain().addLast("executorFilter",
G10CaptureService.executorFilter);
connector.getFilterChain().addLast("gpbMessageFilter",
G10CaptureService.gpbMessageFilter);
connector.getFilterChain().addLast("keepAliveFilter",
G10CaptureService.keepAliveFilter);
connector.setHandler(ioHandler);
connectionLock.lock(); // connectionLock is private variable.
try{
ConnectFuture primaryConnectFuture = connector.connect(primaryAddress,
initializer);
primaryConnectFuture.awaitUninterruptibly(); // no timeout specified.
if (!primaryConnectFuture.isConnected())
{
if (handleIOException(searchExpression,
captureHandler)) {
return;
}
LOG.info("{} Apache mina connection setup time out
happend.",
handleConnectionFailed(primaryAddress, captureHandler,
"Primary IP connection timeout");
return;
}
}
finally
{
connectionLock.unLock();
}
------------------------------------------
M.V.S.Kishore
91-9886412814
