wy96f commented on issue #2558: ARTEMIS-2257 Synchronize SharedEventLoopGroup shutdownGracefully URL: https://github.com/apache/activemq-artemis/pull/2558#issuecomment-466977919 > @wy96f did you see my comment on the JIRA? > > Synchronized SharedEventLoopGroup is not the bug, it's the fix you applied to fix the bug. > > I don't know in what condition the issue happened. > > Can you add more detail on how it happened? a test would be great if possible... if you think a test is almost impossible due to an issue lets discuss and see if there's a way around. > > Fixes without tests should be an exception... not the rule. Sorry for my ambiguousness. We use jms client. We will close ActiveMqConnection if in idle for 5 minutes and recreate connection if some requests come in. These days we found exceptions occasionally as follows: 2019-02-24 08:23:44.294 [] [ThreadPoolAsynchronousProcessor-thread-2] WARN io.netty.channel.AbstractChannel -Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xab32cb4c] java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:479) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) at org.apache.activemq.artemis.core.remoting.impl.netty.DelegatingEventLoopGroup.register(DelegatingEventLoopGroup.java:46) at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:332) at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163) at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145) at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:616) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1004) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1040) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1232) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:862) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:775) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:221) at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:643) at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:748) at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:248) Before the stack trace some logs were printed showing idle connection is closed. Our code is like this: serverLocator = ActiveMQClient.createServerLocatorWithoutHA(transportConfiguration); serverLocator.setUseGlobalPools(true); serverLocator.setRetryInterval(2000); serverLocator.setRetryIntervalMultiplier(2.0); serverLocator.setMaxRetryInterval(10000); serverLocator.setReconnectAttempts(5); serverLocator.setCallFailoverTimeout(0); serverLocator.setCallTimeout(10000); serverLocator.setProducerWindowSize(-1); serverLocator.setMinLargeMessageSize(1024 * 1024); int windowSize = slowConsumer ? 1 : 200 * 1024; serverLocator.setConsumerWindowSize(windowSize); final ActiveMQConnection amqConnection = (ActiveMQConnection) factory.createConnection(username, password); Then we investigated the SharedEventLoopGroup code. There was some race conditions: Thread t1 closes connection and shutdowns the SharedEventLoopGroup. When t1 decrement channelFactoryCount to 0, thread t2 is creating connection and getting the instance. After that t1 schedules a shutdown task and cas shutdown reference. After 10 secs instance used by t1 will shutdown.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
