[
https://issues.apache.org/jira/browse/ARTEMIS-2257?focusedWorklogId=203594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-203594
]
ASF GitHub Bot logged work on ARTEMIS-2257:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 25/Feb/19 11:31
Start Date: 25/Feb/19 11:31
Worklog Time Spent: 10m
Work Description: wy96f commented on issue #2558: ARTEMIS-2257
Synchronize SharedEventLoopGroup shutdownGracefully
URL: https://github.com/apache/activemq-artemis/pull/2558#issuecomment-466977919
> @wy96f did you see my comment on the JIRA?
>
> Synchronized SharedEventLoopGroup is not the bug, it's the fix you applied
to fix the bug.
>
> I don't know in what condition the issue happened.
>
> Can you add more detail on how it happened? a test would be great if
possible... if you think a test is almost impossible due to an issue lets
discuss and see if there's a way around.
>
> Fixes without tests should be an exception... not the rule.
Sorry for my ambiguousness. We use jms client. We will close
ActiveMqConnection if in idle for 5 minutes and recreate connection if some
requests come in. These days we found exceptions occasionally as follows:
2019-02-24 08:23:44.294 [] [ThreadPoolAsynchronousProcessor-thread-2] WARN
io.netty.channel.AbstractChannel -Force-closing a channel whose registration
task was not accepted by an event loop: [id: 0xab32cb4c]
java.util.concurrent.RejectedExecutionException: event executor terminated
at
io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
at
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
at
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
at
io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:479)
at
io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
at
io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
at
io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
at
org.apache.activemq.artemis.core.remoting.impl.netty.DelegatingEventLoopGroup.register(DelegatingEventLoopGroup.java:46)
at
io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:332)
at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:616)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1004)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1040)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1232)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:862)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:775)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:221)
at
org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:643)
at
org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:748)
at
org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:248)
Before the stack trace some logs were printed showing idle connection is
closed.
Our code is like this:
serverLocator =
ActiveMQClient.createServerLocatorWithoutHA(transportConfiguration);
serverLocator.setUseGlobalPools(true);
serverLocator.setRetryInterval(2000);
serverLocator.setRetryIntervalMultiplier(2.0);
serverLocator.setMaxRetryInterval(10000);
serverLocator.setReconnectAttempts(5);
serverLocator.setCallFailoverTimeout(0);
serverLocator.setCallTimeout(10000);
serverLocator.setProducerWindowSize(-1);
serverLocator.setMinLargeMessageSize(1024 * 1024);
int windowSize = slowConsumer ? 1 : 200 * 1024;
serverLocator.setConsumerWindowSize(windowSize);
final ActiveMQConnection amqConnection = (ActiveMQConnection)
factory.createConnection(username, password);
Then we investigated the SharedEventLoopGroup code. There was some race
conditions:
Thread t1 closes connection and shutdowns the SharedEventLoopGroup. When t1
decrement channelFactoryCount to 0, thread t2 is creating connection and
getting the instance. After that t1 schedules a shutdown task and cas shutdown
reference. After 10 secs instance used by t1 will shutdown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 203594)
Time Spent: 0.5h (was: 20m)
> Synchronize SharedEventLoopGroup shutdownGracefully to prevent race conditions
> ------------------------------------------------------------------------------
>
> Key: ARTEMIS-2257
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2257
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.6.4
> Reporter: yangwei
> Priority: Critical
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> While thread a shutdowns shared event loop, between scheduling a task and set
> the shutdown reference thread b calls getInstance() and retrieves ready
> shutting down instance. Add synchronized block to fix the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
