[ 
https://issues.apache.org/jira/browse/ARTEMIS-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jigar Shah updated ARTEMIS-2677:
--------------------------------
    Attachment: broker.xml
                broker - slave.xml

> Artemis 2.11.0 RejectedExecutionException after successful failover
> -------------------------------------------------------------------
>
>                 Key: ARTEMIS-2677
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2677
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>         Environment: Environment: Artemis 2.11.0 (Master-Slave)
>  SharedStore, FilePing. AWS EFS/NFS.
>            Reporter: Jigar Shah
>            Priority: Major
>         Attachments: broker - slave.xml, broker.xml
>
>
> Observed an issue on master shutdown, the slave became active. Right after 
> slave being active it started printing "RejectedExecutionException". Also, 
> consumers from master to slave was not transferred. And client application 
> stopped processing messages.
> Note RejectedExecutionException had [Terminated, pool size = 0, active 
> threads = 0, queued tasks = 0, completed tasks = 59059]
> Following are the logs during failover from master to slave:
> Master1:
> {noformat}
> 2020-03-18 12:34:39,911 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222107: Cleared up resources for session 
> 26fb51af-690c-11ea-959a-12f8371a8293
> 2020-03-18 12:34:39,912 INFO [org.apache.activemq.artemis.core.server] 
> AMQ221029: stopped bridge 
> $.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9
> 2020-03-18 12:34:39,950 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222061: Client connection failed, clearing up resources for session 
> 2655265e-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222107: Cleared up resources for session 
> 2655265e-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222061: Client connection failed, clearing up resources for session 
> 2662e209-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222107: Cleared up resources for session 
> 2662e209-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222061: Client connection failed, clearing up resources for session 
> 2669bfdd-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222107: Cleared up resources for session 
> 2669bfdd-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,993 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222061: Client connection failed, clearing up resources for session 
> 26615b64-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,994 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222107: Cleared up resources for session 
> 26615b64-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,994 WARN [org.apache.activemq.artemis.core.client] 
> AMQ212037: Connection failure to artemis1.sl.idsk.com/10.110.0.17:61616 has 
> been detected: AMQ219015: The connection was disconnected because of server 
> shutdown [code=DISCONNECTED]
> 2020-03-18 12:34:40,123 FINE [org.jgroups.protocols.FILE_PING] remove 
> persistence-fs
> 2020-03-18 12:34:40,137 FINE [org.jgroups.protocols.FD_SOCK] 
> ip-10-110-0-17-18496: socket to ip-10-110-0-88-23224 was closed gracefully
> 2020-03-18 12:34:40,138 FINE [org.jgroups.protocols.TCP] closing sockets and 
> stopping threads
> 2020-03-18 12:34:40,169 WARN [org.apache.activemq.artemis.core.client] 
> AMQ212004: Failed to connect to server.
> 2020-03-18 12:34:41,336 INFO [io.hawt.web.AuthenticationFilter] Destroying 
> hawtio authentication filter
> 2020-03-18 12:34:41,341 INFO [io.hawt.HawtioContextListener] Destroying 
> hawtio services
> 2020-03-18 12:34:41,464 INFO 
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed 
> artemis-plugin plugin
> 2020-03-18 12:34:41,472 INFO 
> [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed 
> activemq-branding plugin
> 2020-03-18 12:34:41,509 INFO [org.apache.activemq.artemis.core.server] 
> AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.11.0 
> [19ade3bb-5f75-11ea-b327-1216d251b187] stopped, uptime 5 hours 2 minutes
> {noformat}
> Slave1:
> {noformat}
> 2020-03-18 12:35:01,613 INFO [org.apache.activemq.artemis.core.server] 
> AMQ221010: Backup Server is now live
> 2020-03-18 12:35:01,626 INFO [org.apache.activemq.artemis.core.server] 
> AMQ221027: Bridge ClusterConnectionBridge@5ced3917 
> [name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9, 
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9,
>  postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187], 
> temp=false]@58d2a652 targetConnector=ServerLocatorImpl 
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@5ced3917 
> [name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9, 
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9,
>  postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187], 
> temp=false]@58d2a652 targetConnector=ServerLocatorImpl 
> [initialConnectors=[TransportConfiguration(name=netty-connector, 
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
>  ?port=61616&host=artemis2-sl-idsk-com], 
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1788190971[nodeUUID=19ade3bb-5f75-11ea-b327-1216d251b187,
>  connector=TransportConfiguration(name=netty-connector, 
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
>  ?port=61617&host=artemis2-sl-idsk-com, address=, 
> server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187])) 
> [initialConnectors=[TransportConfiguration(name=netty-connector, 
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
>  ?port=61616&host=artemis2-sl-idsk-com], discoveryGroupConfiguration=null]] 
> is connected
> 2020-03-18 12:35:20,621 WARN [org.apache.activemq.artemis.core.server] 
> AMQ222054: Error on executing IOCallback: 
> java.util.concurrent.RejectedExecutionException: Task 
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$25/191327468@479377d0
>  rejected from 
> org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@291e146[Terminated,
>  pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 59059]
>  at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
>  [rt.jar:1.8.0_212]
>  at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) 
> [rt.jar:1.8.0_212]
>  at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) 
> [rt.jar:1.8.0_212]
>  at 
> org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:205)
>  [artemis-commons-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:193)
>  [artemis-commons-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:54)
>  [artemis-commons-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.execute(OperationContextImpl.java:238)
>  [artemis-server-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.checkTasks(OperationContextImpl.java:221)
>  [artemis-server-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.done(OperationContextImpl.java:197)
>  [artemis-server-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.IOCallback.lambda$done$0(IOCallback.java:43)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at java.util.ArrayList.forEach(ArrayList.java:1257) [rt.jar:1.8.0_212]
>  at org.apache.activemq.artemis.core.io.IOCallback.done(IOCallback.java:41) 
> [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.DelegateCallback.done(DelegateCallback.java:41)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.doInternalWrite(NIOSequentialFile.java:395)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.internalWrite(NIOSequentialFile.java:359)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.access$100(NIOSequentialFile.java:43)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile$SyncLocalBufferObserver.flushBuffer(NIOSequentialFile.java:434)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:361)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at 
> org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:457)
>  [artemis-journal-2.11.0.jar:2.11.0]
>  at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to