[
https://issues.apache.org/jira/browse/ARTEMIS-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065455#comment-17065455
]
Justin Bertram edited comment on ARTEMIS-2677 at 3/27/20, 2:14 PM:
-------------------------------------------------------------------
Attached broker.xml and broker-slave.xml.
I am running a master on 3GB RAM and a slave on 2GB RAM. In general failover
works.
Load on Artemis we process is around average 5-7M messages (text and binary) a
day (approx single message size text message size 7kb). Max messages around
1GB.
I tried to reproduce again with same configuration in AWS with other pair of
master and slave by shutting down master. Failover worked fine and all
consumers were successfully moved to the slave, producers got updated topology
and Artemis was receiving messages, and didn't get
{{RejectedExecutionException}}.
I will try reproducing it in isolation on one machine. Can you please guide in
which situations Artemis throws {{RejectedExecutionException}} (e.g, is it
related to resources (memory, CPU, or slow NFS) on the machine at given
point-in-time when this happened)?
Please let me know if you have any suggestion related to configuration tunings
or resource changes needs to be applied help reduce the possibility of such
exception?
was (Author: jigaronline):
Attached broker.xml and broker-slave.xml.
I am running a master on 3GB RAM and a slave on 2GB RAM. In-general failover
works.
Load on Artemis we process is around average 5-7M messages (text and binary) a
day (approx single message size text message size7 kb). Max messages around
1GB.
I tried to reproduce again with same configuration in AWS with other pair of
master and slave by shutting down master. Failover worked fine and all
consumers were successfully moved to the slave, producers got updated topology
and Artemis was receiving messages, and didn't get "RejectedExecutionException".
I will try reproducing it, in isolation on one machine. Can you please guide in
which situations Artemis throws "RejectedExecutionException"?
E.g, I mean is it related to resources (memory, cpu or slow NFS) on the machine
at given point-in-time when this happened.
Please let me know if you have any suggestion related to configuration tunings
or resource changes needs to be applied help reduce the possiblity of such
exception?
> Artemis 2.11.0 RejectedExecutionException after successful failover
> -------------------------------------------------------------------
>
> Key: ARTEMIS-2677
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2677
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Environment: Environment: Artemis 2.11.0 (Master-Slave)
> SharedStore, FilePing. AWS EFS/NFS.
> Reporter: Jigar Shah
> Priority: Major
> Attachments: broker - slave.xml, broker.xml
>
>
> Observed an issue on master shutdown, the slave became active. Right after
> slave being active it started printing "RejectedExecutionException". Also,
> consumers from master to slave was not transferred. And client application
> stopped processing messages.
> Note RejectedExecutionException had [Terminated, pool size = 0, active
> threads = 0, queued tasks = 0, completed tasks = 59059]
> Following are the logs during failover from master to slave:
> Master1:
> {noformat}
> 2020-03-18 12:34:39,911 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 26fb51af-690c-11ea-959a-12f8371a8293
> 2020-03-18 12:34:39,912 INFO [org.apache.activemq.artemis.core.server]
> AMQ221029: stopped bridge
> $.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9
> 2020-03-18 12:34:39,950 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 2655265e-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 2655265e-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 2662e209-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 2662e209-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 2669bfdd-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,951 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 2669bfdd-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,993 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 26615b64-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,994 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 26615b64-690c-11ea-9d7c-12f7d16c4f81
> 2020-03-18 12:34:39,994 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to artemis1.sl.idsk.com/10.110.0.17:61616 has
> been detected: AMQ219015: The connection was disconnected because of server
> shutdown [code=DISCONNECTED]
> 2020-03-18 12:34:40,123 FINE [org.jgroups.protocols.FILE_PING] remove
> persistence-fs
> 2020-03-18 12:34:40,137 FINE [org.jgroups.protocols.FD_SOCK]
> ip-10-110-0-17-18496: socket to ip-10-110-0-88-23224 was closed gracefully
> 2020-03-18 12:34:40,138 FINE [org.jgroups.protocols.TCP] closing sockets and
> stopping threads
> 2020-03-18 12:34:40,169 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2020-03-18 12:34:41,336 INFO [io.hawt.web.AuthenticationFilter] Destroying
> hawtio authentication filter
> 2020-03-18 12:34:41,341 INFO [io.hawt.HawtioContextListener] Destroying
> hawtio services
> 2020-03-18 12:34:41,464 INFO
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed
> artemis-plugin plugin
> 2020-03-18 12:34:41,472 INFO
> [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed
> activemq-branding plugin
> 2020-03-18 12:34:41,509 INFO [org.apache.activemq.artemis.core.server]
> AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.11.0
> [19ade3bb-5f75-11ea-b327-1216d251b187] stopped, uptime 5 hours 2 minutes
> {noformat}
> Slave1:
> {noformat}
> 2020-03-18 12:35:01,613 INFO [org.apache.activemq.artemis.core.server]
> AMQ221010: Backup Server is now live
> 2020-03-18 12:35:01,626 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@5ced3917
> [name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187],
> temp=false]@58d2a652 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@5ced3917
> [name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.2c85cd9e-5f75-11ea-9be1-0af9119190b9,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187],
> temp=false]@58d2a652 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=artemis2-sl-idsk-com],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1788190971[nodeUUID=19ade3bb-5f75-11ea-b327-1216d251b187,
> connector=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=artemis2-sl-idsk-com, address=,
> server=ActiveMQServerImpl::serverUUID=19ade3bb-5f75-11ea-b327-1216d251b187]))
> [initialConnectors=[TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=artemis2-sl-idsk-com], discoveryGroupConfiguration=null]]
> is connected
> 2020-03-18 12:35:20,621 WARN [org.apache.activemq.artemis.core.server]
> AMQ222054: Error on executing IOCallback:
> java.util.concurrent.RejectedExecutionException: Task
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$25/191327468@479377d0
> rejected from
> org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@291e146[Terminated,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 59059]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> [rt.jar:1.8.0_212]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> [rt.jar:1.8.0_212]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> [rt.jar:1.8.0_212]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.onAddedTaskIfNotRunning(ProcessorBase.java:205)
> [artemis-commons-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.task(ProcessorBase.java:193)
> [artemis-commons-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.execute(OrderedExecutor.java:54)
> [artemis-commons-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.execute(OperationContextImpl.java:238)
> [artemis-server-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.checkTasks(OperationContextImpl.java:221)
> [artemis-server-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.done(OperationContextImpl.java:197)
> [artemis-server-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.IOCallback.lambda$done$0(IOCallback.java:43)
> [artemis-journal-2.11.0.jar:2.11.0]
> at java.util.ArrayList.forEach(ArrayList.java:1257) [rt.jar:1.8.0_212]
> at org.apache.activemq.artemis.core.io.IOCallback.done(IOCallback.java:41)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.DelegateCallback.done(DelegateCallback.java:41)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.doInternalWrite(NIOSequentialFile.java:395)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.internalWrite(NIOSequentialFile.java:359)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.access$100(NIOSequentialFile.java:43)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile$SyncLocalBufferObserver.flushBuffer(NIOSequentialFile.java:434)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:361)
> [artemis-journal-2.11.0.jar:2.11.0]
> at
> org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:457)
> [artemis-journal-2.11.0.jar:2.11.0]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
