Fwd: ActiveMq Artemis crashes when large amqp messages need are delivered

Michael André Pearce Tue, 06 Mar 2018 21:02:07 -0800

@Clebert

Could this be a blocker for release?

Just sending it to the dev mail list incase it got missed in the users.

Begin forwarded message:

From: andi welchlin <[email protected]>
Date: 6 March 2018 at 09:08:39 GMT
To: [email protected]
Subject: ActiveMq Artemis crashes when large amqp messages need are delivered
Reply-To: [email protected]

Hello,

I tested Artemis as a stanalone broker (the current snapshot 2.5.0).

One program sent a 100MB amqp message to a queue in the broker. This could be done.

But while the message was delivered to a subscriber Artemis crashed.

The queue was defined like this:

<queues>
        <queue name="awe.test.queue">
          <address>awe.test.queue</address>
          <durable>true</durable>
        </queue>
      </queues>

And I set global size to 1000 MB:

      <global-max-size>1000Mb</global-max-size>

When I restarted the broker and connected the subscriber it crashed, again.

Since the log output is large I attached a file to this mail. The end of the output looks like this:

        at java.lang.Object.wait(Native Method)
        - waiting on java.lang.ref.ReferenceQueue$Lock@773bc0c1
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@3e627566
        at java.lang.Object.wait(Native Method)
        - waiting on java.lang.ref.Reference$Lock@3e627566
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

===============================================================================
End Thread dump
*******************************************************************************

I tested the same setup with a QPID C++ broker. It had no problems even with much larger messages.

Kind Regards,
Andreas

2018-03-06 09:48:44,990 INFO  [org.apache.activemq.artemis.core.server] 
AMQ221007: Server is now live
2018-03-06 09:48:44,991 INFO  [org.apache.activemq.artemis.core.server] 
AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.5.0-SNAPSHOT 
[0.0.0.0, nodeID=0d3faab0-211b-11e8-89f1-080027683f03] 
2018-03-06 09:48:45,298 INFO  
[org.apache.activemq.hawtio.branding.PluginContextListener] Initialized 
activemq-branding plugin
2018-03-06 09:48:45,414 INFO  
[org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized 
artemis-plugin plugin
2018-03-06 09:48:45,922 INFO  [io.hawt.HawtioContextListener] Initialising 
hawtio services
2018-03-06 09:48:45,947 INFO  [io.hawt.system.ConfigManager] Configuration will 
be discovered via system properties
2018-03-06 09:48:45,952 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 
1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2018-03-06 09:48:45,958 INFO  [io.hawt.jmx.UploadManager] Using file upload 
directory: /home/andreas/brokers/artemis/broker-config-new-25/tmp/uploads
2018-03-06 09:48:46,003 INFO  [io.hawt.web.AuthenticationFilter] Starting 
hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" 
role principal classes: 
"org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2018-03-06 09:48:46,058 INFO  [io.hawt.web.JolokiaConfiguredAgentServlet] 
Jolokia overridden property: [key=policyLocation, 
value=file:/home/andreas/brokers/artemis/broker-config-new-25/etc/jolokia-access.xml]
2018-03-06 09:48:46,107 INFO  [io.hawt.web.RBACMBeanInvoker] Using MBean 
[hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role 
based access control
2018-03-06 09:48:46,343 INFO  [io.hawt.system.ProxyWhitelist] Initial proxy 
whitelist: [localhost, 127.0.0.1, 10.0.2.15, 10.43.4.239, 
andreas-VirtualBox.dev.quo]
2018-03-06 09:48:46,667 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP 
Server started at http://localhost:8161
2018-03-06 09:48:46,667 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis 
Jolokia REST API available at http://localhost:8161/console/jolokia
2018-03-06 09:48:46,667 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis 
Console available at http://localhost:8161/console
2018-03-06 09:50:57,635 WARN  [org.apache.activemq.artemis.core.server] 
AMQ222061: Client connection failed, clearing up resources for session 
6c34e6ea-211b-11e8-a083-080027683f03
2018-03-06 09:50:57,642 WARN  [org.apache.activemq.artemis.core.server] 
AMQ222107: Cleared up resources for session 6c34e6ea-211b-11e8-a083-080027683f03
2018-03-06 09:53:44,074 ERROR [org.apache.activemq.artemis.core.server] 
AMQ224079: The process for the virtual machine will be killed, as component 
QueueImpl[name=test.awe.queue, postOffice=PostOfficeImpl 
[server=ActiveMQServerImpl::serverUUID=0d3faab0-211b-11e8-89f1-080027683f03], 
temp=false]@2f7c18fd is not responsive
2018-03-06 09:53:44,219 WARN  [org.apache.activemq.artemis.core.server] 
AMQ222199: Thread dump: 
*******************************************************************************
Complete Thread dump 
"Thread-84 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=133 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)



"Thread-83 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=132 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-82 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=131 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-81 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=130 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-80 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=129 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-79 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=128 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-78 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=127 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-77 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=126 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-76 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=125 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-75 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=124 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-74 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=123 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-73 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=122 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-72 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=121 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-71 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=120 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-70 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=119 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-69 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=118 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-68 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=117 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-67 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=116 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-66 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=115 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-65 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=114 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-64 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=113 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-63 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=112 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-62 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=111 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-61 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=110 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-60 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=109 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-59 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=108 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-58 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=107 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-57 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=106 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6086f959
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-3 (activemq-netty-threads)" Id=94 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-2 (activemq-netty-threads)" Id=90 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"threadDeathWatcher-2-1" Id=86 TIMED_WAITING
        at java.lang.Thread.sleep(Native Method)
        at 
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:159)
        at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)


"Thread-1 (activemq-netty-threads)" Id=85 WAITING on 
java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44 owned by 
"Thread-32 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=74
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at 
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:221)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:150)
        at 
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:141)
        at 
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:646)
        at 
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
        at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at 
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        at 
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-37 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=79 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44 
owned by "Thread-32 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=74
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at 
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:281)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.flush(AMQPConnectionContext.java:178)
        at 
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.flush(ActiveMQProtonRemotingConnection.java:136)
        at 
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread$1.run(RemotingServiceImpl.java:719)
        at 
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
        at 
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at 
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
        at 
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$4/1285524499.run(Unknown
 Source)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@30fdc458


"Thread-32 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=74 RUNNABLE
        at 
org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:352)
        at 
org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59)
        at 
org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.head(TransportOutputAdaptor.java:80)
        at 
org.apache.qpid.proton.engine.impl.SaslImpl$SaslTransportWrapper.head(SaslImpl.java:725)
        at 
org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.head(HandshakeSniffingTransportWrapper.java:151)
        at 
org.apache.qpid.proton.engine.impl.TransportImpl.head(TransportImpl.java:1528)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flushBytes(ProtonHandler.java:201)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onTransport(AMQPConnectionContext.java:340)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:89)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:462)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:289)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.flush(AMQPConnectionContext.java:178)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.deliverMessage(ProtonServerSenderContext.java:727)
        at 
org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.sendMessage(AMQPSessionCallback.java:625)
        at 
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:1099)
        at 
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:462)
        at 
org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2891)
        at 
org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2366)
        at 
org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2000(QueueImpl.java:105)
        at 
org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:3164)
        -  locked 
org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner@51ac782b
        at 
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
        at 
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at 
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
        at 
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$4/1285524499.run(Unknown
 Source)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

        Number of locked synchronizers = 2
        - java.util.concurrent.ThreadPoolExecutor$Worker@34fa104b
        - java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44


"DestroyJavaVM" Id=44 RUNNABLE


"Thread-4 (ActiveMQ-scheduled-threads)" Id=43 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Scheduler-120689887" Id=42 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a90616
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a90616
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-41" Id=41 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-40" Id=40 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-39" Id=39 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-38" Id=38 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-37" Id=37 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a9308d9
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-36-acceptor-0@22fc9b57-ServerConnector@2c3b6178{HTTP/1.1,[http/1.1]}{localhost:8161}"
 Id=36 RUNNABLE (in native)
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        -  locked java.lang.Object@84c4280
        at 
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352)
        at 
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-35-lowPrioritySelector" Id=35 BLOCKED on 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer@4cb6cc20 owned by 
"qtp849198527-34" Id=34
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:244)
        -  blocked on 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer@4cb6cc20
        at 
org.eclipse.jetty.util.thread.strategy.ProduceExecuteConsume.produce(ProduceExecuteConsume.java:75)
        at 
org.eclipse.jetty.io.ManagedSelector.lowPriorityProduce(ManagedSelector.java:113)
        at 
org.eclipse.jetty.io.ManagedSelector$$Lambda$28/388802347.run(Unknown Source)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:748)


"qtp849198527-34" Id=34 RUNNABLE (in native)
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        -  locked sun.nio.ch.Util$3@4e7fa16
        -  locked java.util.Collections$UnmodifiableSet@2690ef93
        -  locked sun.nio.ch.EPollSelectorImpl@4c22e183
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:302)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:254)
        -  locked org.eclipse.jetty.io.ManagedSelector$SelectorProducer@4cb6cc20
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:147)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produce(ExecuteProduceConsume.java:97)
        at 
org.eclipse.jetty.io.ManagedSelector$$Lambda$27/756080817.run(Unknown Source)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:748)


"Thread-0 (activemq-netty-threads)" Id=33 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-0 (activemq-netty-threads)" Id=32 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-0 (activemq-netty-threads)" Id=31 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-0 (activemq-netty-threads)" Id=30 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-0 (activemq-netty-threads)" Id=29 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at 
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-3 (ActiveMQ-scheduled-threads)" Id=28 WAITING on 
java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44 owned by 
"Thread-32 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@147ed70f)"
 Id=74
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.ReentrantLock$NonfairSync@2f72be44
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at 
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.tick(ProtonHandler.java:111)
        at 
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext$1.run(AMQPConnectionContext.java:386)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@3b0fe47a


"Thread-2 (ActiveMQ-scheduled-threads)" Id=27 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"activemq-failure-check-thread" Id=26 TIMED_WAITING on 
java.util.concurrent.CountDownLatch$Sync@368ff3eb
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.CountDownLatch$Sync@368ff3eb
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at 
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:745)


"Thread-1 (ActiveMQ-scheduled-threads)" Id=18 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-0 (ActiveMQ-scheduled-threads)" Id=17 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57b26188
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Apache ActiveMQ Artemis libaio poller" Id=16 RUNNABLE (in native)
        at org.apache.activemq.artemis.jlibaio.LibaioContext.blockedPoll(Native 
Method)
        at 
org.apache.activemq.artemis.jlibaio.LibaioContext.poll(LibaioContext.java:356)
        at 
org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread.run(AIOSequentialFileFactory.java:397)


"activemq-buffer-timeout" Id=15 WAITING on 
java.util.concurrent.Semaphore$NonfairSync@d838866
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.Semaphore$NonfairSync@d838866
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at 
org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:484)
        at java.lang.Thread.run(Thread.java:748)


"Thread-0 (-scheduled-threads)" Id=14 RUNNABLE
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at 
org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
        at 
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:910)
        at 
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$0(ActiveMQServerImpl.java:590)
        at 
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$3/1791045777.run(Unknown
 Source)
        at 
org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireAction(CriticalAnalyzerImpl.java:150)
        at 
org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:135)
        at 
org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
        at 
org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284)
        at 
org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:294)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@78aab498


"ActiveMQ Artemis Server Shutdown Timer" Id=10 TIMED_WAITING on 
java.util.TaskQueue@2c4e1232
        at java.lang.Object.wait(Native Method)
        -  waiting on java.util.TaskQueue@2c4e1232
        at java.util.TimerThread.mainLoop(Timer.java:552)
        at java.util.TimerThread.run(Timer.java:505)


"Signal Dispatcher" Id=5 RUNNABLE


"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@773bc0c1
        at java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.ReferenceQueue$Lock@773bc0c1
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)


"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@3e627566
        at java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.Reference$Lock@3e627566
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)


===============================================================================
End Thread dump 
*******************************************************************************

Fwd: ActiveMq Artemis crashes when large amqp messages need are delivered

Reply via email to