[
https://issues.apache.org/jira/browse/AMQ-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua Watkins updated AMQ-4853:
--------------------------------
Attachment: amqAdvirsoryCPUIssue.jpg
+1 Rob.
Tim,
I will try to come up with a stress test for this code. Attached is a Profiling
session when the cpu was at nearly 100% usage to show just how badly it is
performing.
> Advisory support leads to excessive CPU usage
> ---------------------------------------------
>
> Key: AMQ-4853
> URL: https://issues.apache.org/jira/browse/AMQ-4853
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.0
> Reporter: Joshua Watkins
> Attachments: amqAdvirsoryCPUIssue.jpg
>
>
> After upgrading from AMQ 5.8 to 5.9 we have seen cpu usage continually climb
> until AMQ threads are taking nearly all of the CPU Resources while remaining
> fairly idle. This is just a single broker with advisory support on. (Advisory
> support is on in this case as we run the same config for a network of
> brokers.) Turning off advisory support reduced the CPU load to single digits.
> top -H output:
> Cpu(s): 97.8%us, 2.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 7872040k total, 6574324k used, 1297716k free, 301028k buffers
> Swap: 0k total, 0k used, 0k free, 1635392k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 25476 root 16 0 2173m 891m 11m R 14.4 11.6 86:12.10 java
>
>
> 25472 root 15 0 2173m 891m 11m R 13.6 11.6 86:09.77 java
>
>
> 25155 root 16 0 2173m 891m 11m R 9.8 11.6 86:26.13 java
>
>
> 25471 root 16 0 2173m 891m 11m R 9.2 11.6 86:12.93 java
>
>
> 25514 root 16 0 2173m 891m 11m R 9.2 11.6 86:15.12 java
>
>
> 25202 root 16 0 2173m 891m 11m R 8.7 11.6 86:33.20 java
>
>
> 25189 root 16 0 2173m 891m 11m S 8.4 11.6 86:24.65 java
>
>
> 25274 root 16 0 2173m 891m 11m R 8.1 11.6 86:18.45 java
>
>
> 19272 root 15 0 2173m 891m 11m S 8.1 11.6 8:40.19 java
>
>
> 20039 root 15 0 2173m 891m 11m S 8.1 11.6 8:15.53 java
>
>
> 19270 root 15 0 2173m 891m 11m R 7.8 11.6 8:35.85 java
>
>
> 25134 root 16 0 2173m 891m 11m R 7.5 11.6 90:42.29 java
>
>
> 25259 root 15 0 2173m 891m 11m R 7.5 11.6 90:30.02 java
>
>
> 25474 root 16 0 2173m 891m 11m R 7.5 11.6 86:13.24 java
>
>
> 25475 root 16 0 2173m 891m 11m R 7.5 11.6 86:11.74 java
>
>
> 25483 root 16 0 2173m 891m 11m R 7.5 11.6 86:12.30 java
>
>
> 25210 root 15 0 2173m 891m 11m R 7.2 11.6 90:35.24 java
>
>
> 25129 root 16 0 2173m 891m 11m S 6.9 11.6 90:31.27 java
>
>
> 25249 root 16 0 2173m 891m 11m S 6.9 11.6 86:18.35 java
>
>
> 25489 root 16 0 2173m 891m 11m R 6.9 11.6 86:16.93 java
>
>
> 9971 root 15 0 2173m 891m 11m S 6.6 11.6 38:53.66 java
>
>
> 25116 root 16 0 2173m 891m 11m R 6.3 11.6 90:28.39 java
>
>
> 25513 root 16 0 2173m 891m 11m R 6.3 11.6 86:07.62 java
> Thread dump snippets:
> PID 25476 --
> "ActiveMQ Transport: tcp:///10.33.154.95:48799@61616" daemon prio=10
> tid=0x00002aaaf5288800 nid=0x6384 runnable [0x00000000489b1000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
> PID - 25473
> "ActiveMQ Transport: tcp:///10.33.154.95:48802@61616" daemon prio=10
> tid=0x00002aaaf6afc800 nid=0x6381 runnable [0x000000004779f000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
> PID 25471 --
> "ActiveMQ Transport: tcp:///10.33.154.95:48803@61616" daemon prio=10
> tid=0x00002aaaf6f4b800 nid=0x6380 runnable [0x00000000487af000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
