[
https://issues.apache.org/jira/browse/AMQ-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817518#comment-13817518
]
Joshua Watkins commented on AMQ-4853:
-------------------------------------
After doing a bit more research I think the issue is a bit more fundamental
than simply using the Queue vs a Map. (The queue implementation definitely
makes it worse.) I have been looking at the profiling information more and
doing a few tests. I see that the queue is continually growing. Digging a bit
deeper I see that the ConsumerInfo class does not have an equals method.
Without an equals method the consumers.remove() call in
AdvisoryBroker.removeConsumer is pretty useless if we ever create another
consumer info object - which I believe to be the case.
> Advisory support leads to excessive CPU usage
> ---------------------------------------------
>
> Key: AMQ-4853
> URL: https://issues.apache.org/jira/browse/AMQ-4853
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.0
> Reporter: Joshua Watkins
> Attachments: amqAdvirsoryCPUIssue.jpg
>
>
> After upgrading from AMQ 5.8 to 5.9 we have seen cpu usage continually climb
> until AMQ threads are taking nearly all of the CPU Resources while remaining
> fairly idle. This is just a single broker with advisory support on. (Advisory
> support is on in this case as we run the same config for a network of
> brokers.) Turning off advisory support reduced the CPU load to single digits.
> top -H output:
> Cpu(s): 97.8%us, 2.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 7872040k total, 6574324k used, 1297716k free, 301028k buffers
> Swap: 0k total, 0k used, 0k free, 1635392k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 25476 root 16 0 2173m 891m 11m R 14.4 11.6 86:12.10 java
>
>
> 25472 root 15 0 2173m 891m 11m R 13.6 11.6 86:09.77 java
>
>
> 25155 root 16 0 2173m 891m 11m R 9.8 11.6 86:26.13 java
>
>
> 25471 root 16 0 2173m 891m 11m R 9.2 11.6 86:12.93 java
>
>
> 25514 root 16 0 2173m 891m 11m R 9.2 11.6 86:15.12 java
>
>
> 25202 root 16 0 2173m 891m 11m R 8.7 11.6 86:33.20 java
>
>
> 25189 root 16 0 2173m 891m 11m S 8.4 11.6 86:24.65 java
>
>
> 25274 root 16 0 2173m 891m 11m R 8.1 11.6 86:18.45 java
>
>
> 19272 root 15 0 2173m 891m 11m S 8.1 11.6 8:40.19 java
>
>
> 20039 root 15 0 2173m 891m 11m S 8.1 11.6 8:15.53 java
>
>
> 19270 root 15 0 2173m 891m 11m R 7.8 11.6 8:35.85 java
>
>
> 25134 root 16 0 2173m 891m 11m R 7.5 11.6 90:42.29 java
>
>
> 25259 root 15 0 2173m 891m 11m R 7.5 11.6 90:30.02 java
>
>
> 25474 root 16 0 2173m 891m 11m R 7.5 11.6 86:13.24 java
>
>
> 25475 root 16 0 2173m 891m 11m R 7.5 11.6 86:11.74 java
>
>
> 25483 root 16 0 2173m 891m 11m R 7.5 11.6 86:12.30 java
>
>
> 25210 root 15 0 2173m 891m 11m R 7.2 11.6 90:35.24 java
>
>
> 25129 root 16 0 2173m 891m 11m S 6.9 11.6 90:31.27 java
>
>
> 25249 root 16 0 2173m 891m 11m S 6.9 11.6 86:18.35 java
>
>
> 25489 root 16 0 2173m 891m 11m R 6.9 11.6 86:16.93 java
>
>
> 9971 root 15 0 2173m 891m 11m S 6.6 11.6 38:53.66 java
>
>
> 25116 root 16 0 2173m 891m 11m R 6.3 11.6 90:28.39 java
>
>
> 25513 root 16 0 2173m 891m 11m R 6.3 11.6 86:07.62 java
> Thread dump snippets:
> PID 25476 --
> "ActiveMQ Transport: tcp:///10.33.154.95:48799@61616" daemon prio=10
> tid=0x00002aaaf5288800 nid=0x6384 runnable [0x00000000489b1000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
> PID - 25473
> "ActiveMQ Transport: tcp:///10.33.154.95:48802@61616" daemon prio=10
> tid=0x00002aaaf6afc800 nid=0x6381 runnable [0x000000004779f000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
> PID 25471 --
> "ActiveMQ Transport: tcp:///10.33.154.95:48803@61616" daemon prio=10
> tid=0x00002aaaf6f4b800 nid=0x6380 runnable [0x00000000487af000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
