[
https://issues.apache.org/jira/browse/AMQ-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816125#comment-13816125
]
Joshua Watkins commented on AMQ-4853:
-------------------------------------
Timothy,
I just had a look at the code and I believe I see what the issue is. The change
you made[1] moving the consumers hashmap to a queue is causing the performance
issue. We have quite a few consumers so on remove we would have to potentially
traverse all the consumers to remove the one.
As far as our code we do use camel with maxConsumers settings and request reply
messages. So I would expect a bit of churn when it comes consumers coming and
going.
1.
https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commitdiff;h=74dafd7f24028c3503758581166ec10bb3d5116a
> Advisory support leads to excessive CPU usage
> ---------------------------------------------
>
> Key: AMQ-4853
> URL: https://issues.apache.org/jira/browse/AMQ-4853
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.0
> Reporter: Joshua Watkins
>
> After upgrading from AMQ 5.8 to 5.9 we have seen cpu usage continually climb
> until AMQ threads are taking nearly all of the CPU Resources while remaining
> fairly idle. This is just a single broker with advisory support on. (Advisory
> support is on in this case as we run the same config for a network of
> brokers.) Turning off advisory support reduced the CPU load to single digits.
> top -H output:
> Cpu(s): 97.8%us, 2.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 7872040k total, 6574324k used, 1297716k free, 301028k buffers
> Swap: 0k total, 0k used, 0k free, 1635392k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 25476 root 16 0 2173m 891m 11m R 14.4 11.6 86:12.10 java
>
>
> 25472 root 15 0 2173m 891m 11m R 13.6 11.6 86:09.77 java
>
>
> 25155 root 16 0 2173m 891m 11m R 9.8 11.6 86:26.13 java
>
>
> 25471 root 16 0 2173m 891m 11m R 9.2 11.6 86:12.93 java
>
>
> 25514 root 16 0 2173m 891m 11m R 9.2 11.6 86:15.12 java
>
>
> 25202 root 16 0 2173m 891m 11m R 8.7 11.6 86:33.20 java
>
>
> 25189 root 16 0 2173m 891m 11m S 8.4 11.6 86:24.65 java
>
>
> 25274 root 16 0 2173m 891m 11m R 8.1 11.6 86:18.45 java
>
>
> 19272 root 15 0 2173m 891m 11m S 8.1 11.6 8:40.19 java
>
>
> 20039 root 15 0 2173m 891m 11m S 8.1 11.6 8:15.53 java
>
>
> 19270 root 15 0 2173m 891m 11m R 7.8 11.6 8:35.85 java
>
>
> 25134 root 16 0 2173m 891m 11m R 7.5 11.6 90:42.29 java
>
>
> 25259 root 15 0 2173m 891m 11m R 7.5 11.6 90:30.02 java
>
>
> 25474 root 16 0 2173m 891m 11m R 7.5 11.6 86:13.24 java
>
>
> 25475 root 16 0 2173m 891m 11m R 7.5 11.6 86:11.74 java
>
>
> 25483 root 16 0 2173m 891m 11m R 7.5 11.6 86:12.30 java
>
>
> 25210 root 15 0 2173m 891m 11m R 7.2 11.6 90:35.24 java
>
>
> 25129 root 16 0 2173m 891m 11m S 6.9 11.6 90:31.27 java
>
>
> 25249 root 16 0 2173m 891m 11m S 6.9 11.6 86:18.35 java
>
>
> 25489 root 16 0 2173m 891m 11m R 6.9 11.6 86:16.93 java
>
>
> 9971 root 15 0 2173m 891m 11m S 6.6 11.6 38:53.66 java
>
>
> 25116 root 16 0 2173m 891m 11m R 6.3 11.6 90:28.39 java
>
>
> 25513 root 16 0 2173m 891m 11m R 6.3 11.6 86:07.62 java
> Thread dump snippets:
> PID 25476 --
> "ActiveMQ Transport: tcp:///10.33.154.95:48799@61616" daemon prio=10
> tid=0x00002aaaf5288800 nid=0x6384 runnable [0x00000000489b1000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
> PID - 25473
> "ActiveMQ Transport: tcp:///10.33.154.95:48802@61616" daemon prio=10
> tid=0x00002aaaf6afc800 nid=0x6381 runnable [0x000000004779f000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
> PID 25471 --
> "ActiveMQ Transport: tcp:///10.33.154.95:48803@61616" daemon prio=10
> tid=0x00002aaaf6f4b800 nid=0x6380 runnable [0x00000000487af000]
> java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentLinkedQueue.remove(ConcurrentLinkedQueue.java:346)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:270)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:132)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:137)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:619)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:619)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
