Benjamin Huang created AMQ-5486:
------------------------------------
Summary: Thread synchronization overhead is unexpectedly high
Key: AMQ-5486
URL: https://issues.apache.org/jira/browse/AMQ-5486
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.9.1
Environment: UbuntuServer 12.04 x86_64, Linux Kernel 3.20.23, OpenJDK
1.7.0_51 64-Bit Server VM, Xms 2G, Xms 8G, CPU: E5-2620 v2 X 2, 64 GB RAM
Reporter: Benjamin Huang
There's about 20 topics with virtual topic enabled, hundreds of
comsumers/producers connected to MQ on NIO transport connector. During the run
there're about 12000 msg flow in per second, not a very high rate, but ActiveMQ
consumes a lot of CPU resource (about 600%~1000%). To find out what's the most
CPU consuming code path, I use JProfiler to dig into the process.
Among all the NIO worker threads, most of them were frequently blocked and did
a little job between the 'unblocked' time. While they're expected spend most of
their time slices on waiting for work item and processed them.
!nio_worker_blocked_frequently.png!
After reviewing the monitor usage history and stats, I think these NIO workers
were competing fiercely with each other on executing a synchronized method
(DestinationMap::get), which is also the most hot spot in the program . I also
notice that the caller AbstractRegion::getDestinations acquires a read lock
before calling it, so I guess this could be a left out, read lock is the actual
lock type required here.
!monitor_usage_stats.png!
!nio_worker_blocked_1.png!
!nio_worker_blocked_2.png!
As a rookie to Java, it's too difficult for me to list all critical sections
between NIO workers, or between NIO workers and BrokerService which adds up to
the overall synchronize overhead. So I attach the relevant info, with the hope
of finding a feasible solution to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
