Matthew Smigielski created AMQ-8487:
---------------------------------------
Summary: JMS messages fill up in queues under heavy performance
volumes
Key: AMQ-8487
URL: https://issues.apache.org/jira/browse/AMQ-8487
Project: ActiveMQ
Issue Type: Bug
Components: AMQP
Affects Versions: 5.16.3
Reporter: Matthew Smigielski
*Summary:*
Under heavy loads, I'm observing that some JMS queues start to fill up far
beyond the workload being done, causing my application to do needless work.
Once I stop the load generators, the high number of messages usually get
consumed quickly.
*Details:*
I am one of the performance QA testers for a Java based order management
solution. Orders are created and moved through various statuses in the system
(referred to as an order pipeline). Stand alone JVMs (referred to as agents) do
a specific task to move orders through the system. A JDBC database stores data
and JMS is used for the agents to communicate and divide work.
For example, a simple pipeline would be:
{{Created -> Scheduled -> Released -> Shipped}}
There would be 4 agents in this pipeline, one for each of the statuses, and
each agent has its own JMS queue. The agents are all multi-threaded. The first
thread runs a method called getJobs() which goes to the JDBC database to fetch
work. It will fetch a user specified amount of orders (in my case 5,000) and
put them into the JMS queue in batches of 5,000. The other threads of the agent
run a method called executeJobs() which then, in a multi-threaded fashion,
consume these messages and do work. Once the first 5,000 messages are consumed,
5,000 more messages are put in the queue, and so forth. The agents are time
triggered, similar to that of a Unix cron job, in that they run every X
minutes. All of the agents run asynchronously to each other. Some agents are
faster than others (for example, createOrder is faster than scheduleOrder).
An example of the workflow would be:
1.) My tests starts by feeding messages manually into a JMS queue for the
createOrder agent to read from - I use Jmeter for this.
2.) The createOrder agent reads these JMS messages and creates orders for them
in the JDBC database. These orders are marked that they need to be scheduled.
3.) When the scheduleOrder agent triggers, it goes to the JDBC database and
fetches the work it needs by storing it the JMS queue for scheduleOrder. The
remaining agent threads process these orders.
4.) When the releaseOrder agent triggers, the same thing happens as it did for
scheduleOrder, but this time for orders who need to be released.
5.) When the shipping agent triggers, the same thing again happens.
The Problem:
As a performance QA tester, I try to push the system as much as possible by
feeding messages into the createOrder JMS queue as fast as possible and start
up multiple agents of each type and change the number of threads of them to
tune them. If I push messages into the createOrder queue via Jmeter too fast,
createOrder can handle them but scheduleOrder cannot as it has more processing
logic. Over time, the scheduleOrder queue starts to fill up much past 5,000,
which would be the theoretical maximum of any queue in this configuration.
Sometimes it gets into the hundreds of thousands. If I stop the load generator
into the createOrder queue, usually after some time this backlog of messages
will disappear very fast.
Our application supports other JMS vendors such as Weblogic JMS and IBM MQ
which do not see this problem. I am not a developer but have been told that our
code is all JMS compliant and there is nothing we can do on our end. I am a big
fan of ActiveMQ and am trying to push its adoption by our product but this is a
roadblock. This is most likely one or more of the following:
- A bug in ActiveMQ
- Some ActiveMQ config or tuning I need to do
- A bug in our product
I've seen this issue for a few years now and am currently seeing it on ActiveMQ
5.16.3. I've seen this on both RHEL and CentOS 7 and 8. I have not done any
ActiveMQ tuning out of the box. Please let me know if I can provide any
ActiveMQ logs or debugging analysis and I will gladly share it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
