Re: Low throughput with thousands of queues
Hi Flavio,
How healthy does the heap look on the JVM that is running the JMS clients?
We had done a prototype of something like this with java broker, and
throughput drop was mostly attributed to running out of heap space on
broker. This might still happen in C++ broker, however I don't have recent
experience in C++ memory management. Looking at memory on both client and
server is a good option.
We were heading in the direction of 100k queues, but when we saw this
changed our application design to live with smaller number of queues.
Thanks
Vijay
On Tue, Nov 12, 2013 at 8:56 AM, Flavio Baronti f.baro...@list-group.comwrote:
Hi Rob,
thanks for your interest. I'm using the C++ broker on Linux, and the Java
client on windows.
The sender is sending through producers obtained with
MessageProducer topicProducer =
session.createProducer(session.createTopic(topicName
+ ; {node: { type: topic } }));
while the receiver is receiving with a listener set on consumers obtained
with
MessageConsumer consumer = session.createConsumer(
session.createTopic(topicName));
I don't know what to gather from CPU usage. With few sessions it's higher
on the broker, and lower on the client.
Increasing the number of session it's the other way round, higher on
client and lower on broker.
In both cases we're very far from 100%.
I created a bug report, I'm not too familiar with JIRA so I hope I did it
ok.
I added the two test programs I'm using. If there is anything else I can
help with, please let me know.
Flavio
Il 2013/11/12 10:36 AM, Rob Godfrey ha scritto:
Hi Flavio,
a few questions so we can help investigate your issue:
are you using the Java or the C++ broker?
how are you creating the 100k topics/queues?
when you run your test is it obvious whether the broker or the client is
the bottleneck - e.g. is one of these using significantly more CPU time
than the other?
would it be possible to provide for you to provide your test code so that
we can try to replicate the issue (e.g. by attaching to a JIRA)?
thanks,
Rob
On 8 November 2013 12:18, Flavio Baronti f.baro...@list-group.com
wrote:
Hello,
I have an use case where I need to create hundreds of thousands of
queues,
each one subscribed to a different topic (therefore I have as many topics
as queues).
I set up a test with a single producer generating data on a randomly
chosen topic, and a receiver retrieving data from the queues (and
throwing
it away).
I'm using the JMS api, and doing the obvious thing makes the throughput
drop dramatically from 10k msg/sec with a single topic/queue (around the
top my network adapter can sustain) to 20 msg/sec with 100k
topics/queues.
I found out that I can recover performance by using more JMS sessions and
connections - e.g. create 4 connections with 100 sessions each, and
randomly distributing the receiving queues on them.
This however is less than ideal, since with the JMS client a thread is
created for each session, and I definitely don't want 400 threads
receiving
data. The work I have to do is CPU-bound, and I don't want to waste time
in
context switching when 2/4 threads can suffice.
Why does the throughput drop so badly with many topics/queues? Why adding
sessions helps? Am I overlooking something, or doing something wrong?
Thanks
Flavio
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
Re: Low throughput with thousands of queues
Hi Flavio,
Wonder if the 'dispatcher' thread pool on the broker is too small; When
architecting our system, we understood the dispatch is round robin from the
broker side. If the broker thread pool has too many 'queues' to visit
before doing the next dispatch, it might become the bottleneck. Just a
thought, I haven't done much digging here..
Thanks
Vijay
On Tue, Nov 12, 2013 at 9:30 AM, Flavio Baronti f.baro...@list-group.comwrote:
Hi Vijay,
thanks for the suggestion. The broker seems stable from the point of view
of memory, unless I send more messages than the client can receive; in that
case messages start to pile up in queues, and memory usage goes up.
Memory usage on the client is negligible in all cases.
Flavio
Il 2013/11/12 18:08 PM, Vijay Devadhar ha scritto:
Hi Flavio,
How healthy does the heap look on the JVM that is running the JMS clients?
We had done a prototype of something like this with java broker, and
throughput drop was mostly attributed to running out of heap space on
broker. This might still happen in C++ broker, however I don't have
recent
experience in C++ memory management. Looking at memory on both client and
server is a good option.
We were heading in the direction of 100k queues, but when we saw this
changed our application design to live with smaller number of queues.
Thanks
Vijay
On Tue, Nov 12, 2013 at 8:56 AM, Flavio Baronti f.baro...@list-group.com
wrote:
Hi Rob,
thanks for your interest. I'm using the C++ broker on Linux, and the Java
client on windows.
The sender is sending through producers obtained with
MessageProducer topicProducer = session.createProducer(
session.createTopic(topicName
+ ; {node: { type: topic } }));
while the receiver is receiving with a listener set on consumers obtained
with
MessageConsumer consumer = session.createConsumer(
session.createTopic(topicName));
I don't know what to gather from CPU usage. With few sessions it's higher
on the broker, and lower on the client.
Increasing the number of session it's the other way round, higher on
client and lower on broker.
In both cases we're very far from 100%.
I created a bug report, I'm not too familiar with JIRA so I hope I did it
ok.
I added the two test programs I'm using. If there is anything else I can
help with, please let me know.
Flavio
Il 2013/11/12 10:36 AM, Rob Godfrey ha scritto:
Hi Flavio,
a few questions so we can help investigate your issue:
are you using the Java or the C++ broker?
how are you creating the 100k topics/queues?
when you run your test is it obvious whether the broker or the client is
the bottleneck - e.g. is one of these using significantly more CPU time
than the other?
would it be possible to provide for you to provide your test code so
that
we can try to replicate the issue (e.g. by attaching to a JIRA)?
thanks,
Rob
On 8 November 2013 12:18, Flavio Baronti f.baro...@list-group.com
wrote:
Hello,
I have an use case where I need to create hundreds of thousands of
queues,
each one subscribed to a different topic (therefore I have as many
topics
as queues).
I set up a test with a single producer generating data on a randomly
chosen topic, and a receiver retrieving data from the queues (and
throwing
it away).
I'm using the JMS api, and doing the obvious thing makes the throughput
drop dramatically from 10k msg/sec with a single topic/queue (around
the
top my network adapter can sustain) to 20 msg/sec with 100k
topics/queues.
I found out that I can recover performance by using more JMS sessions
and
connections - e.g. create 4 connections with 100 sessions each, and
randomly distributing the receiving queues on them.
This however is less than ideal, since with the JMS client a thread is
created for each session, and I definitely don't want 400 threads
receiving
data. The work I have to do is CPU-bound, and I don't want to waste
time
in
context switching when 2/4 threads can suffice.
Why does the throughput drop so badly with many topics/queues? Why
adding
sessions helps? Am I overlooking something, or doing something wrong?
Thanks
Flavio
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
RE: Batch/Bulk receive messages using java client?
Hi Gordon, Thanks for the suggestion. What is the reason for recommendation to not mix onMessage() and subsequent receive? In theory, this will work well since onMessage() is like a signal and receive() will actually fetch all the data; This will avoid the need for periodic polling of data. In practice, will we run into efficiency issues or some other mixup in the transport layer? I am just trying to understand. Thanks Vijay -Original Message- From: Gordon Sim [mailto:g...@redhat.com] Sent: Thursday, July 12, 2012 10:09 AM To: users@qpid.apache.org Subject: Re: Batch/Bulk receive messages using java client? On 07/12/2012 05:53 PM, Praveen M wrote: Hi, I'm trying to explore if there are ways to batch message processing. Batching message processing would help us improve performance for some of our use cases, where we could chunk messages and process them in a single callback. Have anyone here explored building a layer to batch messages. I am using the Java Broker and the Java client. I would like to stick to the JMS api as much as possible. This is what I currently have, still wondering if it'd work. 1) When the onMessage() callback is triggered, create a consumer a pull more messages to process from the queue where the message was delivered from. 2) Pull messages upto the number of my max chunk size, or upto the messages available in the queue. 3) process all the messages together and commit on the session. I'd like to hear ideas on how to go about this. I wouldn't mix using MessageListener and receive() (and you don't need to create a consumer per batch). Otherwise, using receive() to get N messages (if available), process them all together and then either manually acknowledge or commit seems quite simple and sensible to me. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: Batch/Bulk receive messages using java client?
Thanks Rajith; The reason for onMessage() is to be able to listen on multiple consumers from a single thread; If you used receive(), you need to have one thread per consumer blocked on receive and in a system that is general purpose messaging framework for multiple consumers with varying traffic load on each consumer, reserving thread capacity is not a good problem to solve. Thanks Vijay -Original Message- From: Rajith Attapattu [mailto:rajit...@gmail.com] Sent: Friday, July 13, 2012 10:58 AM To: users@qpid.apache.org Subject: Re: Batch/Bulk receive messages using java client? Both onMessage() and receive() will fetch messages. By default we prefetch 500 messages at a time (if available on the server). I don't see any advantage of using both receive and ML together, it will only complicate your code. There is no periodic polling, when messages become available the broker will push messages to the client subject to credit availability. Rajith On Fri, Jul 13, 2012 at 12:45 PM, Vijay Devadhar vdevad...@salesforce.com wrote: Hi Gordon, Thanks for the suggestion. What is the reason for recommendation to not mix onMessage() and subsequent receive? In theory, this will work well since onMessage() is like a signal and receive() will actually fetch all the data; This will avoid the need for periodic polling of data. In practice, will we run into efficiency issues or some other mixup in the transport layer? I am just trying to understand. Thanks Vijay -Original Message- From: Gordon Sim [mailto:g...@redhat.com] Sent: Thursday, July 12, 2012 10:09 AM To: users@qpid.apache.org Subject: Re: Batch/Bulk receive messages using java client? On 07/12/2012 05:53 PM, Praveen M wrote: Hi, I'm trying to explore if there are ways to batch message processing. Batching message processing would help us improve performance for some of our use cases, where we could chunk messages and process them in a single callback. Have anyone here explored building a layer to batch messages. I am using the Java Broker and the Java client. I would like to stick to the JMS api as much as possible. This is what I currently have, still wondering if it'd work. 1) When the onMessage() callback is triggered, create a consumer a pull more messages to process from the queue where the message was delivered from. 2) Pull messages upto the number of my max chunk size, or upto the messages available in the queue. 3) process all the messages together and commit on the session. I'd like to hear ideas on how to go about this. I wouldn't mix using MessageListener and receive() (and you don't need to create a consumer per batch). Otherwise, using receive() to get N messages (if available), process them all together and then either manually acknowledge or commit seems quite simple and sensible to me. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: Qpid Java Broker with persistance supports only finite number of messages?
Great discussion folks; Original question Praveen posted was indeed motivated by A discussion we were having about how to handle unexpected loads. In the past, I have come across producers who went awry and enqueued millions of messages. I guess we have to start throttling the producer that does something like this to protect the broker from running out of memory; I haven't read docs about flow control etc., so won't ask questions yet. :-) -Original Message- From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] Sent: Thursday, January 05, 2012 10:58 AM To: users@qpid.apache.org Subject: Re: Qpid Java Broker with persistance supports only finite number of messages? On 5 January 2012 19:30, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: Just to jump in on this thread. Re but my opinion is that if you have millions of messages then a Message Broker is the wrong solution to your problem - you want a Database. I can't say I agree with Rob's assertion here!! Well maybe that's a reasonable comment if the *intention* is to have millions of messages hanging around, but what if it's due to an unfortunate circumstance.. Indeed - that is what I meant :-) Planning for failure scenarios should always be carefully considered... As Robbie mentioned we have had people build up millions of messages on persistent queues... and if you use the 64-bit JVM and put large amounts of RAM in your box, then the physical limitations of maximum queue size can be made acceptably large... However I would always caution against designing a system where you expect to hold this number of *persistent* messages as part of your regular process - especially using the Java Broker as it is not designed for this sort of use case. The case you describe is somewhat different in that you are looking at transient messages (where one would expect the rate of production/consumption to be higher), and you want the broker to be capable of buffering for N minutes while temporary network outages occur. Flowing to disk may make sense here as long as the rate at which messages can flow through the disk is higher than the rate at which they are entering the broker (and actually high enough so that the backlog can quickly be reduced). This is where the flow-to-disk work that I hope to be starting soon on the Java Broker would play. -- Rob - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org
Is there a overhead to priority queues?
Hi Qpid users, We have a use case where an application has multiple message types and only some of them use priority; Instead of keeping FIFO queues and PRIORITY queues separate in the transport layer and managing their allocation correctly in the app specific routing layer, I was wondering if I should create all queues in QPID as priority queues and then leave it up to the producers to specify priority or not. This way application does not have to manage message type to FIFO vs PRIORITY requirements; This brings up the question as to what is the overhead on java broker and java client side for Priority queues vs FIFO queues. Any experience around this? Thanks Vijay
RE: Qpid Java Broker performance lower than expected
Hi Robbie, Thanks for detailed email responses; They really help us in understanding the internals of qpid; In this and earlier email to Praveen (my co-worker), I had the sense that you were suggesting that Berkely DB (BDB) persistent layer was better than Derby. Is that the case? If yes, what in terms of design, architecture and interaction with messaging layer makes BDB better than Derby store? Thanks Vijay -Original Message- From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] Sent: Tuesday, October 25, 2011 4:38 PM To: users@qpid.apache.org Subject: Re: Qpid Java Broker performance lower than expected Hi Vinay, I havent done any performance benchmarking of the Derby store to know what a representative number would actually be, but I will try to take a look at some point. I havent actually used QpidBench, so can I ask if there were any specific command(s) you ran so I can try the same scenarios? We havent paid much attention to performance of the Java broker for a while unfortunately because we have been working on various other issues such agetting memory usage under control and sorting out correctness issues etc since adding a newer protocol version and doing some significant refactorings and reimplementations, but as we reach the light at the end of the tunnel on those it is something which should move further up the priority list. It is worth nothing that there is also a BDB persistent store for the Java broker that you might want to look at, as I would expect it to be faster. It has recently been moved into the main repo, but is still an optional module which you need to explicitly ask for to be built (because BDB itself uses the Sleepycat Licence, which invokes restrictions upon distribution that mean it is not Apache Licence compatible). You can build the store module and include it (but not BDB itself) in the broker binary release bundle by using the following build command: ant build release-bin -Dmodules.opt=bdbstore -Ddownload-bdb=true You will find that downloads the bdb je jar into qpid/java/lib/bdbstore, and then creates a broker binary release in qpid/java/broker/release which includes the additional store module. You can make the BDB je jar available to the broker by creating a lib\opt subdir and copying the je jar into it, where it will get picked up automatically assuming you are using Java 6+. You can then use org.apache.qpid.server.store.berkeleydb.BDBMessageStore as the store class config instead of the other stores. Robbie On 24 October 2011 16:25, vipun vinay.punno...@gmail.com wrote: Hi, I'm collecting performance figures for QPID Java based broker. The results which i got after running the QpidBench program are a little lower than expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a message throughput of around 400 messages when both producer and consumer client instances are active. Qpid Java broker is configured to run over Derby and messaging is in persistent mode. I was expecting somewhere around 1000 atleast going by the following blog which does comparisons between different messaging providers. http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/ Do you think, the figures from my tests are correct, or what are the expected performance results, or are there any tweaks which need to be done for performance gains. I am running out of trunk. Thanks Regards Vinay -- View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org
