Hi Flavio, Wonder if the 'dispatcher' thread pool on the broker is too small; When architecting our system, we understood the dispatch is round robin from the broker side. If the broker thread pool has too many 'queues' to visit before doing the next dispatch, it might become the bottleneck. Just a thought, I haven't done much digging here..
Thanks Vijay On Tue, Nov 12, 2013 at 9:30 AM, Flavio Baronti <[email protected]>wrote: > Hi Vijay, > > thanks for the suggestion. The broker seems stable from the point of view > of memory, unless I send more messages than the client can receive; in that > case messages start to pile up in queues, and memory usage goes up. > Memory usage on the client is negligible in all cases. > > Flavio > > > > Il 2013/11/12 18:08 PM, Vijay Devadhar ha scritto: > > Hi Flavio, >> >> How healthy does the heap look on the JVM that is running the JMS clients? >> We had done a prototype of something like this with java broker, and >> throughput drop was mostly attributed to running out of heap space on >> broker. This might still happen in C++ broker, however I don't have >> recent >> experience in C++ memory management. Looking at memory on both client and >> server is a good option. >> >> We were heading in the direction of 100k queues, but when we saw this >> changed our application design to live with smaller number of queues. >> >> Thanks >> Vijay >> >> >> On Tue, Nov 12, 2013 at 8:56 AM, Flavio Baronti <[email protected] >> >wrote: >> >> Hi Rob, >>> >>> thanks for your interest. I'm using the C++ broker on Linux, and the Java >>> client on windows. >>> The sender is sending through producers obtained with >>> MessageProducer topicProducer = session.createProducer( >>> session.createTopic(topicName >>> + "; {node: { type: topic } }")); >>> while the receiver is receiving with a listener set on consumers obtained >>> with >>> MessageConsumer consumer = session.createConsumer( >>> session.createTopic(topicName)); >>> I don't know what to gather from CPU usage. With few sessions it's higher >>> on the broker, and lower on the client. >>> Increasing the number of session it's the other way round, higher on >>> client and lower on broker. >>> In both cases we're very far from 100%. >>> >>> I created a bug report, I'm not too familiar with JIRA so I hope I did it >>> ok. >>> I added the two test programs I'm using. If there is anything else I can >>> help with, please let me know. >>> >>> Flavio >>> >>> Il 2013/11/12 10:36 AM, Rob Godfrey ha scritto: >>> >>> Hi Flavio, >>>> >>>> a few questions so we can help investigate your issue: >>>> >>>> are you using the Java or the C++ broker? >>>> how are you creating the 100k topics/queues? >>>> when you run your test is it obvious whether the broker or the client is >>>> the bottleneck - e.g. is one of these using significantly more CPU time >>>> than the other? >>>> would it be possible to provide for you to provide your test code so >>>> that >>>> we can try to replicate the issue (e.g. by attaching to a JIRA)? >>>> >>>> thanks, >>>> Rob >>>> >>>> >>>> On 8 November 2013 12:18, Flavio Baronti <[email protected]> >>>> wrote: >>>> >>>> Hello, >>>> >>>>> I have an use case where I need to create hundreds of thousands of >>>>> queues, >>>>> each one subscribed to a different topic (therefore I have as many >>>>> topics >>>>> as queues). >>>>> I set up a test with a single producer generating data on a randomly >>>>> chosen topic, and a receiver retrieving data from the queues (and >>>>> throwing >>>>> it away). >>>>> >>>>> I'm using the JMS api, and doing the obvious thing makes the throughput >>>>> drop dramatically from 10k msg/sec with a single topic/queue (around >>>>> the >>>>> top my network adapter can sustain) to 20 msg/sec with 100k >>>>> topics/queues. >>>>> I found out that I can recover performance by using more JMS sessions >>>>> and >>>>> connections - e.g. create 4 connections with 100 sessions each, and >>>>> randomly distributing the receiving queues on them. >>>>> This however is less than ideal, since with the JMS client a thread is >>>>> created for each session, and I definitely don't want 400 threads >>>>> receiving >>>>> data. The work I have to do is CPU-bound, and I don't want to waste >>>>> time >>>>> in >>>>> context switching when 2/4 threads can suffice. >>>>> >>>>> Why does the throughput drop so badly with many topics/queues? Why >>>>> adding >>>>> sessions helps? Am I overlooking something, or doing something wrong? >>>>> >>>>> Thanks >>>>> Flavio >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
