Thanks for writing Robbie. That was a lot of useful info. I sure didn't think of the multiple consumer - single session case. Thanks for the tip.
I'll try spreading by consumers across 2 connections and see if that helps. Thank you, Praveen On Fri, Jan 13, 2012 at 10:10 AM, Robbie Gemmell <[email protected]>wrote: > Hi Praveen, > > 180 connections should also be entirely fine, I have used several > hundred. The only limit is going to be memory based.The key issue with > using large amounts of connections currently is that when we moved > away from Mina as the IO layer for 0.14 we moved to a model that uses > 2 threads per connection, so there are a large number of Threads > created which in turn will require memory for their stacks and heap > for the objects they contain. Additionally, there are currently some > large buffers placed in the heap on a per-connection basis that can > quickly mount up. We are going to look at the impact those have on > performance at particular buffer sizes and see if we can reduce the > default, but it can certainly be reconfigured to meet a particular > users requirements. > > With such a low throughput of <=1msg/s per consumer, there is vastly > reduced scope for write-combining the store updates and so the use of > a single connection becomes far less of issue. There will be scope to > write-combine with other consumer instances and also the various > producers, but it may still be worth looking at something inbetween > the extremes of 20 Consumers per Connection and 1 Consumer per > Connection (eg 10 Consumers per Connection), to see if it is possible > to extract some scalability. > > With your slow onMessage handler the thing you should be most worried > about is avoiding using multiple Consumers on the same Session, > because each Session only has a single thread performing the > deliveries. You will want to make sure you use a Session per Consumer. > > Robbie > > On 11 January 2012 19:02, Praveen M <[email protected]> wrote: > > ah, my bad... I expect to have 20 consumer threads per instance. Sorry, > so > > that'd be 600 consumer threads in the environment. > > > > On Wed, Jan 11, 2012 at 10:14 AM, Praveen M <[email protected]> > wrote: > > > >> Great, that helps. > >> > >> Would you have a similar recommendation for the consumer side? > >> > >> In my environment, I expect about 200 consumer threads per instance. > Total > >> (30 instance * 200 threads/per instance) - 6000 consumer threads > >> > >> I was thinking I'd just use one connection per instance and then spin > off > >> 200 sessions/consumers. > >> > >> my onMessage() callback is expected to take atleast 1 second to process > (I > >> don't see a reason why a message should be submitted for async > processing > >> if it takes lesser than that to process) > >> , so I assumed I must be ok with one connection. > >> > >> A latency is totally acceptable before a message gets picked up. > >> Is there any recommendations towards connection management per > >> instance for the consumer side of things? I'd like to hear what you > think > >> about it. If I'm right to assume 1 connection per > >> instance should be good? > >> > >> Also, > >> Since the consumer side would add more parallel connections to the > broker, > >> I'm curious upon the limits I could push the broker. > >> Is there any tested limit in terms of the number of connections that the > >> broker could support? > >> We are looking at 180 concurrent connections to the broker if I say 1 > >> connection per instance for all the consumers and 5 connections per > >> instance for enqueues Should that be ok? > >> > >> Also, are there any cons for having a lot of connections to the broker > >> which I should be aware of? > >> > >> Thanks a lot, > >> Praveen > >> > >> > >> On Wed, Jan 11, 2012 at 9:52 AM, Robbie Gemmell < > [email protected]>wrote: > >> > >>> 150 connections should be no problem. > >>> > >>> I would expect any locking on the client to be of little significance > >>> to overall performance when using persistent messages, hitting the > >>> disk is going to dominate overall performance. The issue with using > >>> multiple Sessions on a Connection with persistent messages is that it > >>> the Sessions are ultimately multiplexed onto the same TCP connection > >>> and their data processed sequentially on arrival at the broker. If for > >>> example you were to do 5 concurrent sends on 5 Sessions on the same > >>> Connection, they will be processed onto the connection sequentially > >>> and thus into the store sequentially upon arrival at the broker, > >>> whereas if you do 5 concurrent sends for Sessions on individual > >>> Connections there is scope for them to be processed into the store > >>> concurrently and enable some/all of them to be write-combined and > >>> synced to disk at the same time, reducing the required synchronous > >>> disk IO to perform the operations and thus providing increased overall > >>> performance/scalability by effectively 'doing more with less' so to > >>> speak. > >>> > >>> Robbie > >>> > >>> On 11 January 2012 16:24, Praveen M <[email protected]> wrote: > >>> > Hi Robbie, > >>> > > >>> > Thanks for writing. I just had a few followup questions. Please see > >>> in-line. > >>> > > >>> > Thanks a lot for helping out with this, > >>> > Praveen > >>> > > >>> > On Wed, Jan 11, 2012 at 6:58 AM, Robbie Gemmell < > >>> [email protected]>wrote: > >>> > > >>> >> Hi Praveen, > >>> >> > >>> >> If you are predominantly going to use persistent messages, then I > >>> >> would actually look at your publishers using a single single session > >>> >> per connection for the most part. > >>> >> > >>> >> The broker is currently able to best extract performance scaling > from > >>> >> the persistent stores (more so the BDBstore as per Robs recent > >>> >> testing...though the results of which are again slightly out of date > >>> >> now due to the upgrade to BDB5 earlier today) by performing actions > on > >>> >> a per-connection basis, so it can extract parrallism over commits to > >>> >> the store. Using a lower number of connections with 5-10 sessions > each > >>> >> could be noticably slower at peak when using 150-300 publishers. > >>> >> > >>> > > >>> > that totally makes sense. But now that I'd have approximately (30 > >>> instance > >>> > * 5 connections) = 150 > >>> > connections to the broker, I wouldn't be overwhelming the broker > right? > >>> I'm > >>> > guessing it shouldn't be a problem, But just want to run it by you? > >>> > > >>> > I understand that having separate connections is significantly > faster as > >>> > each > >>> > connection would operate on it's own socket. > >>> > Just out of curiosity I'd like to know, what are the performance > >>> > degradation > >>> > causes when using sessions sharing the same connection? I know for > one > >>> that > >>> > the session has to hold on creation locks (for creation) and > connection > >>> > failover locks on send. > >>> > Do you think this will be the cause of the major bottle necks or is > >>> there > >>> > something else too? > >>> > (I'm sure you know the code back to front to answer this better) :) > >>> > > >>> > > >>> > > >>> >> If you are sending to a huge variety of Destinations then the way > you > >>> >> are using your Producers seems appropriate; creating producers isnt > >>> >> necessarily that heavyweight an operation, but not doing so is > >>> >> undeniably far more efficient. > >>> >> > >>> > > >>> > ah, yep. that makes sense. I think I will stick to creating > producers on > >>> > connection pooling > >>> > and creating the session > >>> > > >>> > > >>> >> > >>> >> I have actually never personally used sync_publish, but given that > >>> >> both the underlying sync() call and session.commit() hit the broker > >>> >> synchronously I would guess their performance should be pretty > similar > >>> >> for persistent messages with a batch size of 1, in which case I > would > >>> >> probably just go with transactions for 'pure JMS' purposes. If you > >>> >> ever were using batches of >1 message that would seem to favor use > of > >>> >> transactions due to lowering the number of synchronous round trips > to > >>> >> the broker which would be required, increasing performance. > >>> >> > >>> >> yep, I totally agree with you on using transactions over the > >>> sync_publish > >>> > option > >>> > for pure JMS purposes. > >>> > > >>> > > >>> >> Robbie > >>> >> > >>> >> On 10 January 2012 17:48, Praveen M <[email protected]> > wrote: > >>> >> > Hi Robbie, > >>> >> > > >>> >> > Thanks for writing. > >>> >> > > >>> >> > Here is some more context for my 2nd question. > >>> >> > > >>> >> > I'm expected to have about 30 client instances connecting to one > >>> broker. > >>> >> > About 90% of the messages to the broker are expected to be > persistent > >>> >> > messages. > >>> >> > > >>> >> > I haven't decided whether to use transactions or use the option > >>> >> > sync_publish. The > >>> >> > guarantee that we'd expect is that the broker receives the message > >>> before > >>> >> > proceeding to > >>> >> > send the next message. I'd be happy to hear your recommendations > >>> around > >>> >> > this. > >>> >> > > >>> >> > From the code I read, it looked like sync_publish was working to > that > >>> >> > effect. But one thing I > >>> >> > saw was the sync() call in the client held to a connection > failover > >>> lock, > >>> >> > and I wasn't sure if the sync() call was blocking. > >>> >> > Anyways, that said do you have a recommendation between > transacted vs > >>> >> > sync_publish in a multi client environment (things going in > >>> parallel). > >>> >> I'd > >>> >> > really like to hear what you think about this. > >>> >> > > >>> >> > There will not be any batch enqueues (a top of my head). Almost > all > >>> our > >>> >> > traffic will be having messages > >>> >> > enqueued one at a time. > >>> >> > > >>> >> > The number of destinations we have could be about 4000-5000. But > yes, > >>> >> there > >>> >> > is a strong likelihood of one > >>> >> > queue being more popular and all the traffic getting routed > there. We > >>> >> > sometimes receive bursty enqueues of a certain type > >>> >> > and is fair to assume that 5 or more enqueues of the same type to > the > >>> >> same > >>> >> > queue is possible to happen concurrently. > >>> >> > > >>> >> > We plan to have own framework on top of Qpid which will have the > >>> ability > >>> >> to > >>> >> > load balance between the destinations > >>> >> > if one queue goes over a certain threshold (some hand-picked > >>> number..not > >>> >> > decided on yet) and then route traffic to another destination. > >>> >> > > >>> >> > And the overall traffic expected is somewhere between 10 million > - 20 > >>> >> > million messages a day maybe. > >>> >> > > >>> >> > Please do let me know if you'd like to hear any other information. > >>> >> > > >>> >> > Thank you, > >>> >> > Praveen > >>> >> > > >>> >> > > >>> >> > But yes, we'd be using one of the two (whichever would perform > best > >>> with > >>> >> > multiple concurrent clients). > >>> >> > > >>> >> > On Tue, Jan 10, 2012 at 6:41 AM, Robbie Gemmell < > >>> >> [email protected]>wrote: > >>> >> > > >>> >> >> Hi Praveen, > >>> >> >> > >>> >> >> Using either JNDI or Session.createQueue should work fine, but > the > >>> >> >> obvious limitation of the latter is that the (possibly > >>> >> >> provider-specific) configuration is then part of your code > (unless > >>> you > >>> >> >> look the information up somewhere before using it) and cant be > >>> changed > >>> >> >> without modifying the code. > >>> >> >> > >>> >> >> To better answer the other question, it would be good to first > know > >>> >> >> more specifics about the application and its messaging model, eg: > >>> >> >> How many instances of the application are you likely to be > running? > >>> >> >> Will all your messages be persistent? > >>> >> >> Will you always use transactions (and if so will it always of > batch > >>> >> >> size 1 like below)? > >>> >> >> You mention having maybe 5-10 concurrent publishers, but how many > >>> >> >> different Destinations are you likely to be sending to in total? > >>> >> >> Are any of the above Destinations likely to be used a lot more > than > >>> the > >>> >> >> others? > >>> >> >> > >>> >> >> Robbie > >>> >> >> > >>> >> >> On 10 January 2012 00:27, Praveen M <[email protected]> > >>> wrote: > >>> >> >> > Hi, > >>> >> >> > > >>> >> >> > I'm writing a JMS Client and have a few best practices > >>> questions. > >>> >> >> > > >>> >> >> > 1) I understand that there are different ways ways to create a > >>> >> reference > >>> >> >> to > >>> >> >> > a destination to enqueue a message (using JMS createQueue, > using > >>> JNDI > >>> >> >> > lookup). > >>> >> >> > What would be the recommended way to use? > >>> >> >> > > >>> >> >> > 2) I create a connection to the broker on my application > startup, > >>> and > >>> >> >> > create a session pool. on every enqueue, i checkout a session > >>> from my > >>> >> >> pool > >>> >> >> > use it and return to pool on enqueue. > >>> >> >> > Does this sound reasonable? (code sample below) > >>> >> >> > > >>> >> >> > What would be the pattern that is typically used/recommended > >>> for > >>> >> >> > session management/connection management in the client side? > >>> >> >> > > >>> >> >> > Or is this an overkill to do? Is CreateSession() and > >>> >> CreateProducer() > >>> >> >> > really cheap that it is just done on each enqueue? > >>> >> >> > In my use case, I'd expect quite some concurrency (assume n > >>> (maybe > >>> >> >> > 5-10) threads doing enqueues at the same time) and I'd prefer > it > >>> to > >>> >> not > >>> >> >> > lock/block and go through asap. > >>> >> >> > > >>> >> >> > I do remember from reading the qpid client code that there > is a > >>> >> lock > >>> >> >> > held on the connection object whenever a new session is > created on > >>> >> that > >>> >> >> > connection. I'm assuming it is one of the places where my > enqueue > >>> can > >>> >> >> slow > >>> >> >> > down > >>> >> >> > when going in parallel if it has to create a session for > every > >>> >> enqueue > >>> >> >> > of a message. There might be more such blocks, which I'm not > >>> aware of. > >>> >> >> > > >>> >> >> > Taking performance into mind, what would be the best design > for > >>> >> >> > session/connection management? > >>> >> >> > > >>> >> >> > A snippet of what I have currently is below, > >>> >> >> > > >>> >> >> > // a thread safe queue with the available session producer > >>> pairs > >>> >> which > >>> >> >> > can be used to create a message and enqueue. > >>> >> >> > ConcurrentLinkedQueue<Pair<Session, MessageProducer>> > >>> >> >> > availableSessionProducerPairs = new > >>> >> ConcurrentLinkedQueue<Pair<Session, > >>> >> >> > MessageProducer>> (); > >>> >> >> > > >>> >> >> > // initialize the sessionProducerPairs, create upto n pairs > >>> >> >> > initializePool(Connection qpidConnection) { > >>> >> >> > for (int i = 1 ; i <= SESSION_PRODUCER_PAIRS_COUNT; i > ++ > >>> ) { > >>> >> >> > Session session = qpidConnection.createSession(true, > >>> >> >> > Session.SESSION_TRANSACTED); > >>> >> >> > MessageProducer producer = > >>> session.createProducer(null); > >>> >> >> > > availableSessionProducerPairs.add(Pair.newPair(session, > >>> >> >> > producer)); > >>> >> >> > } > >>> >> >> > } > >>> >> >> > > >>> >> >> > // on enqueue we checkout a session/producer pair use it for > the > >>> >> enqueue > >>> >> >> > and then return it back to the session/produer pair pool. > >>> >> >> > enqueueMessage(String queueName, byte[] message) { > >>> >> >> > // remove the session producer pair > >>> >> >> > Pair<Session, MessageProducer> sessionProducerPair > = > >>> >> >> > availableSessionProducerPairs.poll(); > >>> >> >> > if(sessionProducerPair == null) { > >>> >> >> > // gack here and return null. > >>> >> >> > } > >>> >> >> > Session session = sessionProducerPair.getFirst(); > >>> >> >> > MessageProducer producer = > sessionProducerPair.getSecond(); > >>> >> >> > BytesMessage jmsMessage = session.createBytesMessage(); > >>> >> >> > jmsMessage.writeBytes(message); > >>> >> >> > Queue queueRef = queueNameToReferenceMap.get(queueName); > >>> >> >> > producer.send(queueRef, jmsMessage, > >>> DeliveryMode.PERSISTENT, 0, > >>> >> >> 0); > >>> >> >> > session.commit(); > >>> >> >> > availableSessionProducersPair.add() > >>> >> >> > } > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > Thank you, > >>> >> >> > -Praveen > >>> >> >> > >>> >> >> > >>> --------------------------------------------------------------------- > >>> >> >> Apache Qpid - AMQP Messaging Implementation > >>> >> >> Project: http://qpid.apache.org > >>> >> >> Use/Interact: mailto:[email protected] > >>> >> >> > >>> >> >> > >>> >> > > >>> >> > > >>> >> > -- > >>> >> > -Praveen > >>> >> > >>> >> > --------------------------------------------------------------------- > >>> >> Apache Qpid - AMQP Messaging Implementation > >>> >> Project: http://qpid.apache.org > >>> >> Use/Interact: mailto:[email protected] > >>> >> > >>> >> > >>> > > >>> > > >>> > -- > >>> > -Praveen > >>> > >>> --------------------------------------------------------------------- > >>> Apache Qpid - AMQP Messaging Implementation > >>> Project: http://qpid.apache.org > >>> Use/Interact: mailto:[email protected] > >>> > >>> > >> > >> > >> -- > >> -Praveen > >> > > > > > > > > -- > > -Praveen > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] > > -- -Praveen
