Marnie, Rajith thanks a lot. We try to omit to use transactions because of performance penalty. To achieve the maximum performance and have our messages guaranteed delivered we have the following configuration:
1. cluster of two qpid brokers 2. all queues are cluster-durable 3. all messages are non-persistent (once qpid cluster collapses to a single broker all messages will become durable due to cluster-durable queues) 4. c++ code publishes messages 5. java code receives messages and uses client_ack mode I would say in that case we can achieve maximum performance having all the messages guaranteed delivered. What do you think? Do we have to publish messages synchronously in our c++ code? Best Regards, Sergey Zhemzhitsky Rajith Attapattu <[email protected]> 23.03.2011 23:21 Please respond to [email protected] To Marnie McCormack <[email protected]>, [email protected] cc Subject Re: Qpid CLIENT_ACKNOWLEDGE Marnie, Looks like you forgot to hit replyAll. So I am forwarding your email to the list along with my comments. On Wed, Mar 23, 2011 at 12:47 PM, Marnie McCormack < [email protected]> wrote: > Sorry, which broker are you usiong Sergey ? I'd assumed the Java broker ... > Since the Java broker supports 0-10 now, you could use sync_publish. > > I'd probably disagree that you can achieve guaranteed delivery without a > transaction or its equivalent (which sync publish effectively is ?). > Synchronous publishing is not equivalent to transactions, but I understand what you meant here. You were probably alluding to the fact that in both cases messages are published in a synchronous way. i.e When the tx completes (or when the send method returns in sync-publish) you know that the broker has acked the message. However transactions have a lot more overhead than a simple sync publish, hence a higher performance penalty with transactions. For example look at the following relative numbers for a producer/consumer pair using the same setup. sync publish 15k msg/sec tx size =2 48 msg/sec (thats right just 48 messages per sec) tx size=1000 14k msg/sec Also from an application point of view, non transactional code is more simple and easy to write than transactional code. I would normally stay away from transactions unless I really have to. It basically negates a lot of benefits of using messaging. Regards, Rajith > > Regards, > Marnie > > On Wed, Mar 23, 2011 at 4:21 PM, Rajith Attapattu <[email protected]>wrote: > >> Hi Sergey, >> >> For Qpid to guarantee 'at-least-once' delivery on publish, you don't need >> to use transactions. >> >> Broker failures >> ----------------- >> (1) In a stand-alone-broker you need to use persistence to ensure that >> messages survive a broker crash. >> (2) In a clustered broker you may be able to get away without using >> persistence (*) >> >> (*) Only the c++ broker supports clustering. >> If you don't use persistence, then in the event of a total cluster >> failure you will loose messages. >> Therefore it depends on how much you can tolerate these occasional >> failures. >> >> Application/Client lib failures >> --------------------------------- >> In the event of a failure in the application that sends the messages, you >> may loose messages. >> i.e if there are unacked messages in the client libs replay buffer, they >> will be lost. >> You could avoid that by using >> >> For JMS client >> ------------------- >> (1) Using synchronous publishing. Use >> -Dsync_publish={persistent|transient|all}. >> (2) Transaction - as Marnie pointed out. >> >> Performance wise there isn't much of a difference between option 1 & 2 if >> you use large batches (ex 1000 msg/batch) for transactions. >> But ff you use small tx batches then there is an appreciable difference in >> performance. >> >> Therefore I recommend you use -Dsync_ack=persistent. >> Also the application code is much simpler when writing non transactional >> code. >> >> For other clients >> -------------------- >> If you can handle reliable replay at the application level then you could >> just use asynchronous publishing without worrying about application failure. >> Ex. If your application is retrieving orders from a queue and processing >> it before sending a reply, then you could ack the order messsage after the >> broker acks your reply message. >> >> Hope this helps. >> >> Rajith >> >> >> On Wed, Mar 23, 2011 at 11:21 AM, Marnie McCormack < >> [email protected]> wrote: >> >>> Hi Sergey, >>> >>> To achieve guaranteed message delivery you need to be using transactions >>> on >>> *publish*. Qpid then guarantees 'at least once' delivery. You'd offset >>> the >>> reliable delivery cost against the cost or message loss during a failure >>> from sending non-transacted messages. The use of transactions and >>> persistence for reliable messaging adds cost in the shape of I/O. >>> >>> However, you could opt not to use transactions on consume since you >>> cannot >>> conceptually lose a message during consumption - if there's a failure and >>> a >>> message ack doesn't get back to the broker then you'll simply get the >>> message back again on restart. This assumes that you listen/catch/handle >>> errors and exception on consume such that you don't have application >>> level >>> side effects. >>> >>> Regards, >>> Marnie >>> >>> On Tue, Mar 22, 2011 at 1:03 PM, <[email protected]> wrote: >>> >>> > Hi there >>> > >>> > I'm working with qpid 0.8 by means of JMS API and I'm using >>> > CLIENT_ACKNOWLEDGE mode to get higher performance comparing to >>> > transactinal processing. >>> > As a rule performance is rather pure in both cases. In transactional >>> mode >>> > I can receive about 1.6K messages per second, in CLIENT_ACKNOWLEDGE >>> mode >>> > I'm able to receive about 3.6K messages per second and in >>> AUTO_ACKNOWLEDGE >>> > mode I'm able to receive about 70K messages per second. >>> > >>> > Are there any recommendations how to improve performance and get a >>> > guaranted message delivery? >>> > >>> > >>> > Best Regards, >>> > Sergey >>> > _______________________________________________________ >>> > >>> > The information contained in this message may be privileged and conf >>> > idential and protected from disclosure. If you are not the original >>> intended >>> > recipient, you are hereby notified that any review, retransmission, >>> > dissemination, or other use of, or taking of any action in reliance >>> upon, >>> > this information is prohibited. If you have received this communication >>> in >>> > error, please notify the sender immediately by replying to this message >>> and >>> > delete it from your computer. Thank you for your cooperation. Troika >>> Dialog, >>> > Russia. >>> > If you need assistance please contact our Contact Center (+7495) 258 >>> 0500 >>> > or go to www.troika.ru/eng/Contacts/system.wbp >>> > >>> > >>> >> >> > _______________________________________________________ The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia. If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
