Re: ApacheCon US 2008 New Orleans
Hi devs, I did the presentation last afternoon, first thanks Carl and Rajith to giving me this idea of giving an incubator talk at ApacheCon and preparing slides for me. It was a really interesting one people had discussion about AMQP and Qpid about 7 or 8 minutes and there was Paul Frementle and that really benefited to do some marketing over Qpid. Thanks Paul, for the help and appreciate it a lot. Cheers Lahiru On Wed, Nov 5, 2008 at 5:55 PM, lahiru gunathilake <[EMAIL PROTECTED]>wrote: > Ok Aidan. will do. > > Thanks > Lahiru > > > On Wed, Nov 5, 2008 at 1:37 PM, Aidan Skinner <[EMAIL PROTECTED]> wrote: > >> On Tue, Nov 4, 2008 at 11:30 AM, lahiru gunathilake <[EMAIL PROTECTED]> >> wrote: >> >> > gave a very positive comment on the community :-). I will be giving an >> > incubator talk to promote Apache Qpid tomorrow. >> >> Awesome. If you have reusable material it'd be good to get it up on >> the website. :) >> >> - Aidan >> >> -- >> Apache Qpid - World Domination through Advanced Message Queueing >> http://cwiki.apache.org/qpid >> "Nine-tenths of wisdom consists in being wise in time." - Theodore >> Roosevelt >> > >
Re: ApacheCon US 2008 New Orleans
Ok Aidan. will do. Thanks Lahiru On Wed, Nov 5, 2008 at 1:37 PM, Aidan Skinner <[EMAIL PROTECTED]> wrote: > On Tue, Nov 4, 2008 at 11:30 AM, lahiru gunathilake <[EMAIL PROTECTED]> > wrote: > > > gave a very positive comment on the community :-). I will be giving an > > incubator talk to promote Apache Qpid tomorrow. > > Awesome. If you have reusable material it'd be good to get it up on > the website. :) > > - Aidan > > -- > Apache Qpid - World Domination through Advanced Message Queueing > http://cwiki.apache.org/qpid > "Nine-tenths of wisdom consists in being wise in time." - Theodore > Roosevelt >
Re: ApacheCon US 2008 New Orleans
On Tue, Nov 4, 2008 at 11:30 AM, lahiru gunathilake <[EMAIL PROTECTED]> wrote: > gave a very positive comment on the community :-). I will be giving an > incubator talk to promote Apache Qpid tomorrow. Awesome. If you have reusable material it'd be good to get it up on the website. :) - Aidan -- Apache Qpid - World Domination through Advanced Message Queueing http://cwiki.apache.org/qpid "Nine-tenths of wisdom consists in being wise in time." - Theodore Roosevelt
Re: ApacheCon US 2008 New Orleans
lahiru gunathilake wrote: Hi Devs, I'm there in ApacheCon now and I had an interview with a Researcher called U. Yeliz Eseryel from Syracuse University School of Information Studies. I gave a very positive comment on the community :-). I will be giving an incubator talk to promote Apache Qpid tomorrow. Regards Lahiru Great, thanks for promoting qpid, Carl.
Re: ApacheCon Asia 2006 - performance figures for Qpid
You said "You don't have to use VS to do in-memory replication. The primary can send messages to the backup and when the transaction commits it can get an ack from the backup. Then the transaction can be forced to disk later." I like that model. People like solutions they 'think' they understand, and that is something people think they understand... Thanks for the book reference - I've got some reading to do! John On 22/02/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > I've always been slightly wary of the tight/intimate type of cluster that > VS > solutions end up giving. I'm more of a fan of the first part of your > reply: > > If you replicate the messages in memory (to another machine or two) you do >> not have to force to disk upon commit - you can thus take the disk out >> of >> the critical path. Therefore you can get great throughput even with >> short >> transactions and short messages. >> > > > Two server solutions usually seem more robust than 3+ server solutions; > just > my experience of the Real World. Usually people don't invest enough on > the > infrastructure to make N-way solutions fail deterministically (e.g. they > skimp on network connections and multi-pathing). You don't have to use VS to do in-memory replication. The primary can send messages to the backup and when the transaction commits it can get an ack from the backup. Then the transaction can be forced to disk later. But the reasons I would use the totem protocol in this application are: 1) Using totem it's easy to do active/active, which seems more desirable than primary-backup. Deadlocks are not a problem because all servers see the same messages, and all messages arrive with a globally unique id. 2) There is a complex relationship between the degree of replication and availability, and the optimal availability is probably more than 2 servers. It's interesting what you say about network availability, but VS is based on the concept that nodes can drop off the group at any time, so network availability need not be perfect. 3) Totem provides a really easy programming model, which makes the code more likely to be correct early on, which itself has an impact on availability. > I am interested in how we could add VS to Qpid/Java though. Could you > point > us to a "Dummy's Guide" for this application of VS (obviously using EVS4J > :-) We don't have the brainwidth to be expert in VS in addition to > working > with AMQP and Qpid. You could read the section called "Advanced Replication Techniques", in Ken Birman's book "Building Secure and Reliable Network Applications." It's available for free on the web. It references some other sections so you may have to go back and look at those. Birman uses causal ordering, not total ordering. And of course you can the read the totem article referenced on the evs4j web page. With EVS4J what you need to know is you get an api that allows you to send a message. Each member gets the message, including the sender, and when the message arrives it arrives in the same order at each member. When a fault occurs, you get notified of the membership change. And you get pretty good throughput.
Re: ApacheCon Asia 2006 - performance figures for Qpid
> I've always been slightly wary of the tight/intimate type of cluster that > VS > solutions end up giving. I'm more of a fan of the first part of your > reply: > > If you replicate the messages in memory (to another machine or two) you do >> not have to force to disk upon commit - you can thus take the disk out >> of >> the critical path. Therefore you can get great throughput even with >> short >> transactions and short messages. >> > > > Two server solutions usually seem more robust than 3+ server solutions; > just > my experience of the Real World. Usually people don't invest enough on > the > infrastructure to make N-way solutions fail deterministically (e.g. they > skimp on network connections and multi-pathing). You don't have to use VS to do in-memory replication. The primary can send messages to the backup and when the transaction commits it can get an ack from the backup. Then the transaction can be forced to disk later. But the reasons I would use the totem protocol in this application are: 1) Using totem it's easy to do active/active, which seems more desirable than primary-backup. Deadlocks are not a problem because all servers see the same messages, and all messages arrive with a globally unique id. 2) There is a complex relationship between the degree of replication and availability, and the optimal availability is probably more than 2 servers. It's interesting what you say about network availability, but VS is based on the concept that nodes can drop off the group at any time, so network availability need not be perfect. 3) Totem provides a really easy programming model, which makes the code more likely to be correct early on, which itself has an impact on availability. > I am interested in how we could add VS to Qpid/Java though. Could you > point > us to a "Dummy's Guide" for this application of VS (obviously using EVS4J > :-) We don't have the brainwidth to be expert in VS in addition to > working > with AMQP and Qpid. You could read the section called "Advanced Replication Techniques", in Ken Birman's book "Building Secure and Reliable Network Applications." It's available for free on the web. It references some other sections so you may have to go back and look at those. Birman uses causal ordering, not total ordering. And of course you can the read the totem article referenced on the evs4j web page. With EVS4J what you need to know is you get an api that allows you to send a message. Each member gets the message, including the sender, and when the message arrives it arrives in the same order at each member. When a fault occurs, you get notified of the membership change. And you get pretty good throughput.
Re: ApacheCon Asia 2006 - performance figures for Qpid
You seem to know rather too much about this, so I Googled you :-) and found out you wrote EVS4J - Kudos! I've always been slightly wary of the tight/intimate type of cluster that VS solutions end up giving. I'm more of a fan of the first part of your reply: If you replicate the messages in memory (to another machine or two) you do not have to force to disk upon commit - you can thus take the disk out of the critical path. Therefore you can get great throughput even with short transactions and short messages. Two server solutions usually seem more robust than 3+ server solutions; just my experience of the Real World. Usually people don't invest enough on the infrastructure to make N-way solutions fail deterministically (e.g. they skimp on network connections and multi-pathing). I am interested in how we could add VS to Qpid/Java though. Could you point us to a "Dummy's Guide" for this application of VS (obviously using EVS4J :-) We don't have the brainwidth to be expert in VS in addition to working with AMQP and Qpid. Many thanks for sharing your wisdom with us. John
Re: ApacheCon Asia 2006 - performance figures for Qpid
> You see greater performance if you commit in batches rather than after > each > message, as this means fewer disk synchs... Similarly if the message size > is larger then you get greater Kb/s throughput. If you replicate the messages in memory (to another machine or two) you do not have to force to disk upon commit - you can thus take the disk out of the critical path. Therefore you can get great throughput even with short transactions and short messages. With the totem protocol you can create, say, 3 copies of all messages with a few milliseconds latency and a throughput of 25,000 messages per second that's with 1423-byte messages.
Re: ApacheCon Asia 2006 - performance figures for Qpid
OK, attached (Open Office Calc format) is the set of performance number that I did over the weekend. These numbers are from NAS storage, the faster number I mentioned above (1100 Msg/s) is from SAN - I haven't yet had time to retest against the SAN but I think these numbers are interesting in themselves. To explain the numbers a little, each line represents a test where a number (given in the Queues column) of Sender/receiver pairs are started, each on a separate queue. Each pair sends/receives 1000 messages (so if there are 8 queues then 8000 messages are sent). The Msg/s and Kb/s are totals across all pairs. You see greater performance if you commit in batches rather than after each message, as this means fewer disk synchs... Similarly if the message size is larger then you get greater Kb/s throughput. These numbers are Java Client -> Jave Broker -> Java Client, using the BDB store. Would be interesting to try some other combinations. -- Rob On 20/02/07, John O'Hara <[EMAIL PROTECTED]> wrote: Not at all shabby. On 19/02/07, Robert Godfrey <[EMAIL PROTECTED]> wrote: > > The current 0-8 trunk can do slightly better that that now :-) Certainly > seen peek over 250,000 msg/s. > > Recently I've been testing persistent messages on the same hardware, using > point-to-point messaging (direct exchange) > > With 8 producers, and 8 receivers (all located on the same machine), 1Kb > message size, and each producer-receiver pair using a separate queue; I > see > about 1100 round-trip messages per second (that its 2200 total message > transfers). This is using transactions, with a commit after each publish > or > deliver. Better performance can be had by using larger commit batches. > > When I get time, I'll tabulate the results. > > > -- Rob > > On 19/02/07, Kim van der Riet <[EMAIL PROTECTED]> wrote: > > > > The following is from the Mina documentation page - it is a presentation > > given at ApacheCon Asia 2006. On page 7 is a performance benchmark - > > based on AMQP (and presumably Qpid!) > > > > http://mina.apache.org/documentation.data/ACAsia2006.pdf > > > > > > Advantages: Performance > > • AMQP Test > > • Client and server > > • 4 dual-core Opterons > > • Via Gigabits Ethernet > > • 10 clients > > • Payload: 256+ bytes (excl. AMQP headers) > > • Avg : 180,000 msg/sec > > • Max: 220,000 msg/sec > > Courtesy of Robert J. Greig, JP Morgan Chase & Co. > > > > > > Kim > > > > > Qpid Persistence Performance.ods Description: application/vnd.oasis.opendocument.spreadsheet
Re: ApacheCon Asia 2006 - performance figures for Qpid
Not at all shabby. On 19/02/07, Robert Godfrey <[EMAIL PROTECTED]> wrote: The current 0-8 trunk can do slightly better that that now :-) Certainly seen peek over 250,000 msg/s. Recently I've been testing persistent messages on the same hardware, using point-to-point messaging (direct exchange) With 8 producers, and 8 receivers (all located on the same machine), 1Kb message size, and each producer-receiver pair using a separate queue; I see about 1100 round-trip messages per second (that its 2200 total message transfers). This is using transactions, with a commit after each publish or deliver. Better performance can be had by using larger commit batches. When I get time, I'll tabulate the results. -- Rob On 19/02/07, Kim van der Riet <[EMAIL PROTECTED]> wrote: > > The following is from the Mina documentation page - it is a presentation > given at ApacheCon Asia 2006. On page 7 is a performance benchmark - > based on AMQP (and presumably Qpid!) > > http://mina.apache.org/documentation.data/ACAsia2006.pdf > > > Advantages: Performance > • AMQP Test > • Client and server > • 4 dual-core Opterons > • Via Gigabits Ethernet > • 10 clients > • Payload: 256+ bytes (excl. AMQP headers) > • Avg : 180,000 msg/sec > • Max: 220,000 msg/sec > Courtesy of Robert J. Greig, JP Morgan Chase & Co. > > > Kim > >
Re: ApacheCon Asia 2006 - performance figures for Qpid
The current 0-8 trunk can do slightly better that that now :-) Certainly seen peek over 250,000 msg/s. Recently I've been testing persistent messages on the same hardware, using point-to-point messaging (direct exchange) With 8 producers, and 8 receivers (all located on the same machine), 1Kb message size, and each producer-receiver pair using a separate queue; I see about 1100 round-trip messages per second (that its 2200 total message transfers). This is using transactions, with a commit after each publish or deliver. Better performance can be had by using larger commit batches. When I get time, I'll tabulate the results. -- Rob On 19/02/07, Kim van der Riet <[EMAIL PROTECTED]> wrote: The following is from the Mina documentation page - it is a presentation given at ApacheCon Asia 2006. On page 7 is a performance benchmark - based on AMQP (and presumably Qpid!) http://mina.apache.org/documentation.data/ACAsia2006.pdf Advantages: Performance • AMQP Test • Client and server • 4 dual-core Opterons • Via Gigabits Ethernet • 10 clients • Payload: 256+ bytes (excl. AMQP headers) • Avg : 180,000 msg/sec • Max: 220,000 msg/sec Courtesy of Robert J. Greig, JP Morgan Chase & Co. Kim
Re: ApacheCon
I will be there. Rajith On 8/30/06, Paul Fremantle <[EMAIL PROTECTED]> wrote: I'll be there. Paul On 8/30/06, Carl Trieloff <[EMAIL PROTECTED]> wrote: > > Yes, I will be there and Rajith will as he is speaking on Axis. > > Carl. > > Brian McCallister wrote: > > Any Qpid folks coming to ApacheCon in October? I'd love to buy you a > > beer or two ;-) > > > > -Brian > > -- Paul Fremantle VP/Technology, WSO2 and OASIS WS-RX TC Co-chair http://bloglines.com/blog/paulfremantle [EMAIL PROTECTED] "Oxygenating the Web Service Platform", www.wso2.com
Re: ApacheCon
> Any Qpid folks coming to ApacheCon in October? I'd love to buy you a beer or two ;-) I can't imagine making it this year. Maybe next year. More likely the european apache conference! Steve. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you.
Re: ApacheCon
I'll be there. Paul On 8/30/06, Carl Trieloff <[EMAIL PROTECTED]> wrote: Yes, I will be there and Rajith will as he is speaking on Axis. Carl. Brian McCallister wrote: > Any Qpid folks coming to ApacheCon in October? I'd love to buy you a > beer or two ;-) > > -Brian -- Paul Fremantle VP/Technology, WSO2 and OASIS WS-RX TC Co-chair http://bloglines.com/blog/paulfremantle [EMAIL PROTECTED] "Oxygenating the Web Service Platform", www.wso2.com
Re: ApacheCon
Yes, I will be there and Rajith will as he is speaking on Axis. Carl. Brian McCallister wrote: Any Qpid folks coming to ApacheCon in October? I'd love to buy you a beer or two ;-) -Brian
