The performance cost for clustering is heavy
I'm posting some of my test results in this email. The test rig is a 3 node qpid-0.12 cluster benched with the supplied perfest tools. All nodes are kvm 64bit virtual machines with 12G ram, 4cpu, virtio drivers, running lucid with 30G disk. Two nodes run on the same host, the third on a separate host. The network runs at Gigbit speeds and is switched. root@uk-mq1-test:~# /usr/lib/qpid-cpp/qpid/tests/qpid-perftest Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 1000000 transfers of 1024 bytes in 56.8549 seconds. Publish transfers/sec: 8800.31 Average: 8800.31 Subscribe transfers/sec: 8793.73 Average: 8793.73 Total transfers/sec: 17588.6 Total Mbytes/sec: 17.1764 -------------------------------------------------------------------- Vs non clustered setup root@uk-mq1-test:~# /usr/lib/qpid-cpp/qpid/tests/qpid-perftest Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 1000000 transfers of 1024 bytes in 11.4287 seconds. Publish transfers/sec: 50283.8 Average: 50283.8 Subscribe transfers/sec: 43611.7 Average: 43611.7 Total transfers/sec: 87498.8 Total Mbytes/sec: 85.4481 So there is a significant drop in throughput to the tune of 70K msgs/s just for the UDP corosync traffic. I managed to kill our network while stressing the cluster and using multicast transport ... it seems very hard to get reliable multicast in a data centre with mixed vendor switches (its fail open so the traffic is broadcast over the entire network doh) the unicast udp transport is more reliable in this regard. The bottle neck by my estimation was the corosync udp traffic (a 2node openvz co-tenanted test rig was observed to have a udp bandwidth of 4 times the physical network and much higher throughput) Clustering of qpid provides availability. You are not really buying extra capacity (in terms of through put or total in-flight messages). If you need to scale out I suspect (I have not adequately measured/tested federation as I ran out of time to get it going in my test setup) that queue replication and federation are the better bet. If you can live with the potential loss of messages in transit. Which is fairly small for durable queues plus messages sent with delivery properties of persistent and a suitable message store plugin. Also consider pushing some of the reliability into the application(s) rather than the infrastructure so message loss becomes acceptable. Queue replication test Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 1000000 transfers of 1024 bytes in 33.1998 seconds. Publish transfers/sec: 15107.2 Average: 15107.2 Subscribe transfers/sec: 15044.3 Average: 15044.3 Total transfers/sec: 30120.7 Total Mbytes/sec: 29.4147 *Note the above test is acking 50 msgs at a time, higher throughput may be achievable by choosing a value which better suits the workload throughput/latency requirements. I suggest putting together a test rig and attempt to measure if splitting the connections over the brokers or different topologies, gives you any wins. The qpid-perftest tool allows you to simulate this very easily. It supports starting a bunch of subscribers/publishers on different hosts connected to different brokers and control them via messaging (qpid-perftest --control). Cheers Gordon > -----Original Message----- > From: Zhemzhitsky Sergey [mailto:[email protected]] > Sent: 29 September 2011 01:17 > To: [email protected] > Subject: RE: QPID Cluster and load balancing > > Hello Gordon, > > Thanks a lot for the information. I suppose we will try to use a load > balancer to understand the difference between balanced and unbalanced > client connections to a qpid cluster. > > > Best Regards, > Sergey Zhemzhitsky > > > -----Original Message----- > From: Gordon Sim [mailto:[email protected]] > Sent: Thursday, September 29, 2011 11:49 AM > To: [email protected] > Subject: Re: QPID Cluster and load balancing > > On 09/28/2011 12:09 PM, Zhemzhitsky Sergey wrote: > > Hi Jakub, > > > > I understand that all the events are synchronized between all nodes > of the cluster. > > What I'd like to understand is: > > > > 1. Whether it is possible to distribute the connections between the > nodes of the cluster transparently for the clients of the cluster? > > 2. Does it make sense to distribute connections between the nodes if > we have about 100 connections to the first node and no connections to > the second one? > > 3. Are there any benefits of using two-node cluster where the first > node has 50 connections and the second node has the same amount of > connections comparing to the two-node cluster where the first node has > 100 connections and the second node has no connections at all? > > There is no built in support for load balancing; there is some benefit > in doing so, however. The node to which a connection is actually > connected to will do a little more work. Spreading that work should > result in some performance improvement, even though the majority of the > load will be replicated on each node. > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] > > > _______________________________________________________ > > The information contained in this message may be privileged and conf > idential and protected from disclosure. If you are not the original > intended recipient, you are hereby notified that any review, > retransmission, dissemination, or other use of, or taking of any action > in reliance upon, this information is prohibited. If you have received > this communication in error, please notify the sender immediately by > replying to this message and delete it from your computer. Thank you > for your cooperation. Troika Dialog, Russia. > If you need assistance please contact our Contact Center (+7495) 258 > 0500 or go to www.troika.ru/eng/Contacts/system.wbp > > > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom. Company Reg No 2096520. VAT Reg No GB 991 2418 08. --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]
