Reposting On 25 Jun 2013, at 09:38, Tim Watson <[email protected]> wrote:
> What does your publishing code look like? The figures below are expected in > that the consumer can keep pace with the producer - it could hardly be > expected to consume faster than messages are arriving in the queue(s). So the > slowness is very likely on the producing side. > > Are you using persistent messages and either publisher confirms or > transactions? If so, how often are you waiting on confirms/commits? > > With the official clients we typically see avg rates of 50 - 60Khz with > non-persistent messages. Persistence slows things down a tad, as do confirms > (and more so transactions) but even with persistent messages and confirms, > rates >= 5Khz are expected. It /sounds/ like you might be publishing > persistent messages with confirms enabled and waiting for a confirm (ack) > from the broker for each message. That involves disk I/O on the server for > each message plus network latency, effectively making publishing synchronous > (and very slow by comparison). > > Cheers, > Tim > > On 25 Jun 2013, at 08:44, Priyanki Vashi <[email protected]> wrote: > >> Hi there, >> >> I am doing a performance study of RabbitMQ-3.1.1 and this is my first time >> to do such a performance study with any messaging broker :))- >> >> 1) I have thoroughly gone through rabbitMQ in action' and learnt important >> concepts. >> >> 2) Tried single node broker to get a feel of how it is working and then set >> up a four node cluster (with two disk and two RAM type of node). Also >> configured HAproxy TCP Load balancer so that I can just provide single port >> to connect to the Cluster. >> >> 3) I am simulating producer and consumers through Python scripts ( using >> Python-pika library methods to connect to server , publish subscribe etc.) >> >> 4) My scripts are working fine but where I am stuck is no matter what I do >> my throughout is always 300 msg/sec. >> >> 5) I have defines durable exchanges and queues >> >> My final requirement is to run atleast 10 to 15 producer and 60 to 70 >> consumer simultaneously and I want to start with linear increase in number >> of producer and consumer so that I can make conclusions about throughout, >> fault handling, processor utilization etc. etc but I am seriously stuck now >> after trying to start in initial steps only. This group's help would be >> really appreciated. >> >> I have started with following different scenarios but no matter what I do my >> throughput is more or less remaining same (300 msg/sec) except for Scenario-1 >> >> Scenario-1 >> -1 producer and No consumer and no queue binded to exchange >> -Producer is running in infinite loop and publishing to one fanout exchange >> - publisher/confirm disabled >> -Publisher rate - 6200 msg/sec ( checked through web management plugin) >> >> Tried scenario-1 with also fanout type of exchange and it's the same publish >> rate >> I know that Scenario-1 is not really useful, since there are no queues and >> ultimately messages will be dropped but as a part of debugging process I >> tried this and I see above mentioned results. >> >> Scenario-2 >> -1 producer and 1 consumer >> -Producer is running in infinite loop and publishing to one direct exchange >> -A consumer has it's own dedicated queue and listening to above exchange >> - publisher/confirm and consumer ack are disabled >> Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and >> deliver rate - 300 msg/sec) >> >> Tried Scenario-2 also with fanout type of exchange and enabling publisher >> confirm and consumer ack >> Still the same throughput as 300 msg/sec >> >> Scenario-3 >> -1 producer and 4 consumer >> -Producer is running in infinite loop and publishing to four direct exchange >> -A consumer has it's own dedicated queue and listening to respective >> exchange >> - publisher/confirm and consumer ack are disabled >> Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and >> deliver rate - 300 msg/sec) >> >> Tried Scenario-3 also with fanout type of exchange and enabling publisher >> confirm and consumer ack >> Still the same throughput as 300 msg/sec >> >> Tried configuring prefetch_count parameters also to 100 but it still gives >> me same throughput of 300 msg/sec. >> I am honestly going crazy with this. >> >> After seeing this behavior, I am seriously suspecting that there is some >> serious limitation with my simulated producers and consumers. >> Has anyone else has tried Python-pika client and any idea on throughput with >> this version of rabbit ? >> Did anyone have rough idea about throughout with rabbitMQ-3.1.1 ? >> >> I can also share my python scripts if required but I would really appreciate >> some light on this situation >> Also what points to take care, in order to improve throughput ? >> >> Best Regards, >> Priyanki >> _______________________________________________ >> rabbitmq-discuss mailing list >> [email protected] >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -- You received this message because you are subscribed to the Google Groups "rabbitmq-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/rabbitmq-discuss. For more options, visit https://groups.google.com/groups/opt_out.
