Robert, You're running 3 processes on a 4-core box; each 0MQ process needs at least 2 cores for maximum performance, one for the application thread and one for the I/O thread. This is probably the main cause for latency building up. You should try with higher HWMs, and giving each process at least 2 cores.
-Pieter On Tue, May 29, 2012 at 12:07 PM, Robert Evans <[email protected]> wrote: > Hi Pieter, > > I ran your suggested C latency tests and the results seemed to be okay. > > local_lat and remote_lat with 10 bytes message with 100 000 roundtrips > showed a latency of 43 microseconds, inproc_lat 16 microseconds. I think the > problem is my broker. When I used a direct connection between publisher with > HWM=500 and subscriber without a broker the latency was about 0,3 > milliseconds. Once I added the broker again, the latency increased to ~15 > milliseconds. > > My CPU has 4 cores and the network is capable of 100 mbit/s. I'm currently > running everything (pub, sub and broker) on the same machine. Maybe this > leads to some problems with the CPU or the network. I will try moving the > broker to an own machine and see what will happen. I once had an experience > with AleriMQ, when the latency was significantly lower when the broker ran > on a seperate machine - perhaps it could be the same here. > > Regards, > Robert > > 2012/5/29 Pieter Hintjens <[email protected]> >> >> Robert, >> >> First, try some tests in plain C (using the low level API or CZMQ), to >> make sure that it's not the C# binding adding latency. >> >> There are command-line test programs you can use (perf/local_lat, >> perf/remote_lat). >> >> Secondly, if the latency increases when you send messages faster, >> there is something bogus happening. 0MQ will normally batch small >> messages which reduces latency if you send larger volumes. The main >> reason for setting a HWM is if your subscriber or network is too slow, >> and you need to guard against memory exhaustion. >> >> It sounds like you have a CPU contention issue. How many cores on your >> box? What kind of network? >> >> -Pieter >> >> >> >> On Tue, May 29, 2012 at 10:26 AM, Robert Evans >> <[email protected]> wrote: >> > Hello everyone, >> > >> > I am currently developing a Publish-Subscribe broker using the C# >> > binding >> > clrzmq v. 2.2.3. My operating system is Windows 7 Prof. and my system is >> > a >> > Intel i7 with 16 GByte RAM. >> > >> > The broker works without any problems, I have added one publisher and >> > one >> > subscriber and get message rates about 250 000 - 150 000 msg/second when >> > sending 10 byte messages on localhost. The broker simply forwards the >> > messages from the publisher to the subscriber, no big magic there. >> > >> > My problem is the increasing latency when sending many messages at short >> > time. When I only send one message and sleep for one millisecond the >> > latency >> > is < 1ms, but the message rate drops to 9000 msg/s. When I'm sending >> > messages straight without sleeping at ~ 200 000 msg/s, the latency >> > increases >> > with the number of messages sent. I set the HWM on the broker for the >> > pub >> > and sub socket to 500, which seems to narrow the latency to 10 - 20 ms. >> > This >> > is okay, but a value between 1 and 3 ms would be even better. >> > >> > Is there a way to reduce it even more? I don't really need the queuing >> > of >> > messages with the HWM; I would rather like to drop messages instantly >> > instead of queuing as low latency is very important in my project (but I >> > still need a message rate about 100 000 msg/s). As to say shortly: >> > Better >> > drop some messages for lower latency, instead of buffering them. >> > >> > It would be great if someone has an idea or already has some experience >> > with >> > tweaking the latency. Thank you! >> > >> > Regards, >> > Robert >> > >> > _______________________________________________ >> > zeromq-dev mailing list >> > [email protected] >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
