Before tweaking the send calls, I'd advise making a *minimal* test case that shows the effect, in C. We can then see exactly where the latency spike is coming from. No point in trying to remove an effect we don't understand (in fact, counter productive, we need to understand the spike first.)
On Wed, Sep 30, 2015 at 8:31 AM, <[email protected]> wrote: > We also agree that there must be a buffer somewhere in TCP stack or > OS/memory(page allocation)/scheduling. It becomes clear, like Ben mentioned, > when the sender sends a lot more messages than the receiver receives. > > A time delay for the TCP connection could be a reason for that, as Peter > asked. But in our scenario we do a zmq_connect/zmq_bind at the beginning, > which is not part of the measured results. > We use inproc communication with ZMQ 4.0.4 in a C++ project with "zmq_send". > > All message data we have is memory aligned structures (memory blocks). Does > this mean we could also utilize "zmq_send_const"? We could then eliminate at > least one copy operation if I am right. > > > >> Date: Tue, 29 Sep 2015 09:44:19 +0200 >> From: Pieter Hintjens <[email protected]> >> Subject: Re: [zeromq-dev] Bad ZMQ latency on first 500 messages per >> connection >> To: ZeroMQ development list <[email protected]> >> Message-ID: >> >> <CADL5_sgpYKi=k2doeObsrAkmGbfzv6SMcfe=q2qwp0_zcoa...@mail.gmail.com> >> Content-Type: text/plain; charset=UTF-8 > >> >> Have you excluded the cost of establishing the TCP connection itself >> (something around 10 msec, depending on the setup)? >> >> On Tue, Sep 29, 2015 at 8:49 AM, Ben Kloosterman <[email protected]> >> wrote: >>> "As can be seen, sender and receiver throughput's differ at the beginning >>> of >>> the test. This seems to suggest that there is some kind of buffering >>> involved on the lower layers of the stack" >>> >>> Nagle is turned off but it looks like it keeps increasing some sort of >>> buffer in the tcp stack.. could also be OS / memory ( page allocation) / >>> Scheduling priority. >>> >>> Note in those 500 messages the sender was sending a lot more messages , >>> than >>> the receiver was receiving so a buffer would be growing somewhere. >>> >>> Ben >>> >>> On Tue, Sep 29, 2015 at 4:35 PM, <[email protected]> >>> wrote: >>>> >>>> Hi all, >>>> >>>> when sending many messages at once there is a known latency issue: The >>>> first 500 messages are sent with a high latency shown in the graph in >>>> the >>>> whitepaper > http://zeromq.org/whitepapers:measuring-performance before it > >>>> gets to very high speed messaging. >>>> >>>> The author summarizes: >>>> * For latency: ... the latency for first 500 messages is quite poor (up >>>> to >>>> 1,500 microseconds), however, afterwards it stabilises at approximately >>>> 150 >>>> microseconds, with occasional peaks up to 400 microseconds. >>>> * For throughput: This seems to suggest that there is some kind of >>>> buffering involved on the lower layers of the stack. >>>> >>>> In our scenario, we use a Dealer/Router connection, which is in fact >>>> like >>>> REQ/REP. >>>> >>>> Question: >>>> Why have the first 500 messages this latency issue? What is the reason >>>> for >>>> this and how could we solve this issue, because we require a high >>>> performance application even for the first messages. >>>> >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
