On Thu, 2019-10-24 at 16:55 -0400, Brett Viren via zeromq-dev wrote: > Hi again, > > Doron Somech < > [email protected] > > writes: > > > You need to create multiple connections to enjoy the multiple io > > threads. > > > > So in the remote/local_thr connect to the same endpoint 100 times > > and create 10 io > > threads. > > > > You don't need to create multiple sockets, just call connect > > multiple times with same > > address. > > I keep working on evaluating ZeroMQ against this 100 Gbps network > when I > get a spare moment. You can see some initial results in the attached > PNG. As-is, things look pretty decent but there are two effects I > see > which I don't fully understand and I think are important to achieving > something closer to saturation. > > 1) As you can see in the PNG, as the message size increases beyond 10 > kB > the 10 I/O threads become less and less active. This activity seems > correlated with throughput. But, why the die-off as we go to higher > message size and why the resurgence at ~1 MB? Might there be some > additional tricks to lift the throughput? Are such large messages > simply not reasonable? > > 2) I've instrumented my tester to include a sequence count in each > message and it uncovers that this multi-thread/multi-connect trick > may > lead to messages arriving to the receiver out-of-order. Given PUSH > is > round-robin and PULL is fair queued, I naively didn't expect > this. But > seeing it, I have two guesses. 1) I don't actually know what > "fair-queued" really means :) and 2) if a mute state is getting hit > then > maybe all bets are off. I do wonder if adding "credit based" > transfers > might solve this ordering. Eg, if N credits (or fewer) are used > given N > connects, might the round-robin/fair-queue ordering stay in lock > step? > > > Any ideas are welcome. Thanks! > > -Brett.
The zero-copy receiver uses by default an 8KB buffer. If the message received is larger, a new buffer is allocated per each message. That's probably why you are seeing a drop just around that size. A socket option ZMQ_IN_BATCH_SIZE has been recently added (TO BE USED WITH CARE) to change that default - maybe try experimenting with that and see if this assumption holds true. -- Kind regards, Luca Boccassi
signature.asc
Description: This is a digitally signed message part
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
