Min - just a quick follow up on this. I had a little spare time today (don't tell anyone) so I put together a single socket to single socket poller / no poller test in C with libzmq.
The ratio was almost the same as pyzmq - I had ~ 1.37 million messages a second with the poller, and ~ 4.7 million messages a second without it. So, the python ration (with the updated pyzmq with your poller improvements) is ~ 2.9x and in C it's ~ 2.7x. Brian On Tue, Jun 25, 2013 at 10:56 AM, Brian Knox <[email protected]> wrote: > Min - I merged the poll request on my little laptop and gave it a spin. > Before your pull request: > > > mps: 28066 > mps: 28116 > mps: 29745 > mps: 28912 > mps: 28679 > > after your pull request: > > mps: 78138 > mps: 78364 > mps: 78411 > mps: 78065 > mps: 78175 > > Excellent. > > By the way, just for fun I imported zmq.green instead of zmq without > changing anything else > > mps: 24738 > mps: 24779 > mps: 24759 > mps: 24630 > mps: 24702 > > And it's performance is well below straight zmq Poller. > > > Thanks for the quick patch! > > Brian > > > On Tue, Jun 25, 2013 at 8:45 AM, Brian Knox <[email protected]> wrote: > >> Thanks! The 4x degradation is within an acceptable range for what we're >> doing and will allow me to keep some things in python for awhile longer. >> I'll patch locally and take it for a test drive today. >> >> Brian >> >> >> On Mon, Jun 24, 2013 at 11:09 PM, MinRK <[email protected]> wrote: >> >>> From my tests this evening, it looks like zmq_poll does have an >>> appreciable cost (plus the overhead of being wrapped in Python), and will >>> be notably slower than a simple `while True: s.recv()`. But I did find >>> some inefficiencies in the Poller implementation, which should be addressed >>> by PR #381 <https://github.com/zeromq/pyzmq/pull/381>, bringing the >>> difference closer to 4x from 10x. >>> >>> Your tests on my laptop: >>> >>> pure recv: >>> mps: 241121 >>> mps: 419644 >>> mps: 413874 >>> mps: 434421 >>> mps: 428623 >>> mps: 404847 >>> >>> poll (master): >>> mps: 48544 >>> mps: 54143 >>> mps: 51642 >>> mps: 48888 >>> mps: 47591 >>> mps: 54218 >>> >>> poll (after 381): >>> mps: 110436 >>> mps: 110876 >>> mps: 104220 >>> mps: 110690 >>> mps: 100544 >>> mps: 110922 >>> >>> -MinRK >>> >>> >>> >>> On Mon, Jun 24, 2013 at 12:18 PM, Brian Knox <[email protected]> wrote: >>> >>>> Thanks Min - additionally if I'm incorrect in my assumption that the >>>> poll look in my example code should perform better, or if there's a better >>>> way to do what I'm doing just let me know. >>>> >>>> The example code polls only one socket (which isn't that useful, hah) >>>> but I wanted to keep the test case dirt simple. The actual thing I was >>>> working on has one socket that should receive data at a relatively high >>>> rate (say 50k to 100k msgs/s) , and one socket that receives data at a >>>> much lower rate (say 1 msg/s) (for heartbeating, command and control, etc). >>>> >>>> >>>> Brian >>>> >>>> >>>> On Mon, Jun 24, 2013 at 12:12 PM, Min RK <[email protected]> wrote: >>>> >>>>> Thanks for the report, I will look into whether I have introduced a >>>>> performance degradation in the last few iterations. >>>>> >>>>> -MinRK >>>>> >>>>> On Jun 24, 2013, at 11:27, Brian Knox <[email protected]> wrote: >>>>> >>>>> > It's been awhile since I've used pyzmq, and I'm running into a >>>>> performance issue using Poller.poll(). >>>>> > >>>>> > With a simple blocking recv() in a while True loop, I get ~ 300k >>>>> messages a second. >>>>> > >>>>> > With the simplest case of the same code using a poll on the socket, >>>>> I get ~ 30k messages a second. >>>>> > >>>>> > (pyzeromq)taotetek@Moya:~/src/performance_example$ python >>>>> ./receiver.py >>>>> > mps: 226049 >>>>> > mps: 348582 >>>>> > mps: 271728 >>>>> > mps: 300389 >>>>> > mps: 411059 >>>>> > mps: 276749 >>>>> > >>>>> > (pyzeromq)taotetek@Moya:~/src/performance_example$ python >>>>> ./poll_receiver.py >>>>> > mps: 28066 >>>>> > mps: 28116 >>>>> > mps: 29745 >>>>> > mps: 28912 >>>>> > mps: 28679 >>>>> > >>>>> > I don't remember a 10x performance degradation last time I used a >>>>> poller - but perhaps I'm remembering something wrong, or making a >>>>> painfully >>>>> obvious error in my code .. I've been using czmq from C mostly for the >>>>> last >>>>> while. >>>>> > >>>>> > I'm using pyzmq from git master, and the latest Cython (0.19.1) with >>>>> libzmq 3.2.3. >>>>> > >>>>> > I've attached the test cases I'm using. >>>>> > >>>>> > <performance_example.tar> >>>>> > _______________________________________________ >>>>> > zeromq-dev mailing list >>>>> > [email protected] >>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> _______________________________________________ >>>>> zeromq-dev mailing list >>>>> [email protected] >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>> >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
