Btw, I realized that I actually already hit this "phenomenon" of poll() being called so fast!!! This is the link of the thread where I raised the issue: https://lists.zeromq.org/pipermail/zeromq-dev/2017-October/031974.html
At that time I solved by simply using the ZMQ_RCVTIMEO option as suggested by Luca... this time it's slightly different because I'm already using that option but now I have 2 zmq sockets to dequeue so that "inbound poll rate" issue is doubled... I will check if changing that zmq non-configurable internal parameter makes a performance difference or not! Francesco 2018-04-12 22:47 GMT+02:00 Francesco <[email protected]>: > [sorry I hit "send" too early! here's the complete email] > > Hi Luca, Hi Bill, > thanks for the answers. > Actually I verified that the configure script of zmq seems to be using > epoll: > > configure: Choosing polling system from 'kqueue epoll devpoll pollset > poll select'... > configure: Using 'epoll' polling system with CLOEXEC > > and indeed inside platform.hpp I found: > > /* Use 'epoll' polling system */ > #define ZMQ_USE_EPOLL 1 > > /* Use 'epoll' polling system with CLOEXEC */ > #define ZMQ_USE_EPOLL_CLOEXEC 1 > > I also read the links you sent me Bill: actually since I'm monitoring only > 2 file descriptors (2 zmq sockets) I think that poll() and epoll() > difference is hardly measurable. > My feeling is just that whatever code ends up calling poll() so often will > pay a cost in performance due to the syscall overhead: > > http://arkanis.de/weblog/2017-01-05-measurements-of-system- > call-performance-and-overhead > http://man7.org/linux/man-pages/man7/vdso.7.html > > So my question is actually the following one: > In a ZMQ application, in which thread context are the "real" TCP socket > file descriptors actually polled? > Is that polling happening in ZMQ background threads? > Or rather it happens inside the thread that calls the zmq_msg_recv() / > zmq_poll() / zmq_poller_wait_all() ? > > Because from looking briefly at zmq code it looks like zmq_msg_recv() is > just talking with the zmq background thread via some kind of inter-thread > signaler. > So in my view I _guess_ that the real polling of the TCP socket should > happen in zmq background threads and the application threads just dequeues > some kind of queue that lives in those background threads. However that > seem to be proved wrong by a simple test: I tried to put a breakpoint on > the syscall poll()... the stacktrace I get is: > > #0 poll () at ../sysdeps/unix/syscall-template.S:84 > #1 0x00007ffff63db0ba in zmq::signaler_t::wait(int) () > #2 0x00007ffff63bb045 in zmq::mailbox_t::recv(zmq::command_t*, int) () > #3 0x00007ffff63dba37 in zmq::socket_base_t::process_commands(int, bool) > #4 0x00007ffff63ddbca in zmq::socket_base_t::recv(zmq::msg_t*, int) () > #5 0x00007ffff6400620 in zmq_msg_recv () > [my code stack frames] > > so it seems like it's the application threads that ends up calling that > poll(). > Note that I set the RX timeout on my socket equal to 0 (ZMQ_DONTWAIT)... > > > Thanks, > Francesco > > > > > > 2018-04-12 22:39 GMT+02:00 Francesco <[email protected]>: > >> Hi Luca, Hi Bill, >> thanks for the answers. >> Actually I verified that the configure script of zmq seems to be using >> epoll: >> >> configure: Choosing polling system from 'kqueue epoll devpoll pollset >> poll select'... >> configure: Using 'epoll' polling system with CLOEXEC >> >> and indeed inside platform.hpp I found: >> >> /* Use 'epoll' polling system */ >> #define ZMQ_USE_EPOLL 1 >> >> /* Use 'epoll' polling system with CLOEXEC */ >> #define ZMQ_USE_EPOLL_CLOEXEC 1 >> >> I also read the links you sent me Bill: actually since I'm monitoring >> only 2 file descriptors (2 zmq sockets) I think that poll() and epoll() >> difference is hardly measurable. >> My feeling is just that whatever code ends up calling poll() so often >> will pay a lot of >> >> http://arkanis.de/weblog/2017-01-05-measurements-of-system- >> call-performance-and-overhead >> >> >> >> >> >> >> 2018-04-11 21:05 GMT+02:00 Bill Torpey <[email protected]>: >> >>> Well, a little googling found this, which is a pretty good writeup: >>> https://jvns.ca/blog/2017/06/03/async-io-on-linux-- >>> select--poll--and-epoll/ >>> >>> >>> On Apr 11, 2018, at 2:42 PM, Bill Torpey <[email protected]> wrote: >>> >>> So, are there any benchmark tests that can be used to quantify the >>> overhead of zmq_poll? It seems like this question keeps coming up, and it >>> would certainly be nice to have some real numbers (and the code used to >>> generate them). >>> >>> Having said that, there are several mechanisms that zmq_poll can use, >>> and there are apparently significant performance differences between them. >>> My understanding of the conventional wisdom is that epoll is preferable to >>> poll, which is preferable to select — but I don’t have any data to back >>> that up. >>> >>> In any event, you can examine the output of the libzmq build to see >>> which mechanism is being used: >>> >>> -- Looking for kqueue >>> -- Looking for kqueue - not found >>> -- Looking for epoll_create >>> -- Looking for epoll_create - found >>> -- Looking for epoll_create1 >>> -- Looking for epoll_create1 - found >>> *-- Detected epoll polling method* >>> >>> I’ve found one (very old!) post that discusses these differences: >>> https://cs.uwaterloo.ca/~brecht/servers/epoll/. If anyone can suggest >>> additional sources, please do. >>> >>> >>> On Apr 11, 2018, at 1:46 PM, Luca Boccassi <[email protected]> >>> wrote: >>> >>> On Wed, 2018-04-11 at 18:47 +0200, Francesco wrote: >>> >>> Hi all, >>> >>> I'm using zmq_poller_wait_all() API in one of my threads because I >>> need to >>> poll over 2 ZMQ sockets. >>> I'm receiving a lot of traffic on both these sockets. >>> I think the performance of my software is bad and that may be due >>> IMHO to >>> the huge amount of poll() syscalls that my thread does... I think the >>> overhead of doing the system call is what is killing me... >>> >>> so my question is: when zmq_poller_wait_all() polls zmq FDs it is >>> actually >>> polling the real socket FD or rather some ZMQ internal structure? >>> If the latter is true, do I have a way to poll more than 1 zmq socket >>> without doing a system call? >>> >>> >>> THanks, >>> Francesco >>> >>> >>> I'm not familiar with the zmq_poller API so someone else might help - >>> in general, a common optimisation is to drain the socket using non- >>> blocking receives when poll wakes up (taking care of using heuristic to >>> avoid starving the other sockets), so that you can process multiple >>> messages per poll >>> >>> -- >>> Kind regards, >>> Luca Boccassi_______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >>> >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
