On Thu, 2018-04-12 at 22:47 +0200, Francesco wrote: > [sorry I hit "send" too early! here's the complete email] > > Hi Luca, Hi Bill, > thanks for the answers. > Actually I verified that the configure script of zmq seems to be > using > epoll: > > configure: Choosing polling system from 'kqueue epoll devpoll > pollset > poll select'... > configure: Using 'epoll' polling system with CLOEXEC > > and indeed inside platform.hpp I found: > > /* Use 'epoll' polling system */ > #define ZMQ_USE_EPOLL 1 > > /* Use 'epoll' polling system with CLOEXEC */ > #define ZMQ_USE_EPOLL_CLOEXEC 1 > > I also read the links you sent me Bill: actually since I'm monitoring > only > 2 file descriptors (2 zmq sockets) I think that poll() and epoll() > difference is hardly measurable. > My feeling is just that whatever code ends up calling poll() so often > will > pay a cost in performance due to the syscall overhead: > > http://arkanis.de/weblog/2017-01-05-measurements-of- > system-call-performance-and-overhead > http://man7.org/linux/man-pages/man7/vdso.7.html > > So my question is actually the following one: > In a ZMQ application, in which thread context are the "real" TCP > socket > file descriptors actually polled? > Is that polling happening in ZMQ background threads? > Or rather it happens inside the thread that calls the > zmq_msg_recv() / > zmq_poll() / zmq_poller_wait_all() ? > > Because from looking briefly at zmq code it looks like > zmq_msg_recv() is > just talking with the zmq background thread via some kind of inter- > thread > signaler. > So in my view I _guess_ that the real polling of the TCP socket > should > happen in zmq background threads and the application threads just > dequeues > some kind of queue that lives in those background threads. However > that > seem to be proved wrong by a simple test: I tried to put a breakpoint > on > the syscall poll()... the stacktrace I get is: > > #0 poll () at ../sysdeps/unix/syscall-template.S:84 > #1 0x00007ffff63db0ba in zmq::signaler_t::wait(int) () > #2 0x00007ffff63bb045 in zmq::mailbox_t::recv(zmq::command_t*, int) > () > #3 0x00007ffff63dba37 in zmq::socket_base_t::process_commands(int, > bool) > #4 0x00007ffff63ddbca in zmq::socket_base_t::recv(zmq::msg_t*, int) > () > #5 0x00007ffff6400620 in zmq_msg_recv () > [my code stack frames] > > so it seems like it's the application threads that ends up calling > that > poll(). > Note that I set the RX timeout on my socket equal to 0 > (ZMQ_DONTWAIT)...
TCP lives in the background I/O thread, the application threads use poll to wait for inproc control commands, which is already staggered etc etc to avoid having an impact on performance. > 2018-04-12 22:39 GMT+02:00 Francesco <[email protected]>: > > > Hi Luca, Hi Bill, > > thanks for the answers. > > Actually I verified that the configure script of zmq seems to be > > using > > epoll: > > > > configure: Choosing polling system from 'kqueue epoll devpoll > > pollset > > poll select'... > > configure: Using 'epoll' polling system with CLOEXEC > > > > and indeed inside platform.hpp I found: > > > > /* Use 'epoll' polling system */ > > #define ZMQ_USE_EPOLL 1 > > > > /* Use 'epoll' polling system with CLOEXEC */ > > #define ZMQ_USE_EPOLL_CLOEXEC 1 > > > > I also read the links you sent me Bill: actually since I'm > > monitoring only > > 2 file descriptors (2 zmq sockets) I think that poll() and epoll() > > difference is hardly measurable. > > My feeling is just that whatever code ends up calling poll() so > > often will > > pay a lot of > > > > http://arkanis.de/weblog/2017-01-05-measurements-of- > > system-call-performance-and-overhead > > > > > > > > > > > > > > 2018-04-11 21:05 GMT+02:00 Bill Torpey <[email protected]>: > > > > > Well, a little googling found this, which is a pretty good > > > writeup: > > > https://jvns.ca/blog/2017/06/03/async-io-on-linux-- > > > select--poll--and-epoll/ > > > > > > > > > On Apr 11, 2018, at 2:42 PM, Bill Torpey <[email protected]> > > > wrote: > > > > > > So, are there any benchmark tests that can be used to quantify > > > the > > > overhead of zmq_poll? It seems like this question keeps coming > > > up, and it > > > would certainly be nice to have some real numbers (and the code > > > used to > > > generate them). > > > > > > Having said that, there are several mechanisms that zmq_poll can > > > use, and > > > there are apparently significant performance differences between > > > them. My > > > understanding of the conventional wisdom is that epoll is > > > preferable to > > > poll, which is preferable to select — but I don’t have any data > > > to back > > > that up. > > > > > > In any event, you can examine the output of the libzmq build to > > > see which > > > mechanism is being used: > > > > > > -- Looking for kqueue > > > -- Looking for kqueue - not found > > > -- Looking for epoll_create > > > -- Looking for epoll_create - found > > > -- Looking for epoll_create1 > > > -- Looking for epoll_create1 - found > > > *-- Detected epoll polling method* > > > > > > I’ve found one (very old!) post that discusses these differences: > > > https://cs.uwaterloo.ca/~brecht/servers/epoll/. If anyone can > > > suggest > > > additional sources, please do. > > > > > > > > > On Apr 11, 2018, at 1:46 PM, Luca Boccassi <[email protected] > > > om> > > > wrote: > > > > > > On Wed, 2018-04-11 at 18:47 +0200, Francesco wrote: > > > > > > Hi all, > > > > > > I'm using zmq_poller_wait_all() API in one of my threads because > > > I > > > need to > > > poll over 2 ZMQ sockets. > > > I'm receiving a lot of traffic on both these sockets. > > > I think the performance of my software is bad and that may be due > > > IMHO to > > > the huge amount of poll() syscalls that my thread does... I think > > > the > > > overhead of doing the system call is what is killing me... > > > > > > so my question is: when zmq_poller_wait_all() polls zmq FDs it is > > > actually > > > polling the real socket FD or rather some ZMQ internal structure? > > > If the latter is true, do I have a way to poll more than 1 zmq > > > socket > > > without doing a system call? > > > > > > > > > THanks, > > > Francesco > > > > > > > > > I'm not familiar with the zmq_poller API so someone else might > > > help - > > > in general, a common optimisation is to drain the socket using > > > non- > > > blocking receives when poll wakes up (taking care of using > > > heuristic to > > > avoid starving the other sockets), so that you can process > > > multiple > > > messages per poll > > > > > > -- > > > Kind regards, > > > Luca Boccassi_______________________________________________ > > > zeromq-dev mailing list > > > [email protected] > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > > > > > > > _______________________________________________ > > > zeromq-dev mailing list > > > [email protected] > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Kind regards, Luca Boccassi
signature.asc
Description: This is a digitally signed message part
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
