Charles-François Natali added the comment: The implementation can't simply drain a queue, because it can be level-triggered (which is the case by default), so you want to keep events around. The kernel actually uses a red-black tree, but I didn't really check how it's done (there's probably very good reasosns for that).
Anyway, it can turn out to be a problem, for two reasons: - performance: by specifying a maxevents value too low, several calls to epoll_wait() must be made, instead of being able to process all events at once - the main problem that came to my mind is really starvation: let's say you have 2*FD_SETSIZE client sockets registered in your poll object. The first call to epoll_wait() returns sockets from 0 to FD_SETSIZE-1: you process them, so they're not ready anymore. The next call returns the clients from FD_SETSIZE to 2*FD_SETSIZE, same thing. But by the time you call epoll_wait() for the third time, if the first FD_SETSIZE clients are ready again, they will be returned, etc. So the 2*FD_SETSIZE th client may very well never be reported ready: that's starvation. I actually wrote a script to reproduce this issue: """ $ ./python /home/cf/test_epoll.py Working with 4080 FDs, -1 maxevents Number of missing FDs:4080 Number of ready FDs: 1023 Number of missing FDs:3057 Number of ready FDs: 0 Number of missing FDs:3057 Number of ready FDs: 1023 Number of missing FDs:2034 Number of ready FDs: 0 Number of missing FDs:2034 Number of ready FDs: 1023 Number of missing FDs:2034 Number of ready FDs: 0 Number of missing FDs:2034 Number of ready FDs: 1023 Number of missing FDs:2034 Number of ready FDs: 0 [...] """ If you specify a large enough maxevents: """ $ ./python /home/cf/test_epoll.py 64000 Working with 4080 FDs, 64000 maxevents Number of missing FDs:4080 Number of ready FDs: 4080 """ Note that it's really a corner issue, but I stumpled upon this problem while writing a test in issue #16853, and several projects (Tulip, Tornado, pyftpdlib) fall into this trap. I see several options: 1) just keep it that way (i.e. with maxevents set to FD_SETSIZE), and add a note in the documentation. I think it's always better to handle problems in the library than let the users get bitten. 2) increase `maxevents` default value. The problem is that I don't like magic values, and a too large value could incur increased memory consumption (and with the current implementation reduced performance because the epoll events buffer is allocated at each poll(), see issue #16876. 3) use a simple heuristic: start with a reasonable value for `maxevents` (FD_SETSIZE seems like a good candidate), and if epoll_wait() ever returns `maxevents` events, double the value (that's what libevent does, with a capping to 32000). ---------- Added file: http://bugs.python.org/file28591/test_epoll.py _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue16873> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
