Hi Apache devs, as part of my documentation duties I would like to add some details about how connections are accepted in Event (for http://httpd.apache.org/docs/current/misc/perf-tuning.html and https://httpd.apache.org/docs/2.4/mod/event.html). I am not super expert with the MPMs code so what I am going to say could be terribly wrong, please be patient :)
Everything started reading the documentation for SO_REUSEPORT, in which event seems the only MPM non getting the same performance improvements as worker/prefork. I tried to read the code to get a better idea about the why, and I tried to compare the various MPMs. My understanding is: 1) only one listening socket configured will leverage modern kernel features (when available) avoiding the use of a process mutex before accept (essentially serializing the accept calls). 2) multiple listening sockets configured needs some sort of control since processes/threads needs to know what socket is ready for accept (or other events of course). APR offers the apr_pollset_* functions to solve this problem, for example using select/[e]poll over multiple listening sockets. The thundering herd problem arises when select/[e]poll is used by all the processes/threads over the same listening sockets, because each event wakes all up at the same time causing extra kernel work (and cpu utilization). SAFE_ACCEPT() is usually implemented in the various MPMs to solve 2) essentially serializing select/[e]poll/accept calls with a mutex (like SysV semaphores). Prefork has to do this in all its processes because there is a 1:1 correspondence between process and connection served, meanwhile Worker is a bit smarter and delegates only one thread per process to the role of "listener", assigning the accepted connection/fd to the first worker thread available. I was a bit puzzled not finding any SAFE_ACCEPT in event, but eventually (and thanks to the #dev IRC channel) I think I know why: Event uses more recent epolls/kqueue/etc.. when available and sets the pollset to APR_SO_NONBLOCK; apr_pollset_poll is configured to just wait a little timeout to catch events, returning (and not blocking) otherwise. The only lock used is thread based (one per process) since the pollset is updated by the worker threads periodically (Keep alive, lingering close, etc. forces the worker to give back control of the fd to the listener to be free to do something else). The thundering herd issue in event is not really a major problem since it has been mitigated using a combination of timeouts and non blocking I/O in the listeners (plus the listeners are usually few compared to the total number of worker threads). Last but not the least, https://httpd.apache.org/docs/current/mod/core.html#mutex is therefore not needed when using event. Does this vaguely resemble reality? If not, is there anybody kind enough to give me some direction about what to look/read? Documentation will be updated in return, I promise :) Thanks! Regards, Luca
