Hi Marc, Thanks for your fast answer. If I correctly understand what you're saying, the same ev_async watcher started ev_async_init()-ed and ev_async_start()-ed in each worker thread has to be passed to the ev_async_send() call ? It means it cannot be a local variable anymore and has to be accessible from the worker threads and the main (accept) thread ?
I changed my code a little bit so that the ev_async watchers are now "global" and persistent, and the whole thing seems to work (but keep reading): https://pastee.org/up2hp I also commented the console trace out so that I may now bench the code using a simple Apache bench command like this one: $ ab -c 100 -n 10000 http://localhost:6666/ (using ab was the whole purpose of simulating a simple HTTP server, as I didn't want to write a bench client as well ^_^). Most of the time, this will work, but sometimes it'll get stuck before the 10k requests are actually done. My guess is I have an overwrite condition while passing the accepted file descriptors form the main thread to the worker thread, using the "handle" member of the global CONTEXT structure (one per worker thread). If the server receives a large load of connections, the main thread will probably overwrite the "handle" member of the global structure too often, or at least before the corresponding worker thread had time to "read" it (in the async callback) and start the appropriate ev_io watcher in its own event loop. In other terms, some connections are never answered because the associated fd is never "seen" by a worker thread (and accumulate, accepted in the server, never to be released). So I guess I'm stuck back to my piping queue mechanism in this case, because a simple eventfd counter is not enough to hold an high-rate fd flow from the main thread. As you pointed out in your documentation (http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#Queueing), "ev_async does not support queuing of data in any way". Or I could use multiple av_async structures per worker threads, but it's just an ugly patch and will also eventually fail as the connection rate increases. Locking to make sure the previous handle was read before sending the next one is not really an option as it would kill the accept performance. I could also combine the pipe (for accepted handles queuing) and the eventfd signaling mechanism (indicating that the pipe needs to be read for new fds), but it's probably enough to just add the reading side of each pipe to an ev_io watcher in each worker thread (I know this is less optimal than an eventfd, but still seems to deliver great performance (with 4 worker threads on a dual bi-core server (4 way), I get about 16.000 req/s, which is not bad afterall)). In simple words, each communication pipe is acting as a buffering queue between the main thread and the worker threads. In the main thread, I've got something like this (in the network accept callback) : { char handle[10]; int client; client = accept(...); fcntl(client, F_SETFL, fcntl(client, F_GETFL) | O_NONBLOCK); sprintf(handle, "%d\n", client); write(loops[worker_index].cpipe[1], handle, strlen(handle)); } ` and in each worker thread, I've added the reading side of the communication pipe (i.e. loops[worker_index].cpipe[0]) to the worker thread event loop, with a read callback parsing the handles from the communication pipe and adding them to the worker thread event loop. Is there a chance to see a similar generic API directly into libev sometime soon ? It would avoid duplicating this code all over the place as I think this is a common pattern in high performance network servers where the load has to be spreaded among multiple threads to benefit from multiple cores servers. Or maybe you know of a different approach ? >> I'm quite new to libev and I'm having a hard time figuring out why a call to >> ev_async_send() will not trigger the corresponding async handler >> in the specified target loop. > > Any watcher, including an async watcher, must be started first. It will > likely work much better if you add an appropriate. > > ev_async_start (loop, &w); > >> As you can see, the round-robin distribution to worker threads seems to be >> fine, but the async_cb is never called. Despite the fact that >> I need to add a timer watcher to each worker thread loop (to keep in alive >> in absence of any other), I've been digging a little but into the > > Adding a timer is a possibility, another is to call ev_ref (loop). Absolutely, I forgot this one. Thx. >> libev code and it seems the internal evpipe_init() function (responsible for >> creating and initializing a communication eventfd/pipe >> watcher is actually never sent. ev_async_start() on the other end will call >> evpipe_init(), but my understanding is that it's not thread-safe > > Calling evpipe_init is indeed the job of ev_async_start. > >> (because it's not using the communication pipe and changing directly the >> loop internal async table from another thread). > > You have to start the watcher in the thread that waits for it. > >> Am i missing something here ? Am I using the right workflow ? > > Probably not - start the ev_async watcher in the thread that runs the > loop, then use ev_async_send from other threads. > > Whether you then use one async watcher per thread or just one globally (or > something else) is then a matter of design. > >> For now, I'm back at using my own pipes between the main and the worker >> threads, adding an ev_io watcher to the reading side of each >> pipe, and writing accepted network handles to the pipes in a round-robin way >> (like the above). I actually mimic the ev_async_send() >> behavior, and it works quite fine. But I'd really like to use the provided >> libev async facility if possible, instead of re-inventing the wheel >> in my code. > > Indeed, it will also likely be faster. Thanks a lot, Pierre-Yves
_______________________________________________ libev mailing list [email protected] http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
