Hi Jason! On 04/21/2016 11:40 PM, Jason Madden wrote: > Hi all, > > (There's a TL;DR at the bottom---I'd like to contribute code to help libuv > work after a fork() on unix.) > > I'm one of the gevent maintainers. gevent is a fairly popular[1] Python > library for asynchronous event-driven programming (primarily networking, but > other things as well). Currently gevent is based on libev and supports both > unix and Windows while implementing an API compatible with the Python > standard library. > > There's been interest expressed by our community in moving off of libev and > onto libuv, primarily for improved Windows support. So recently I've > implemented a prototype backend for gevent using libuv [4]. > > Let me start by saying how much of a joy the libuv API is to work with. It's > clean and consistent, and the source code is easy to read. Not all event > loops are like that. I really appreciate the work that the maintainers and > contributors have put into libuv. >
Thanks for the kind words! Great work taking over the maintenance of Gevent, I like gevent myself and attempted to do this in the past [0] though it was mostly a test, so I'm really happy to see interest in getting libuv into gevent. > I've been able to get gevent's test suite to pass with libuv on OS X and > Linux, with the exception of those tests that involve forking but *not* > execing. Being able to continue to run Python code in a forked child process > is a crucial part of the way many Python programs on unix are written, > including not just network servers like gunicorn but those that use the > standard library 'multiprocessing' feature to improve concurrency. > > Currently after a fork, libuv either is in an interesting state where things > may or may not function (three different results in gevent's test suite on > three different linux distros) or it simply aborts the whole process (kqueue, > aix and sunos-based systems). I've been unable to find a reliable workaround > to this. > > Unless gevent can survive a fork, I won't be able to base it on libuv. As a > gevent maintainer, that makes be sad because libuv is so nice to work with > (and I'm really not interested in maintaining both a libev and libuv backend). > > > TL;DR: I'm willing to write code to help libuv survive a fork() (I can easily > test on OS X and Linux). But before I start, would the maintainers be willing > to accept such a patch? I realize that libuv strives to be fully > cross-platform, and I also realize that similar issues have been brought up > before (e.g., [2], [3]), but I didn't find anywhere that specifically said > "no way." There are multiple approaches that could work for gevent, > everything from a libev-style 'uv_loop_fork()' call where all the watchers > survive (obviously ideal from my point of view, especially if fork watchers > got added too :), to simply terminating the loop in the child and letting > gevent create a new loop and watchers. If this is something the maintainers > are interested in, I look forward to working together to find the best > approach. (And if this is best discussed in another venue like a github issue > or IRC I'm happy to continue there.) > I went through those issues quickly, and from the top of my head, I'd say this is what you can attempt to do: - fix the broken global state after forking (IIRC it's just a signal pipe and the threadpool) - destroy the loop and recreate it in the child, re-creating all necessary watchers. The last part could be tricky because you'd need to dup all fds you want to reuse (I guess) and close everything nicely in the child. This *could* work, but I guess the only way to know is to put up the sleeves and give it a go. Unless there is strong opposition from other maintainers, I'd say go sketch up some APIs and open an issue/PR so we start discussing the code/design. Cheers, [0]: https://github.com/saghul/uvent -- Saúl Ibarra Corretgé bettercallsaghul.com -- You received this message because you are subscribed to the Google Groups "libuv" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/libuv. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: OpenPGP digital signature
