On Feb 3, 2010, at 1:51 PM, Craig Citro wrote: >> For most projects where build time actually matters, I bet there's >> a lot >> more to gain from getting distutils to build multiple modules in >> parallel, >> than from reducing the amount of code that is built in each step. >> > > FWIW, this isn't so hard to do. I implemented this for Sage, and I 2am > actually planning to try and push something upstream soon. The biggest > issue (well, other than "the state of the distutils community" :P) is > that there's currently a single method in build_ext which both (1) > decides whether or not an extension needs rebuilt, and (2) actually > rebuilds it. If you just naively call this in parallel, especially on > a rebuild, you end up spawning a whole bunch of threads which decide > not to rebuild, and end up losing the speedup from doing the work in > parallel. The obvious fix is to just split this logic; this could > definitely be done in such a way as to be backward-compatible, but it > means that any project that implements its own builder (i.e. > subclasses build_ext) can't take advantage of this without reworking > their code. (We ran into exactly this problem with numpy when I first > tried to merge this into Sage.)
What I think we should do is provide tools to handle the entire .pyx - > .c conversion before passing off to distutile/build_ext (or any other build system), like we so in Sage. - Robert _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
