> For most projects where build time actually matters, I bet there's a lot > more to gain from getting distutils to build multiple modules in parallel, > than from reducing the amount of code that is built in each step. >
FWIW, this isn't so hard to do. I implemented this for Sage, and I 2am actually planning to try and push something upstream soon. The biggest issue (well, other than "the state of the distutils community" :P) is that there's currently a single method in build_ext which both (1) decides whether or not an extension needs rebuilt, and (2) actually rebuilds it. If you just naively call this in parallel, especially on a rebuild, you end up spawning a whole bunch of threads which decide not to rebuild, and end up losing the speedup from doing the work in parallel. The obvious fix is to just split this logic; this could definitely be done in such a way as to be backward-compatible, but it means that any project that implements its own builder (i.e. subclasses build_ext) can't take advantage of this without reworking their code. (We ran into exactly this problem with numpy when I first tried to merge this into Sage.) -cc _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
