On Tue, Jun 30, 2009 at 12:25 PM, Stefan Behnel<[email protected]> wrote: > > Robert Bradshaw wrote: >> On Jun 29, 2009, at 5:45 AM, Dag Sverre Seljebotn wrote: >>> do you know if e.g. Sage builds are IO bound? >> >> Not sure, but that's certainly a big component. > > Now that you mention it, how do you parallelise the builds anyway? > (assuming that you do that) > > Stefan
I wrote half the code to do this (=build Sage's Cython library in parallel), so I'll comment briefly. I've cc'd Craig Citro who wrote the other half, so maybe he can comment. The Cython part of the Sage library consists of 237 distinct Cython modules. We build these modules in parallel in two stages: (1) Run Cython on all modules that need to be regenerated (as determined by our automatic dependency checking system) in parallel. This is done in parallel using the standard Python-2.6 multiprocessing module. (2) One (1) is completely done, run gcc in parallel, again I think using multiprocessing. This is done by modifying Python's distutils. The Sage project's standard developer build machine has 24 cores, so having the above is a huge time saver. For those using Sage who want to use the above, just do "export MAKE='make -j12'" say, to tell "sage -br" to run up to 12 processes at once during the build process. -- William > > _______________________________________________ > Cython-dev mailing list > [email protected] > http://codespeak.net/mailman/listinfo/cython-dev > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
