#4652: [with code, needs testing] make distutils compile Cython extensions in
parallel
------------------------+---------------------------------------------------
Reporter: craigcitro | Owner: craigcitro
Type: defect | Status: new
Priority: major | Milestone: sage-3.2.2
Component: build | Keywords:
------------------------+---------------------------------------------------
Currently, we've got excellent code to run Cython in parallel as part of
the Sage build process. However, once we ask distutils to begin compiling
that code, everything is done in serial, because distutils works solely in
serial.
== The Code ==
The attached file changes that. This (somewhat brutally) hacks distutils
to dispatch the calls to build C/C++ extensions in parallel, using
pyprocessing. (I totally jacked William's code from #3765 for this.)
Here's how to put this code in place:
* download the attached file (`build_ext.py`)
* replace
`$SAGE_ROOT/local/lib/python2.5/distutils/command/build_ext.py` with the
new version.
Now, if you want to test the new code, do the following:
* set the environment variable `SAGE_PARALLEL_DIST` to something. (The
code just checks to see if the variable is defined at all.)
* set the environment variable `MAKE` to `MAKE -j2`, where `2` is
replaced by the number of simultaneous build processes you want.
* build.
== Notes ==
If you want to test this, don't go around touching the `.pyx` files in the
Sage library, since Cython is much slower than `gcc`. Instead, simply go
around touching the `.c` and `.cpp` files in `$SAGE_ROOT/devel/sage/sage`.
One of the cool features we added with the new build system is that these
files get recompiled when they change.
There is now a line that prints as part of the build process that looks
like:
{{{
Total time spent compiling C/C++ extensions: 5.2876701355 seconds.
}}}
So try touching a bunch of files in the Sage library, and seeing what kind
of speedups you get.
There are two caveats I want to offer with this code:
* Michael points out that numpy does a lot with distutils. I could very
well have broken their use of distutils.
* I don't do anything involving dependency tracking between extensions.
In particular, if there are extensions that have to be built in a certain
order, this code could break it. (This code still compiles all source
files '''within''' an extension in the usual way.) Neither Michael nor I
could think of a situation where this would break anything in Sage, but
who knows ...
== The Plan ==
So while this code is cool, and will definitely save a ton of CPU time on
`sage.math` ('''cough''' mabshoff '''cough'''), the plan is '''not''' to
maintain it as a part of the Sage Python spkg indefinitely. Instead, if it
seems to work well, then we should try to clean this code up a bit more
and upstream it, since pyprocessing is standard in Python 2.6.
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/4652>
Sage <http://sagemath.org/>
Sage - Open Source Mathematical Software: Building the Car Instead of
Reinventing the Wheel
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en
-~----------~----~----~----~------~----~------~--~---
