#8306: Parallel inter/intra-spkg builds
----------------------------+-----------------------------------------------
Reporter: mpatel | Owner: GeorgSWeber
Type: enhancement | Status: needs_work
Priority: major | Milestone: sage-4.3.4
Component: build | Keywords:
Author: Mitesh Patel | Upstream: N/A
Reviewer: David Kirkby | Merged:
Work_issues: |
----------------------------+-----------------------------------------------
Changes (by mpatel):
* cc: mvngu (added)
Old description:
> Along with a primed [http://ccache.samba.org/ ccache], compiling multiple
> spkgs in parallel may significantly speed up Sage builds on multicore
> machines. See [http://groups.google.com/group/sage-
> release/browse_thread/thread/4c915ae814dd6514 sage-release] for some
> information.
>
> The attached `spkg/install` and `spkg/standard/deps` depend on 4.3.3 +
> #7552 + #8191 + #7943 and a [attachment:eclib_makefiles.patch patched]
> eclib spkg. Notes:
>
> * To build multiple spkgs simultaneously, please replace
> `PARALLEL_SPKG_BUILD="no"` with `PARALLEL_SPKG_BUILD="yes"` near the top
> of `install` and `export MAKE="make -jX"`, where `X > 1`. With the
> default (`"no"`), builds should proceed as before.
>
> * It seems that prefixing a command with `+` (see `deps`) tells GNU make
> to attempt to propagate `$MAKE` to sub-makes. Please see the
> [http://www.gnu.org/software/automake/manual/make/MAKE-Variable.html
> documentation] (and the bottom of
> [http://www.gnu.org/software/automake/manual/make/Error-Messages.html
> #Error-Messages this page]) for an exceptionally clear explanation.
>
> * In order to determine which commands could "handle" `+`, I prefixed
> them all and went through many build / test / edit cycles on sage.math.
> Besides the setuptools packages (see below), the packages that are not
> built with `+` are atlas, ecl, maxima, and singular.
>
> * I grouped packages installed by setuptools (twisted, zodb, pygments,
> jinja, jinja2, sphinx, sqlalchemy, sagenb) into one target, because
> setuptools does not consistently update `SAGE_LOCAL/lib/python/site-
> packages/easy-install.pth` in parallel.
>
> * I put the per-spkg build logs in `SAGE_ROOT/spkg/logs`. Feel free to
> suggest a different place. Note: I haven't (re)moved `install.log`.
>
> * I added iconv to gd's dependencies (cf. #8191).
>
> * I dropped mpir from gap's dependencies (see
> [http://groups.google.com/group/sage-devel/msg/c97a99f23ea576b7 sage-
> devel]).
>
> * I removed `$(GNUCRYPTOLIB)` and `$(TWISTEDWEB2)`, which appear to be
> unnecessary.
>
> * I removed several comments about Gnuplot, dvipng, etc. These also
> seem to be obsolete. I wasn't sure about the (Py)OpenSSL comments, so I
> left them.
>
> I'm not a `Makefile` expert. Feel free to suggest improvements!
New description:
Along with a primed [http://ccache.samba.org/ ccache], compiling multiple
spkgs in parallel may significantly speed up Sage builds on multicore
machines. See [http://groups.google.com/group/sage-
release/browse_thread/thread/4c915ae814dd6514 sage-release] for some
information.
The attached `spkg/install` and `spkg/standard/deps` depend on 4.3.3 +
#7552 + #8191 + #7943 and the latest spkg at #8357. Notes:
* It seems that prefixing a command with `+` (see `deps`) tells GNU make
to attempt to propagate `$MAKE` to sub-makes. Please see the
[http://www.gnu.org/software/automake/manual/make/MAKE-Variable.html
documentation] (and the bottom of
[http://www.gnu.org/software/automake/manual/make/Error-Messages.html
#Error-Messages this page]) for an exceptionally clear explanation. :)
* In order to determine which commands could "handle" `+`, I prefixed
them all and went through many build / test / edit cycles on sage.math.
Besides the setuptools packages (see below), the packages that are not
built with `+` are atlas, ecl, maxima, R, and singular.
* I put packages installed by setuptools (twisted, zodb, pygments, jinja,
jinja2, sphinx, sqlalchemy, sagenb) in a linear chain, because setuptools
does not consistently update `SAGE_LOCAL/lib/python/site-packages/easy-
install.pth` in parallel.
* I removed `MAKE_REL` commands from `deps`, since `sage-spkg` always
does this.
* I put the per-spkg build logs in `SAGE_ROOT/spkg/logs`. Feel free to
suggest a different place. Note: I haven't (re)moved `install.log`.
* I added iconv to gd's dependencies (cf. #8191).
* I dropped mpir from gap's dependencies (see
[http://groups.google.com/group/sage-devel/msg/c97a99f23ea576b7 sage-
devel]).
* I removed `$(GNUCRYPTOLIB)` and `$(TWISTEDWEB2)`, which appear to be
unnecessary.
* I removed several comments about Gnuplot, dvipng, etc. These also seem
to be obsolete. I wasn't sure about the (Py)OpenSSL comments, so I left
them.
To test the changes:
* Update `spkg/install` and `spkg/standard/deps`.
* `export SAGE_PARALLEL_SPKG_BUILD="yes"`
* `export MAKE="make -jX"` where `X > 1`
* `make`
To make `sage -i foo-X.Y.spkg` ''also'' append to `spkg/logs/foo-X.Y.log`,
apply [attachment:trac_8306_scripts-spkg_log_files.patch] to the scripts
repository.
--
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/8306#comment:18>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
