#8306: Parallel inter/intra-spkg builds
----------------------------+-----------------------------------------------
Reporter: mpatel | Owner: GeorgSWeber
Type: enhancement | Status: needs_review
Priority: major | Milestone: sage-4.3.4
Component: build | Keywords:
Author: Mitesh Patel | Upstream: N/A
Reviewer: David Kirkby | Merged:
Work_issues: |
----------------------------+-----------------------------------------------
Old description:
> Along with a primed [http://ccache.samba.org/ ccache], compiling multiple
> spkgs in parallel may significantly speed up Sage builds on multicore
> machines. See [http://groups.google.com/group/sage-
> release/browse_thread/thread/4c915ae814dd6514 sage-release] for some
> information.
>
> The attached `spkg/install` and `spkg/standard/deps` depend on
> 4.3.4.alpha0 + #7943 + #8357's spkg. There's an implicit dependence on
> #8432. Notes:
>
> * It seems that prefixing a command with `+` (see `deps`) tells GNU make
> to attempt to propagate `$MAKE` to sub-makes. Please see the
> [http://www.gnu.org/software/automake/manual/make/MAKE-Variable.html
> documentation] (and the bottom of
> [http://www.gnu.org/software/automake/manual/make/Error-Messages.html
> #Error-Messages this page]) for an exceptionally clear explanation. :)
>
> * In order to determine which commands could "handle" `+`, I prefixed
> them all and went through many build / test / edit cycles on sage.math.
> Besides the setuptools packages (see below), the packages that are not
> built with `+` are atlas, ecl, maxima, R, and singular. Note: It may be
> possible to build in parallel without any plus signs, but when I try
> this, I get a strange make jobserver error with palp.
>
> * I put packages installed by setuptools (twisted, zodb, pygments,
> jinja, jinja2, sphinx, sqlalchemy, sagenb) in a linear chain, because
> setuptools does not consistently update `SAGE_LOCAL/lib/python/site-
> packages/easy-install.pth` in parallel.
>
> * I removed `MAKE_REL` commands from `deps`, since `sage-spkg` always
> does this.
>
> * I put the per-spkg build logs in `SAGE_ROOT/spkg/logs`. The patch
> [attachment:trac_8306_scripts-spkg_log_files.patch] does the same for
> `sage -i` and `sage -f`. Note: I have not (re)moved `install.log`.
>
> * I dropped mpir from gap's dependencies (see
> [http://groups.google.com/group/sage-devel/msg/c97a99f23ea576b7 sage-
> devel]).
>
> * I removed `$(GNUCRYPTOLIB)` and `$(TWISTEDWEB2)`, which appear to be
> unnecessary.
>
> * I removed several comments about Gnuplot, dvipng, etc. These also
> seem to be obsolete. I wasn't sure about the (Py)OpenSSL comments, so I
> left them.
>
> To test the changes:
>
> * Update `spkg/install` and `spkg/standard/deps`.
> * `export SAGE_PARALLEL_SPKG_BUILD="yes"`
> * `export MAKE="make -jX"` where `X > 1`
> * `make`
>
> To make `sage -i foo-X.Y.spkg` ''also'' append to
> `spkg/logs/foo-X.Y.log`, apply [attachment:trac_8306_scripts-
> spkg_log_files.patch] to the scripts repository.
New description:
Along with a primed [http://ccache.samba.org/ ccache], compiling multiple
spkgs in parallel may significantly speed up Sage builds on multicore
machines. See [http://groups.google.com/group/sage-
release/browse_thread/thread/4c915ae814dd6514 sage-release] for some
information.
The attached `spkg/install` and `spkg/standard/deps` depend on
4.3.4.alpha0 + #7943 + #8357's spkg. There's an implicit dependence on
#8432. Notes:
* It seems that prefixing a command with `+` (see `deps`) tells GNU make
to attempt to propagate `$MAKE` to sub-makes. Please see the
[http://www.gnu.org/software/automake/manual/make/MAKE-Variable.html
documentation] (and the bottom of
[http://www.gnu.org/software/automake/manual/make/Error-Messages.html
#Error-Messages this page]) for an exceptionally clear explanation. :)
* In order to determine which commands could "handle" `+`, I prefixed
them all and went through many build / test / edit cycles on sage.math.
Besides the setuptools packages (see below), the packages that are not
built with `+` are atlas, ecl, maxima, R, and singular. Note: It may be
possible to build in parallel without any plus signs, but when I try this,
I get a strange make jobserver error with palp.
* I put packages installed by setuptools (twisted, zodb, pygments, jinja,
jinja2, sphinx, sqlalchemy, sagenb) in a linear chain, because setuptools
does not consistently update `SAGE_LOCAL/lib/python/site-packages/easy-
install.pth` in parallel.
* I removed `MAKE_REL` commands from `deps`, since `sage-spkg` always
does this.
* I put the per-spkg build logs in `SAGE_ROOT/spkg/logs`. The patch
[attachment:trac_8306_scripts-spkg_log_files.patch] does the same for
`sage -i` and `sage -f`. Note: I have not (re)moved `install.log`.
* I dropped mpir from gap's dependencies (see
[http://groups.google.com/group/sage-devel/msg/c97a99f23ea576b7 sage-
devel]).
* I removed `$(GNUCRYPTOLIB)` and `$(TWISTEDWEB2)`, which appear to be
unnecessary.
* I removed several comments about Gnuplot, dvipng, etc. These also seem
to be obsolete. I wasn't sure about the (Py)OpenSSL comments, so I left
them.
To test the changes:
* Update `spkg/install` and `spkg/standard/deps`.
* Add #8357's eclib-*.spkg.
* `export SAGE_PARALLEL_SPKG_BUILD="yes"`
* `export MAKE="make -jX"` where `X > 1`
* `make`
To make `sage -i foo-X.Y.spkg` ''also'' append to `spkg/logs/foo-X.Y.log`,
apply [attachment:trac_8306_scripts-spkg_log_files.patch] to the scripts
repository.
--
Comment(by mpatel):
The eclib and sage packages did not build:
{{{
#!sh
$ grep "An error occurred"
/dev/shm/mvngu/sandbox/sage-4.3.4.alpha0/spkg/logs/*
eclib-20080310.p9.log:sage: An error occurred while installing
eclib-20080310.p9
sage-4.3.4.alpha0.log:sage: An error occurred while installing
sage-4.3.4.alpha0
}}}
Ticket #8357's `eclib-*.spkg` should fix this. I apologize for not
mentioning it in "To test the changes".
Questions:
* If I untar `sage-*.tar`, copy over `install`, etc., and start the
build, then at some point, `install` is overwritten with the original
version. Do you know which package or script does this? For my tests,
I've run `chmod u-w install` before `make`.
* I don't know why the build didn't stop when eclib failed to build.
This seems strange. The file `installed/eclib*` is missing and according
to `deps`, it's required for the `sage` target. Similarly,
`installed/sage*` is a prerequisite for gap. Does anyone know why this
happens? Can we modify `deps` to avoid it?
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/8306#comment:32>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
