#8306: Parallel inter/intra-spkg builds
----------------------------+-----------------------------------------------
Reporter: mpatel | Owner: GeorgSWeber
Type: enhancement | Status: needs_review
Priority: major | Milestone: sage-4.3.4
Component: build | Keywords:
Author: Mitesh Patel | Upstream: N/A
Reviewer: David Kirkby | Merged:
Work_issues: |
----------------------------+-----------------------------------------------
Description changed by mpatel:
Old description:
> Along with a primed [http://ccache.samba.org/ ccache], compiling multiple
> spkgs in parallel may significantly speed up Sage builds on multicore
> machines. See [http://groups.google.com/group/sage-
> release/browse_thread/thread/4c915ae814dd6514 sage-release] for some
> information.
>
> The attached `spkg/install` and `spkg/standard/deps` depend on
> 4.3.4.alpha0 + #7943 + #8357's spkg. There's an implicit dependence on
> #8432. Notes:
>
> * It seems that prefixing a command with `+` (see `deps`) tells GNU make
> to attempt to propagate `$MAKE` to sub-makes. Please see the
> [http://www.gnu.org/software/automake/manual/make/MAKE-Variable.html
> documentation] (and the bottom of
> [http://www.gnu.org/software/automake/manual/make/Error-Messages.html
> #Error-Messages this page]) for an exceptionally clear explanation. :)
>
> * In order to determine which commands could "handle" `+`, I prefixed
> them all and went through many build / test / edit cycles on sage.math.
> Besides the setuptools packages (see below), the packages that are not
> built with `+` are atlas, ecl, maxima, R, and singular. Note: It may be
> possible to build in parallel without any plus signs, but when I try
> this, I get a strange make jobserver error with palp.
>
> * I put packages installed by setuptools (twisted, zodb, pygments,
> jinja, jinja2, sphinx, sqlalchemy, sagenb) in a linear chain, because
> setuptools does not consistently update `SAGE_LOCAL/lib/python/site-
> packages/easy-install.pth` in parallel.
>
> * I removed `MAKE_REL` commands from `deps`, since `sage-spkg` always
> does this.
>
> * I put the per-spkg build logs in `SAGE_ROOT/spkg/logs`. The patch
> [attachment:trac_8306_scripts-spkg_log_files.patch] does the same for
> `sage -i` and `sage -f`. Note: I have not (re)moved `install.log`.
>
> * I dropped mpir from gap's dependencies (see
> [http://groups.google.com/group/sage-devel/msg/c97a99f23ea576b7 sage-
> devel]).
>
> * I removed `$(GNUCRYPTOLIB)` and `$(TWISTEDWEB2)`, which appear to be
> unnecessary.
>
> * I removed several comments about Gnuplot, dvipng, etc. These also
> seem to be obsolete. I wasn't sure about the (Py)OpenSSL comments, so I
> left them.
>
> To test the changes:
>
> * Update `spkg/install` and `spkg/standard/deps`.
> * Add #8357's eclib-*.spkg.
> * `export SAGE_PARALLEL_SPKG_BUILD="yes"`
> * `export MAKE="make -jX"` where `X > 1`
> * `make`
>
> To make `sage -i foo-X.Y.spkg` ''also'' append to
> `spkg/logs/foo-X.Y.log`, apply [attachment:trac_8306_scripts-
> spkg_log_files.patch] to the scripts repository.
New description:
Along with a primed [http://ccache.samba.org/ ccache], compiling multiple
spkgs in parallel may significantly speed up Sage builds on multicore
machines. See [http://groups.google.com/group/sage-
release/browse_thread/thread/4c915ae814dd6514 sage-release] for some
information. Notes:
* I put packages installed by setuptools (twisted, zodb, pygments, jinja,
jinja2, sphinx, sqlalchemy, sagenb) in a linear chain, because setuptools
does not consistently update `SAGE_LOCAL/lib/python/site-packages/easy-
install.pth` in parallel.
* I dropped mpir from gap's dependencies (see
[http://groups.google.com/group/sage-devel/msg/c97a99f23ea576b7 sage-
devel]).
To test the changes:
* Update `spkg/install` and `spkg/standard/deps` from
[http://sage.math.washington.edu/home/mpatel/trac/8306 here].
* Add #8357's `eclib-*.spkg`.
* Add #8477's `palp-*.spkg`.
* `export SAGE_PARALLEL_SPKG_BUILD="yes"`
* `export MAKE="make -jX"` where `X > 1`
* `make`
To build with `SAGE_CHECK="ok"`, add #7091's `sqlalchemy-*.spkg`. To make
`sage -i foo-X.Y.spkg` ''also'' append to `spkg/logs/foo-X.Y.log`, apply
[attachment:trac_8306_scripts-spkg_log_files.patch] to the scripts
repository.
--
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/8306#comment:40>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
