#12016: parallelism in Sage: just use value of 'MAKE'
---------------------------------------------+------------------------------
Reporter: jhpalmieri | Owner: GeorgSWeber
Type: enhancement | Status: needs_review
Priority: critical | Milestone: sage-4.8
Component: build | Keywords:
Work_issues: | Upstream: N/A
Reviewer: John Palmieri, Jeroen Demeyer | Author: John Palmieri,
Jeroen Demeyer
Merged: | Dependencies: sage-4.8.alpha3
+ #12096, #12137, #12138, #12139
---------------------------------------------+------------------------------
Old description:
> The various parallel aspects of Sage should be controlled by setting the
> `-j` (possible also `-l`) flags in `MAKE` or `MAKEFLAGS`. That is, if
> `MAKE='make -j16'`, then
>
> - running `make` will build spkg's in parallel, using 16 processes (this
> was done in #11959). This is standard `make` behaviour, but we need to
> patch `spkg/standard/deps` to ensure that `make` recognizes that we are
> doing a recursive make.
>
> - running `make ptestlong` or `sage -tp 0 <files>` will doctest in
> parallel using 16 threads. If the `-j` flag in `MAKE` is not set, then
> determine the number of threads as before: `min(8, cpu_count())`.
>
> - running `./sage -b` will build the Sage library using 16 threads. If
> the `-j` flag in `MAKE` is not set, then use only 1 thread.
>
> '''Testing this ticket''': you can set the environment variable
> `SAGE_NUM_CORES` to the number of cores you want to pretend to have. For
> example, running
> {{{
> SAGE_NUM_CORES=24 make ptestlong
> }}}
> should run 8 threads (see `sage-num-threads.py`; this is undocumented
> because the only purpose I see is for testing this ticket).
>
> '''Notes''':
> With the patches applied, building spkgs in parallel works well, except
> for race conditions in:
> * python (#12096)
> * singular (#12137)
> * zlib (#12138)
> * mpir (#12139)
> and a "jobserver unavailable" warning in:
> * ntl
> * singular (#12137)
> * rubiks
>
> '''Apply''':
> 1. [attachment:12016-root.patch] to the `SAGE_ROOT` repository.
> 1. [attachment:12016-base.patch] to `spkg/base`.
> 1. [attachment:12016-scripts.patch] to the `SCRIPTS` repository.
> 1. [attachment:12016-sage.patch] to the Sage library.
>
> See also: #6495 to implement the same behavior for doc building.
New description:
The various parallel aspects of Sage should be controlled by setting the
`-j` (possible also `-l`) flags in `MAKE` or `MAKEFLAGS`. That is, if
`MAKE='make -j16'`, then
- running `make` will build spkg's in parallel, using 16 processes (this
was done in #11959). This is standard `make` behaviour, but we need to
patch `spkg/standard/deps` to ensure that `make` recognizes that we are
doing a recursive make.
- running `make ptestlong` or `sage -tp 0 <files>` will doctest in
parallel using 16 threads. If the `-j` flag in `MAKE` is not set, then
determine the number of threads as before: `min(8, cpu_count())`.
- running `./sage -b` will build the Sage library using 16 threads. If
the `-j` flag in `MAKE` is not set, then use only 1 thread.
'''Testing this ticket''': you can set the environment variable
`SAGE_NUM_CORES` to the number of cores you want to pretend to have. For
example, running
{{{
SAGE_NUM_CORES=24 make ptestlong
}}}
should run 8 threads (see `sage-num-threads.py`; this is undocumented
because the only purpose I see is for testing this ticket).
'''Notes''':
With the patches applied, building spkgs in parallel works well, except
for race conditions in:
* python (#12096)
* singular (#12137)
* zlib (#12138)
* mpir (#12139)
and a "jobserver unavailable" warning in:
* ntl
* singular (#12137)
* rubiks
'''Apply''':
1. [attachment:12016-root.patch] to the `SAGE_ROOT` repository.
1. [attachment:12016-base.patch] to `spkg/base`.
1. [attachment:12016-scripts.patch] and [attachment:trac_12016-scripts-
ref.patch] to the `SCRIPTS` repository.
1. [attachment:12016-sage.patch] to the Sage library.
See also: #6495 to implement the same behavior for doc building.
--
Comment(by jhpalmieri):
I'm happy with this except for a few small changes I want to make in sage-
num-threads.py: we should use subprocess instead of popen, since popen has
been deprecated. Also, we should catch errors if `sysctl` fails to run --
it's not present on all platforms. Finally, we might as well search for
`max-load` in addition to `load-average`. See the referee patch. If
you're happy with that, the whole thing can get a positive review.
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/12016#comment:49>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
