#12016: parallelism in Sage: just use value of 'MAKE'
---------------------------------------------+------------------------------
Reporter: jhpalmieri | Owner: GeorgSWeber
Type: enhancement | Status: needs_review
Priority: critical | Milestone: sage-4.8
Component: build | Keywords:
Work_issues: | Upstream: N/A
Reviewer: John Palmieri, Jeroen Demeyer | Author: John Palmieri,
Jeroen Demeyer
Merged: | Dependencies: sage-4.8.alpha3
+ #12096
---------------------------------------------+------------------------------
Changes (by jdemeyer):
* status: needs_work => needs_review
* reviewer: => John Palmieri, Jeroen Demeyer
Old description:
> With the attached patches, along with the changes from #11959, the
> various parallel aspects of Sage should be controlled by setting the `-j`
> flag in `MAKE`. That is, if `MAKE='make -j16'`, then
>
> - running `make` will build spkg's in parallel, using 16 processes (this
> was done in #11959). This is standard `make` behaviour, but we need to
> patch `spkg/standard/deps` to ensure that `make` recognizes that we are
> doing a recursive make.
>
> - running `make ptestlong` or `sage -tp 0 <files>` will doctest in
> parallel using 16 threads. If the `-j` flag in `MAKE` is not set, then
> determine the number of threads as before: `min(8, cpu_count())`.
>
> - running `./sage -b` will build the Sage library using 16 threads. If
> the `-j` flag in `MAKE` is not set, then use only 1 thread.
>
> In #6495, we should implement the same behavior for doc building.
>
> '''Apply''':
> 1. [attachment:12016-root.patch] to the `SAGE_ROOT` repository.
> 1. [attachment:12016-base.patch] to `spkg/base`.
> 1. [attachment:12016-scripts.patch] to the `SCRIPTS` repository.
> 1. [attachment:12016-sage.patch] to the Sage library.
>
> '''Notes''':
> With the patches applied, building spkgs in parallel works well, except
> for a "jobserver unavailable" warning in:
> * ntl
> * singular
> * rubiks
New description:
With the attached patches, along with the changes from #11959, the various
parallel aspects of Sage should be controlled by setting the `-j` flag in
`MAKE`. That is, if `MAKE='make -j16'`, then
- running `make` will build spkg's in parallel, using 16 processes (this
was done in #11959). This is standard `make` behaviour, but we need to
patch `spkg/standard/deps` to ensure that `make` recognizes that we are
doing a recursive make.
- running `make ptestlong` or `sage -tp 0 <files>` will doctest in
parallel using 16 threads. If the `-j` flag in `MAKE` is not set, then
determine the number of threads as before: `min(8, cpu_count())`.
- running `./sage -b` will build the Sage library using 16 threads. If
the `-j` flag in `MAKE` is not set, then use only 1 thread.
In #6495, we should implement the same behavior for doc building.
Concerning testing this ticket: you can set the environment variable
`SAGE_NUM_CORES` to the number of cores you want to pretend to have. For
example, running
{{{
SAGE_NUM_CORES=24 make ptestlong
}}}
should run 8 threads (see `sage-num-threads.py`; this is undocumented
because the only purpose I see is for testing this ticket).
'''Apply''':
1. [attachment:12016-root.patch] to the `SAGE_ROOT` repository.
1. [attachment:12016-base.patch] to `spkg/base`.
1. [attachment:12016-scripts.patch] to the `SCRIPTS` repository.
1. [attachment:12016-sage.patch] to the Sage library.
'''Notes''':
With the patches applied, building spkgs in parallel works well, except
for a "jobserver unavailable" warning in:
* ntl
* singular
* rubiks
--
Comment:
Replying to [comment:34 jhpalmieri]:
> This looks good to me. Is it ready for review?
Yes, it is. I just didn't want to put "needs review" because I have not
really tested it.
> Am I allowed to review it since I wrote early drafts of some of the
patches?
I would say yes, since I certainly looked at all your code. So consider
all your code to be positively reviewed by me.
> For a future ticket, it would be nice if you could set `MAKE='make -j
-lN'`, for some reasonable choice of `N`, and have it work. When I try
this, I have problems with the following spkgs, and I'm not sure why:
It '''should''' work.
> - zlib on OS X (2 cores) fails most of the time with `MAKE='make -j
-l3'`. [http://sage.math.washington.edu/home/palmieri/misc/zlib-1.2.5.log
Here's a log.]
> - singular on sage.math fails all of the time, I think, with
`MAKE='make -j -l30'`.
[http://sage.math.washington.edu/home/palmieri/misc/singular-3-1-3-3.p1.log
Here's a log.]
Question for both cases: does MAKE="make -jN" work for various values of
N? Because I don't see a fundamental difference between "make -jN" and
"make -j -lN".
> (The Sage spkg used to fail before the latest round of patches using the
`-l` setting for a cap on the number of threads, and the same goes for
parallel doctesting. Setting `MAKE='make -j'` still causes these
failures. Should we just regard this setting as too dangerous?
Yes. Since "make" will simply run as many threads as it can, I think Sage
should do the same, no matter how stupid that is.
By the way, I would really like to merge this in the sage-4.8 release,
because it cleans up some stuff which will also help future tickets like
#11073 (which hopefully will be merged in sage-5.0).
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/12016#comment:35>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
