#12016: parallelism in Sage: just use value of 'MAKE'
---------------------------+------------------------------------------------
Reporter: jhpalmieri | Owner: GeorgSWeber
Type: enhancement | Status: needs_review
Priority: major | Milestone: sage-4.8
Component: build | Keywords:
Work_issues: | Upstream: N/A
Reviewer: | Author: John Palmieri, Jeroen Demeyer
Merged: | Dependencies: #11969
---------------------------+------------------------------------------------
Changes (by jhpalmieri):
* status: new => needs_review
Old description:
> With the attached patches, along with the changes from #11959, the
> various parallel aspects of Sage should be controlled by setting the `-j`
> flag in `MAKE`. That is, if `MAKE='make -j16'`, then
>
> - running `make` will build spkg's in parallel, using 16 processes (this
> was done in #11959). ''This is standard `make` behaviour, no patches are
> needed.''
>
> - running `make ptestlong` or `sage -tp 0 <files>` will doctest in
> parallel using 16 threads. If the `-j` flag in `MAKE` is not set, then
> determine the number of threads as before: `min(8, cpu_count())`.
>
> - '''TODO''': running `./sage -b` will build the Sage library using 16
> threads. If the `-j` flag in `MAKE` is not set, then use only 1 thread.
>
> In #6495, we should implement the same behavior for doc building.
>
> '''TODO:''' We should also support:
> - `make -j16 ptestlong` (as opposed to `MAKE="make -j16" make
> ptestlong`). Use `MAKEFLAGS`.
> - `make -j` (unlimited number of jobs, say set to 9999 if we really need
> a number).
> - `make -j37 -j1` (last option takes precedence).
>
> '''Apply''':
> 1. [attachment:12016-root.v2.patch] to the `SAGE_ROOT` repository.
> 1. [attachment:12016-scripts.v2.patch] to the `SCRIPTS` repository.
> 1. [attachment:12016-sage.v2.patch] to the Sage library.
New description:
With the attached patches, along with the changes from #11959, the various
parallel aspects of Sage should be controlled by setting the `-j` flag in
`MAKE`. That is, if `MAKE='make -j16'`, then
- running `make` will build spkg's in parallel, using 16 processes (this
was done in #11959). ''This is standard `make` behaviour, no patches are
needed.''
- running `make ptestlong` or `sage -tp 0 <files>` will doctest in
parallel using 16 threads. If the `-j` flag in `MAKE` is not set, then
determine the number of threads as before: `min(8, cpu_count())`.
- running `./sage -b` will build the Sage library using 16 threads. If
the `-j` flag in `MAKE` is not set, then use only 1 thread.
In #6495, we should implement the same behavior for doc building.
'''Apply''':
1. [attachment:trac_12016-root.v2.patch] to the `SAGE_ROOT` repository.
1. [attachment:trac_12016-scripts.v2.patch] to the `SCRIPTS` repository.
1. [attachment:trac_12016-sage.v2.patch] to the Sage library.
--
Comment:
Here are new patches. These use `SAGE_NUM_THREADS` if it is set, and
otherwise try to extract a number from `MAKE`. (My method for doing this
is probably not ideal, but the options This is done in sage-env. Running
`sage -b` should use this setting now, also.
I don't know how to get the number of threads from
{{{
make -j16 ptestlong
}}}
so I removed that from the "to do" list in the ticket description.
In the file sage-ptest, I removed the "FIXME" comment in
{{{
#!python
try:
# FIXME: Nice, but <NUMTHREADS> should immediately follow '-tp'
etc.,
# i.e., be the next argument. We might have file or
directory
# names that properly convert to an int...
numthreads = int(argv[1])
infiles = argv[2:]
except ValueError: # can't convert first arg to an integer: arg was
probably omitted
numthreads = 1
}}}
The script sage-ptest doesn't get a "tp" argument; it is instead called by
sage-sage, and the way it is called, the first argument to sage-ptest is
precisely what ever came after "-tp". So I don't think anything needs
fixing. If we ever rewrite sage-sage (#21) to properly parse arguments,
we can make sure that "-tp" has a default numerical argument of zero.
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/12016#comment:11>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
