#11622: Add some targets to the top-level Makefile (and other make-related
issues)
-------------------------------------------------+-------------------------
Reporter: leif | Owner: leif
Type: enhancement | Status: new
Priority: major | Milestone: sage-6.9
Component: build | Resolution:
Keywords: make upgrade documentation | Merged in:
rebuild parallel sequential SAGE_CHECK | Reviewers:
SAGE_PARALLEL_SPKG_BUILD | Work issues:
Authors: | Commit:
Report Upstream: N/A | Stopgaps:
Branch: |
Dependencies: |
-------------------------------------------------+-------------------------
Old description:
> This perhaps just gets a meta-ticket to collect various additions (or
> changes), to be discussed and implemented on their own tickets.
>
> There are a few things I would like to have a `make` target for,
> partially just for convenience (at the moment random choice / incomplete,
> random order):
> * `build-check`, perhaps also `all-check`, as an "alias" for
> {{{
> #!sh
> env SAGE_CHECK=yes make build
> }}}
> * `build-parallel` or something like that to build Sage in parallel with
> a default number of jobs (preferably the number of cores or hardware
> threads), or at least (currently just) setting
> `SAGE_PARALLEL_SPKG_BUILD=yes` and doing the usual thing
> * `build-sequential` in case we one day default to a parallel build
> * similar things for `doc` (see below) and `all`
>
> '''Please add further suggestions (and the tickets implementing them).'''
>
> ----
>
> Related:
>
> * As of Sage 4.7.1.rc0, running `./sage -b` before running tests is
> currently not performed consistently. It's perhaps discussable whether we
> should do that in ''all'' test receipts or none of them, or provide
> ''separate'' targets which do that (which is a trivial task except for
> the choice of names).
>
> * The current behaviour of `devel/sage/setup.py` w.r.t. to parallel
> building is IMHO undesirable or sub-optimal:
> * It uses the number of hardware threads even if `MAKE` isn't set
> (''of course^TM^'' regardless of the system load). (If `MAKE` is set to
> e.g. `make -j`, which means an "unlimited" number of jobs, it uses twice
> the number of hardware threads.) [[BR]]
> One can of course circumvent that by setting `MAKE` to `make -j1`,
> but that's IMHO kind of odd.
> * It complains about an ''"impossibly large number of threads"'' if
> the number of jobs (`-j...`) found in `$MAKE` exceeds twice the number of
> hardware threads, where the number of hardware threads is equal to the
> number of cores on processors without what Intel calls Hyper-Threading
> (HT), then falling back to (at most) the number of hardware threads.
>
> * We should stop unsetting `MAKEFLAGS` and setting `MAKE` to `make`
> whenever `SAGE_PARALLEL_SPKG_BUILD!=yes` in `spkg/install`, which is
> simply wrong for a couple of reasons:
> * It invalidates e.g. `MAKE=gmake`.
> * It clears all flags passed to `make` on the command line as well as
> those eventually added in `MAKE` (or to `MAKEFLAGS`), e.g. `-d`, `-k`
> (`--keep-going`), and even `-n` (`--dry-run`) and `-q`. [[BR]]
>
> It should instead (if at all) just add `-j1` to `MAKE` if
> `SAGE_PARALLEL_SPKG_BUILD!=yes`. [[BR]]
>
> Oh, just noticed this has meanwhile slightly changed (but don't want to
> rewrite all the above):
> {{{
> #!sh
> if [ "$SAGE_PARALLEL_SPKG_BUILD" = "yes" ] && [ -n "$MAKE" ]; then
> time $MAKE -f standard/deps $1
> else
> time make -f standard/deps $1
> fi
> }}}
> So we do no longer clear `MAKEFLAGS`, but not using `$MAKE` there
> remains. (And this won't prevent a parallel build invoked through `-j`
> without a number of jobs.) Note that `MAKE` will always be set at that
> point, unless some user tries to call `spkg/install` directly (or doesn't
> use GNU `make`), which is an error condition by itself, to be catched
> separately. [[BR]]
>
> '''Note:''' This is meanwhile also fixed by #11959, i.e.
> `${MAKE:-make}` is now consistently used.
>
> * The sequence top-level Makefile (`make`) -> `spkg/install` (shell
> script) -> `spkg/standard/deps`, another Makefile run by a ''different''
> `make` instance is IMHO bad and not necessary anyway, since it is
> possible to set all environment (or `make`) variables in
> `spkg/standard/deps` (ill name for a Makefile that isn't included by
> another one) itself. [[BR]]
> Changing this is of course a bigger task. (We have to keep the two-
> stage make procedure, i.e. two separate Makefiles, to support upgrading
> at least, which I think isn't bad since it is more modular.)
>
> ----
>
> In case we disagree or have contrary opinions on / suggestions for some
> changes or in design, we can move the discussion of ''individual
> aspects'' to `sage-devel` (or `sage-release` :-) ) threads, but we IMHO
> shouldn't move the ''whole'' discussion there. There'll certainly be
> follow-up or "real" (non-meta) tickets anyway, so it should be possible
> to not mix up different aspects in a single thread.
New description:
This perhaps just gets a meta-ticket to collect various additions (or
changes), to be discussed and implemented on their own tickets.
There are a few things I would like to have a `make` target for, partially
just for convenience (at the moment random choice / incomplete, random
order):
* `build-check`, perhaps also `all-check`, as an "alias" for
{{{
#!sh
env SAGE_CHECK=yes make build
}}}
* `build-parallel` or something like that to build Sage in parallel with
a default number of jobs (preferably the number of cores or hardware
threads), or at least (currently just) setting
`SAGE_PARALLEL_SPKG_BUILD=yes` and doing the usual thing
* `build-sequential` in case we one day default to a parallel build
* similar things for `doc` (see below) and `all`
'''Please add further suggestions (and the tickets implementing them).'''
----
Related:
* The current behaviour of `devel/sage/setup.py` w.r.t. to parallel
building is IMHO undesirable or sub-optimal:
* It uses the number of hardware threads even if `MAKE` isn't set (''of
course^TM^'' regardless of the system load). (If `MAKE` is set to e.g.
`make -j`, which means an "unlimited" number of jobs, it uses twice the
number of hardware threads.) [[BR]]
One can of course circumvent that by setting `MAKE` to `make -j1`,
but that's IMHO kind of odd.
* It complains about an ''"impossibly large number of threads"'' if the
number of jobs (`-j...`) found in `$MAKE` exceeds twice the number of
hardware threads, where the number of hardware threads is equal to the
number of cores on processors without what Intel calls Hyper-Threading
(HT), then falling back to (at most) the number of hardware threads.
* We should stop unsetting `MAKEFLAGS` and setting `MAKE` to `make`
whenever `SAGE_PARALLEL_SPKG_BUILD!=yes` in `spkg/install`, which is
simply wrong for a couple of reasons:
* It invalidates e.g. `MAKE=gmake`.
* It clears all flags passed to `make` on the command line as well as
those eventually added in `MAKE` (or to `MAKEFLAGS`), e.g. `-d`, `-k`
(`--keep-going`), and even `-n` (`--dry-run`) and `-q`. [[BR]]
It should instead (if at all) just add `-j1` to `MAKE` if
`SAGE_PARALLEL_SPKG_BUILD!=yes`. [[BR]]
Oh, just noticed this has meanwhile slightly changed (but don't want to
rewrite all the above):
{{{
#!sh
if [ "$SAGE_PARALLEL_SPKG_BUILD" = "yes" ] && [ -n "$MAKE" ]; then
time $MAKE -f standard/deps $1
else
time make -f standard/deps $1
fi
}}}
So we do no longer clear `MAKEFLAGS`, but not using `$MAKE` there
remains. (And this won't prevent a parallel build invoked through `-j`
without a number of jobs.) Note that `MAKE` will always be set at that
point, unless some user tries to call `spkg/install` directly (or doesn't
use GNU `make`), which is an error condition by itself, to be catched
separately. [[BR]]
'''Note:''' This is meanwhile also fixed by #11959, i.e.
`${MAKE:-make}` is now consistently used.
* The sequence top-level Makefile (`make`) -> `spkg/install` (shell
script) -> `spkg/standard/deps`, another Makefile run by a ''different''
`make` instance is IMHO bad and not necessary anyway, since it is possible
to set all environment (or `make`) variables in `spkg/standard/deps` (ill
name for a Makefile that isn't included by another one) itself. [[BR]]
Changing this is of course a bigger task. (We have to keep the two-
stage make procedure, i.e. two separate Makefiles, to support upgrading at
least, which I think isn't bad since it is more modular.)
----
In case we disagree or have contrary opinions on / suggestions for some
changes or in design, we can move the discussion of ''individual aspects''
to `sage-devel` (or `sage-release` :-) ) threads, but we IMHO shouldn't
move the ''whole'' discussion there. There'll certainly be follow-up or
"real" (non-meta) tickets anyway, so it should be possible to not mix up
different aspects in a single thread.
--
Comment (by jdemeyer):
Replying to [ticket:11622 leif]:
> It's perhaps discussable whether we should do that in ''all'' test
receipts
It is done in all test targets.
--
Ticket URL: <http://trac.sagemath.org/ticket/11622#comment:10>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/d/optout.
