#11622: Add some targets to the top-level Makefile (and other make-related
issues)
-------------------------------------------------+-------------------------
Reporter: leif | Owner: leif
Type: enhancement | Status: new
Priority: major | Milestone: sage-6.9
Component: build | Resolution:
Keywords: make upgrade documentation | Merged in:
rebuild parallel sequential SAGE_CHECK | Reviewers:
SAGE_PARALLEL_SPKG_BUILD | Work issues:
Authors: | Commit:
Report Upstream: N/A | Stopgaps:
Branch: |
Dependencies: |
-------------------------------------------------+-------------------------
Old description:
> This perhaps just gets a meta-ticket to collect various additions (or
> changes), to be discussed and implemented on their own tickets.
>
> There are a few things I would like to have a `make` target for,
> partially just for convenience (at the moment random choice / incomplete,
> random order):
> * `build-check`, perhaps also `all-check`, as an "alias" for
> {{{
> #!sh
> env SAGE_CHECK=yes make build
> }}}
> * `build-check-ignore` once we support `SAGE_CHECK=ignore` to (build
> and) run all test suites without treating failed tests as fatal errors
> (since e.g. the Python test suite fails on many systems)
> * `build-parallel` or something like that to build Sage in parallel with
> a default number of jobs (preferably the number of cores or hardware
> threads), or at least (currently just) setting
> `SAGE_PARALLEL_SPKG_BUILD=yes` and doing the usual thing
> * `build-sequential` in case we one day default to a parallel build
> [[BR]]
> '''Note:''' #11959 now enables parallel spkg building by default, and
> adds a target `build-serial`.
> * similar things for `doc` (see below) and `all`
> * perhaps `library` as an alias for `./sage -b`
> * ...
>
> I could imagine to also have
> * different targets to build the documentation sequentially or in
> parallel (cf. #6495),
> * ...
>
> '''Please add further suggestions (and the tickets implementing them).'''
>
> ----
>
> Related:
>
> * As of Sage 4.7.1.rc0, running `./sage -b` before running tests is
> currently not performed consistently. It's perhaps discussable whether we
> should do that in ''all'' test receipts or none of them, or provide
> ''separate'' targets which do that (which is a trivial task except for
> the choice of names).
>
> * The current behaviour of `devel/sage/setup.py` w.r.t. to parallel
> building is IMHO undesirable or sub-optimal:
> * It uses the number of hardware threads even if `MAKE` isn't set
> (''of course^TM^'' regardless of the system load). (If `MAKE` is set to
> e.g. `make -j`, which means an "unlimited" number of jobs, it uses twice
> the number of hardware threads.) [[BR]]
> One can of course circumvent that by setting `MAKE` to `make -j1`,
> but that's IMHO kind of odd.
> * It complains about an ''"impossibly large number of threads"'' if
> the number of jobs (`-j...`) found in `$MAKE` exceeds twice the number of
> hardware threads, where the number of hardware threads is equal to the
> number of cores on processors without what Intel calls Hyper-Threading
> (HT), then falling back to (at most) the number of hardware threads.
>
> * We should stop unsetting `MAKEFLAGS` and setting `MAKE` to `make`
> whenever `SAGE_PARALLEL_SPKG_BUILD!=yes` in `spkg/install`, which is
> simply wrong for a couple of reasons:
> * It invalidates e.g. `MAKE=gmake`.
> * It clears all flags passed to `make` on the command line as well as
> those eventually added in `MAKE` (or to `MAKEFLAGS`), e.g. `-d`, `-k`
> (`--keep-going`), and even `-n` (`--dry-run`) and `-q`. [[BR]]
>
> It should instead (if at all) just add `-j1` to `MAKE` if
> `SAGE_PARALLEL_SPKG_BUILD!=yes`. [[BR]]
>
> Oh, just noticed this has meanwhile slightly changed (but don't want to
> rewrite all the above):
> {{{
> #!sh
> if [ "$SAGE_PARALLEL_SPKG_BUILD" = "yes" ] && [ -n "$MAKE" ]; then
> time $MAKE -f standard/deps $1
> else
> time make -f standard/deps $1
> fi
> }}}
> So we do no longer clear `MAKEFLAGS`, but not using `$MAKE` there
> remains. (And this won't prevent a parallel build invoked through `-j`
> without a number of jobs.) Note that `MAKE` will always be set at that
> point, unless some user tries to call `spkg/install` directly (or doesn't
> use GNU `make`), which is an error condition by itself, to be catched
> separately. [[BR]]
>
> '''Note:''' This is meanwhile also fixed by #11959, i.e.
> `${MAKE:-make}` is now consistently used.
>
> * The sequence top-level Makefile (`make`) -> `spkg/install` (shell
> script) -> `spkg/standard/deps`, another Makefile run by a ''different''
> `make` instance is IMHO bad and not necessary anyway, since it is
> possible to set all environment (or `make`) variables in
> `spkg/standard/deps` (ill name for a Makefile that isn't included by
> another one) itself. [[BR]]
> Changing this is of course a bigger task. (We have to keep the two-
> stage make procedure, i.e. two separate Makefiles, to support upgrading
> at least, which I think isn't bad since it is more modular.)
>
> ----
>
> In case we disagree or have contrary opinions on / suggestions for some
> changes or in design, we can move the discussion of ''individual
> aspects'' to `sage-devel` (or `sage-release` :-) ) threads, but we IMHO
> shouldn't move the ''whole'' discussion there. There'll certainly be
> follow-up or "real" (non-meta) tickets anyway, so it should be possible
> to not mix up different aspects in a single thread.
New description:
This perhaps just gets a meta-ticket to collect various additions (or
changes), to be discussed and implemented on their own tickets.
There are a few things I would like to have a `make` target for, partially
just for convenience (at the moment random choice / incomplete, random
order):
* `build-check`, perhaps also `all-check`, as an "alias" for
{{{
#!sh
env SAGE_CHECK=yes make build
}}}
* `build-parallel` or something like that to build Sage in parallel with
a default number of jobs (preferably the number of cores or hardware
threads), or at least (currently just) setting
`SAGE_PARALLEL_SPKG_BUILD=yes` and doing the usual thing
* `build-sequential` in case we one day default to a parallel build
* similar things for `doc` (see below) and `all`
'''Please add further suggestions (and the tickets implementing them).'''
----
Related:
* As of Sage 4.7.1.rc0, running `./sage -b` before running tests is
currently not performed consistently. It's perhaps discussable whether we
should do that in ''all'' test receipts or none of them, or provide
''separate'' targets which do that (which is a trivial task except for the
choice of names).
* The current behaviour of `devel/sage/setup.py` w.r.t. to parallel
building is IMHO undesirable or sub-optimal:
* It uses the number of hardware threads even if `MAKE` isn't set (''of
course^TM^'' regardless of the system load). (If `MAKE` is set to e.g.
`make -j`, which means an "unlimited" number of jobs, it uses twice the
number of hardware threads.) [[BR]]
One can of course circumvent that by setting `MAKE` to `make -j1`,
but that's IMHO kind of odd.
* It complains about an ''"impossibly large number of threads"'' if the
number of jobs (`-j...`) found in `$MAKE` exceeds twice the number of
hardware threads, where the number of hardware threads is equal to the
number of cores on processors without what Intel calls Hyper-Threading
(HT), then falling back to (at most) the number of hardware threads.
* We should stop unsetting `MAKEFLAGS` and setting `MAKE` to `make`
whenever `SAGE_PARALLEL_SPKG_BUILD!=yes` in `spkg/install`, which is
simply wrong for a couple of reasons:
* It invalidates e.g. `MAKE=gmake`.
* It clears all flags passed to `make` on the command line as well as
those eventually added in `MAKE` (or to `MAKEFLAGS`), e.g. `-d`, `-k`
(`--keep-going`), and even `-n` (`--dry-run`) and `-q`. [[BR]]
It should instead (if at all) just add `-j1` to `MAKE` if
`SAGE_PARALLEL_SPKG_BUILD!=yes`. [[BR]]
Oh, just noticed this has meanwhile slightly changed (but don't want to
rewrite all the above):
{{{
#!sh
if [ "$SAGE_PARALLEL_SPKG_BUILD" = "yes" ] && [ -n "$MAKE" ]; then
time $MAKE -f standard/deps $1
else
time make -f standard/deps $1
fi
}}}
So we do no longer clear `MAKEFLAGS`, but not using `$MAKE` there
remains. (And this won't prevent a parallel build invoked through `-j`
without a number of jobs.) Note that `MAKE` will always be set at that
point, unless some user tries to call `spkg/install` directly (or doesn't
use GNU `make`), which is an error condition by itself, to be catched
separately. [[BR]]
'''Note:''' This is meanwhile also fixed by #11959, i.e.
`${MAKE:-make}` is now consistently used.
* The sequence top-level Makefile (`make`) -> `spkg/install` (shell
script) -> `spkg/standard/deps`, another Makefile run by a ''different''
`make` instance is IMHO bad and not necessary anyway, since it is possible
to set all environment (or `make`) variables in `spkg/standard/deps` (ill
name for a Makefile that isn't included by another one) itself. [[BR]]
Changing this is of course a bigger task. (We have to keep the two-
stage make procedure, i.e. two separate Makefiles, to support upgrading at
least, which I think isn't bad since it is more modular.)
----
In case we disagree or have contrary opinions on / suggestions for some
changes or in design, we can move the discussion of ''individual aspects''
to `sage-devel` (or `sage-release` :-) ) threads, but we IMHO shouldn't
move the ''whole'' discussion there. There'll certainly be follow-up or
"real" (non-meta) tickets anyway, so it should be possible to not mix up
different aspects in a single thread.
--
Comment (by jdemeyer):
Replying to [ticket:11622 leif]:
> * `build-check-ignore` once we support `SAGE_CHECK=ignore` to (build
and) run all test suites without treating failed tests as fatal errors
(since e.g. the Python test suite fails on many systems)
We don't have {{{SAGE_CHECK=ignore}}} yet, so this isn't applicable.
> * `build-sequential` in case we one day default to a parallel build
Well, we don't build in parallel by default.
> * perhaps `library` as an alias for `./sage -b`
Very bad idea. We shouldn't encourage `./sage -b`, just run `make` or
`make build` or `make sagelib` if needed.
--
Ticket URL: <http://trac.sagemath.org/ticket/11622#comment:9>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/d/optout.
