[gentoo-portage-dev] Max parallelization setting
Resending to portage dev ml to solicit comments...
Thoughts welcome, since it's an issue that must be dealt with to
sanely do efficient parallelization (monkeying with make.conf and
hardcoding vals isn't much of a solution).
Zac- comments? You seem to have totally missed the original email ;)
-
Date: Sun, 1 Oct 2006 12:27:13 -0700
To: gentoo-dev@lists.gentoo.org
From: Brian Harring [EMAIL PROTECTED]
Subject: Re: [gentoo-dev] Setting number of parallel builds for other
build-systems than 'make'
On Sun, Oct 01, 2006 at 09:52:14AM -0700, Donnie Berkholz wrote:
Tiziano Müller wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi everyone
It seems that setting the number of parallel builds using '-jN' does not
only work for make, but also for scons and bjam (and maybe others as well).
Since it isn't save to assume that '-jN' is the only option in MAKEOPTS,
some filtering is needed.
Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf and used
whenever one of those build-systems is being used. But we would probably
have to add a ...OPTS for every build-system.
What do you think? Better ideas?
I think adding it for each build system is probably a good idea,
nobody's guaranteeing option-level compatibility with make. Optionally
falling back to using the few valid MAKEOPTS might be a nice addition.
I might be daft (likely), but why not just introduce a var indicating
max parallelization instead? Tweak portage to push that setting into
MAKEOPTS=${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}.
Might sound daft, but -j is hardcoded parallelization; if you're
trying to run 3 build processes, the original var of -j isn't all that
useful, nor will the hardcoded -j# var set for 3 package build
processes be useful if you're doing a single build.
Depending on number of seperate package build processes possible, the
# of build processes allocated per build process needs to vary
(essentially).
Now... that's a bit ahead of what portage is doing atm, but folks
*will* parallelize portage building (see bug 147516 if in doubt), so
its not too far out.
Getting back to the actual topic, set this var, it's a raw int that a
scons/bjam eclass can use to easily set appropriate var with the
parallelization requested, thus unifying this particular knob; more
importantly, it gives them a way to do what they're after while
exposing a knob that the pkg manager can easily fiddle with for global
parallelization needs.
Only downside to it I see is that it requires mangling the pkg manager
to translate the parallelization setting into MAKEOPTS+=-j#; can't
really get around that however due to the fact MAKEOPTS is already
forced in installed configuration though.
Thoughts?
~harring
pgpZOWWRmNkmO.pgp
Description: PGP signature
Re: [gentoo-portage-dev] Max parallelization setting
On Tue, Oct 10, 2006 at 03:20:55AM -0700, Zac Medico wrote:
Brian Harring wrote:
I might be daft (likely), but why not just introduce a var indicating
max parallelization instead? Tweak portage to push that setting into
MAKEOPTS=${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}.
The idea sounds good, but I'm not clear on all the details. It
seems like there are several distinct parts:
1) Ebuild maintainter sets a metadata variable to indicate the level
of parallelization possible in a build.
RESTRICT moreso indicating if it's parallelizable or not.
2) Gentoo user sets a configuration option indicating the maximum
level of parallelization spread across multiple builds at a given time.
3) Package manager uses the user's config to allocate an appropriate
PARALLELIZATION at build time, based on 1 and 2 above. Then the
src_compile() function of the ebuild translates PARALLELIZATION into
the appropriate build system flags (possibly with the help of an
eclass function).
Not necessarily src_compile, but close enough; the details of how
MAX_PARALLELIZATION gets shoved in is semi package specific, although
it's kind of implicit at this point that -j# for MAKEOPTS would likely
be directly fooled with.
For other build systems, rely on eclasses handling it; for MAKEOPTS,
would be preferable to do the same imo, but that's not an easy
transition.
Mind you since there isn't a way to adjust the allowed slices
(essentially) while a compile is underway, this won't hit 100%
utilization- further, src_install still abides by MAKEOPTS, but it's
not like -jN is going to help much there.
That said, it's better then the current crapshoot required for trying
to do parallel builds; either you have to monkey patch make.conf
everytime, or try env overrides for it, both of which aren't
incredibly friendly/simple if you're just trying to do an upgrade that
abuses your duo/quad.
~harring
pgpZKzky1JHbB.pgp
Description: PGP signature
Re: [gentoo-portage-dev] Max parallelization setting
On Tue, 10 Oct 2006 00:04:57 -0700
Brian Harring [EMAIL PROTECTED] wrote:
I might be daft (likely), but why not just introduce a var indicating
max parallelization instead? Tweak portage to push that setting into
MAKEOPTS=${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}.
Might sound daft, but -j is hardcoded parallelization; if you're
trying to run 3 build processes, the original var of -j isn't all
that useful, nor will the hardcoded -j# var set for 3 package build
processes be useful if you're doing a single build.
Depending on number of seperate package build processes possible, the
# of build processes allocated per build process needs to vary
(essentially).
Seems useful as *alternative* to the low level vars, but shouldn't
replace them (so if the low level vars are set they override the global
setting). So the order in your MAKEOPTS assignment above would have to
be reversed (assuming sequential option parsing).
Marius
--
Public Key at http://www.genone.de/info/gpg-key.pub
In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
--
gentoo-portage-dev@gentoo.org mailing list
