Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 9/11/2012 9:54 AM, Ian Stakenvicius wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/09/12 12:43 PM, Zac Medico wrote: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. I personally wonder about why this would be necessary from the perspective of the user; if the user's system at emerge time can handle X concurrent processes per emerge-job , i don't see why it would matter what phase these jobs would be launched from. At the ebuild level, certainly, but that's one of the reasons for EJOBS in the first place, so that it can be overridden consistently within a phase, if necessary for the ebuild (regardless of build system type), right? I would hope so. There are definitely real-world reasons to want to restrict jobs -- usually to just one -- in a particular phase... several ebuilds, notably several in @system, do this already by injecting -j1 somewhere. To say it's all about performance seems to be forgetting a major reason somebody might want to do this as a user or an ebuild author. Hmm... I should preface this by saying that it really pains me to say this, because now this just starts to seem so fucking complicated -- but now that I think about it, this really seems to highlight a kind of semantic discrepancy that this thread has mostly dusted under the rug. There are really TWO things we seem to care about -- one is some kind of global build-parallelism frob-set, and the other is the ability to turn off all parallelism for certain parts of an ebuild and those are actually kind-of orthogonal. In other words, imagine if we had some kind of global parallellism arbiter thingy that looked at EJOBS or ELOADAVG or maybe some other things we haven't thought of yet, and decided things like: o should portage start a new parallel emerge process? o What is the appropriate MAKEOPTS for the emake that foo-1.0.ebuild just issued? or even, one could imagine, such things as: o should portage decide that parallelism has gotten out of control somehow and suspend or kill a running ebuild for subsequent resuming or restarting? o should portage re-nice some of its processes? Note that, effectively, we have this already, and it's called portage. But one could certainly make a case for modularizing it better, since, in truth, we are talking about a very common, very abstract problem here which portage shares with any number of batch-build systems. Such an engine could very well do exactly the right thing if it were faced with a constraint that a certain part of a certain build needed to proceed without parallelism due to limitations coming from the build. Also, there are very large parts of most builds -- configure comes to mind -- that don't parallelize even if, perhaps, they should. In such cases, a really smart global parallelism arbiter could easily respond by spawning more jobs from other builds. Not sure what I'm suggesting we do about it, exactly, but just pointing out that maybe a completely correct solution requires a much more elaborate implementation than just a bunch of syntactic sugar around what we currently call MAKEOPTS. Whether or not Gentoo wants to take that all on, right now, as part of the next EAPI, is certainly debatable -- in fact, I'm inclined to say maybe it's not the best idea. Perhaps we should really be asking: is the status-quo really so problematic/inelegant that it needs fixing? -- before we decide how best to fix it. -gmt
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 12/09/12 05:55 AM, Gregory M. Turner wrote: Note that, effectively, we have this already, and it's called portage. But one could certainly make a case for modularizing it better, since, in truth, we are talking about a very common, very abstract problem here which portage shares with any number of batch-build systems. Such an engine could very well do exactly the right thing if it were faced with a constraint that a certain part of a certain build needed to proceed without parallelism due to limitations coming from the build. Also, there are very large parts of most builds -- configure comes to mind -- that don't parallelize even if, perhaps, they should. In such cases, a really smart global parallelism arbiter could easily respond by spawning more jobs from other builds. So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Advanced HPC systems (sys-cluster/torque along with an appropriate scheduler, for instance) can do such things with their jobs when the jobs are properly built; I could see portage being able to handle this as well given most of what is necessary is already known (ebuild phases, build system type (via eclass), etc). However, given the limitations already put on parallelism in terms of emerge order, etc, I could see this solution needing to be -very- complex and integration needing to occur on multiple levels. We'd also need to consider distcc (and other cluster-shared compilation methods if there are any??).. It would be an interesting project, though. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQhwAACgkQ2ugaI38ACPCKrgD+JNlHPUl7ETYDC6u3lYWRSz8J fpWC/puDfCYu51yNOVIA/0E+U6x9Ds8GV8r/RinkTqss3/fcd06w24GRvZOda3Mj =XONZ -END PGP SIGNATURE-
Re: [gentoo-dev] EJOBS variable for EAPI 5?
Il 11/09/2012 18:43, Zac Medico ha scritto: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. ok, but in my experience load-average really is too limited so I relaunch with the ability to control the following: - disk io - network - memory - cpu - jobs just tough that being able to control just jobs in a phase specific manner could have been sufficed ;-) Also this seem is a good job for containers, already implemented in the linux kernel, but will let someone with experience with them comment on the mattter.
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. Hans signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On Wed, Sep 12, 2012 at 12:33 PM, Hans de Graaff gra...@gentoo.org wrote: On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. I've been thinking about this, but that only works until you get to the huge link step of, e.g. chromium, firefox, libreoffice. I've had programs with memory leaks in the past, but I've never seen a program validly consume as much memory as ld during those builds. To cover something like that, you would need to be able to freeze and swap out an entire process (such as ld) to allow something else to complete quickly...but there's no good way I can think of to prioritize sanely between the one big process and the few dozen smaller ones which might be allowed to spawn and complete first. -- :wq
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 09/12/2012 09:33 AM, Hans de Graaff wrote: On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. Well, I think it's still good to limit the number of jobs at least, since otherwise you could become overloaded with processes that don't consume a lot of memory at first but by the time they complete they have consumed much more memory than desired (using swap). -- Thanks, Zac
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 12/09/12 12:58 PM, Zac Medico wrote: On 09/12/2012 09:33 AM, Hans de Graaff wrote: On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. Well, I think it's still good to limit the number of jobs at least, since otherwise you could become overloaded with processes that don't consume a lot of memory at first but by the time they complete they have consumed much more memory than desired (using swap). I think this would need to be dealt with by having the parent emerge process monitor all children and specifically block individual processes (ie, 'make' , 'ld' , etc) once resources are unavailable until they become so. Swap may be hit by the big processes but they wouldn't continue to be processed while in swap, at least. I don't have a solution to the potential 'thrashing' issue, though, which i expect would be a problem even if there's enough memory. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQwFoACgkQ2ugaI38ACPAiAwD/foU8Xw1BQM3jeO6IiVfCGOnw xHtIufwVmMpsGVdJQRIA/3W7Utg92foSc6ZtKMzBP5Fj0qB2BXMt/RS2I4pLsCQm =gy9K -END PGP SIGNATURE-
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 9/12/2012 5:58 AM, Ian Stakenvicius wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 12/09/12 05:55 AM, Gregory M. Turner wrote:
Note that, effectively, we have this already, and it's called
portage. But one could certainly make a case for modularizing it
better, since, in truth, we are talking about a very common, very
abstract problem here which portage shares with any number of
batch-build systems.
Such an engine could very well do exactly the right thing if it
were faced with a constraint that a certain part of a certain build
needed to proceed without parallelism due to limitations coming
from the build.
Also, there are very large parts of most builds -- configure comes
to mind -- that don't parallelize even if, perhaps, they should.
In such cases, a really smart global parallelism arbiter could
easily respond by spawning more jobs from other builds.
So essentially what you're saying here is that it might be worthwhile
to look into parallelism as a whole and possibly come up with a
solution that combines 'emerge --jobs' and build-system parallelism
together to maximum benefit?
Yeah, couldn't have said it better myself ... apparently :)
Advanced HPC systems (sys-cluster/torque along with an appropriate
scheduler, for instance) can do such things with their jobs when the
jobs are properly built; I could see portage being able to handle this
as well given most of what is necessary is already known (ebuild
phases, build system type (via eclass), etc). However, given the
limitations already put on parallelism in terms of emerge order, etc,
I could see this solution needing to be -very- complex and integration
needing to occur on multiple levels. We'd also need to consider
distcc (and other cluster-shared compilation methods if there are
any??).. It would be an interesting project, though.
ACK all of the above.
Tempting to think more deeply about this but probably the last thing I
need to do right now is to talk myself into another speculative project.
I've hurt my wrist a bit -- probably an RSI -- so should help deter me :S
Only a few major sources of parallelism exist in portage: --jobs /
--load-average in emerge opts, multiprocessing eclass equiv. ebuild
helper, distcc, and make... Infrastructure is already in place for all
of those, so perhaps a good holistic solution exists that isn't /too/
complicated.
...OK another f!#!%$^ brainstorm incoming :)
For JOBS syntax... what really seems missing in portage are:
o a clean way to say dont parallelize this particular make
invocation in ebuilds
o a clean way to globally say try to use this parallelization
strategy when emerging.
So what about something like:
o EMERGE_JOBS and EMERGE_LOAD_AVERAGE make.conf vars equiv. to
--jobs and --load-average emerge options
o EBUILD_JOBS and EBUILD_LOAD_AVERAGE make.conf vars
o If the latter are not specified, they are copied respectively from
the former (debatable for *_JOBS, since now we get 16 processes when
we asked for four).
o MAKEOPTS is auto-extended to reflect EBUILD_JOBS/EBUILD_LOAD_AVERAGE
if only if -j|--jobs|-l|--load-average options aren't provided in
make.conf/profile/envvar MAKEOPTS
o however, if MAKEOPTS override EBUILD_JOBS or EBUILD_LOAD_AVERAGE,
issue a conspicuous yellow-stars warning
o extend emake to accept a --non-parallel option which will
strip all -j|--jobs|-l|--load-average options from MAKEOPTS;
perhaps support an equivalent EBUILD_NON_PARALLEL envvar as well,
with support for override in profile.bashrc. Don't warn about this
overriding EBUILD_JOBS -- treat as SOP.
o debatable: respect EBUILD_NON_PARALLEL in multiprocessing, etc?
or, perhaps, something like:
EMAKE_NON_PARALLEL=${EMAKE_NON_PARALLEL:-${EBUILD_NON_PARALLEL:-no}}
could be used to distinguish between don't use any parallelism
and don't use GNU's make parallelism in emake. Also maybe a
better name exists that doesn't use double-negatives.
?
Seems to me something vaguely like the above would provide
o backward compatibility for ebuilds and make.conf
o not so vastly different than what we have
o a decent way to specify what we really want globally;
insofar as portage doesn't do the best job effecting the requested
parallelization strategy, more ambitious tactics could be
implemented later, hopefully without huge interface revisions.
-gmt
P.S.:
(Kind-of-crazy additional idea: put ceil(sqrt(EMERGE_JOBS)) into
EBUILD_JOBS when only the former is specified, and then let
effective_emerge_jobs equal floor(EMERGE_JOBS/EBUILD_JOBS) but maybe
too much automagic for this to be a good idea.)
Re: [gentoo-dev] EJOBS variable for EAPI 5?
Il 04/09/2012 19:15, Zac Medico ha scritto: On 09/04/2012 04:00 AM, Walter Dnes wrote: On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. One question about the suggested EJOBS variable; will it over-ride MAKEOPTS? Every so often on the Gentoo-user list, someone comes along with a mysterious build failure. The first suggestion is to reset MAKEOPTS to -j1. And on some occasions, that is indeed the solution to the mysterious build failure. That would be due to a missing dependency in the Makefiles, and using -j1 is just a workaround. The ebuild can be hardcoded to use emake -j1 until the Makefile gets fixed. I set -j1 and leave it that way. Yes, the builds take longer, but the resulting binary is just as fast. And the amount of time I save will be blown away the first time I end up screwing around a couple of hours trying to fix a mysterious build failure. That's why I want the user to have the option of over-riding EJOBS, should it ever be implemented. You could use EXTRA_EMAKE for that. You can do per-package settings via /etc/portage/package.env. Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. TIA
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. -- Thanks, Zac
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/09/12 12:43 PM, Zac Medico wrote: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. I personally wonder about why this would be necessary from the perspective of the user; if the user's system at emerge time can handle X concurrent processes per emerge-job , i don't see why it would matter what phase these jobs would be launched from. At the ebuild level, certainly, but that's one of the reasons for EJOBS in the first place, so that it can be overridden consistently within a phase, if necessary for the ebuild (regardless of build system type), right? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBPbLEACgkQ2ugaI38ACPA1qAD/bvjy7aB6nk5YboJHnCpQ8C56 QolKD9BPHL9eN8Xf41oA/iTZU+tyB+BDl+woZAlVGbaa6AR2r6Qp8rwOzkUWSwV/ =FhKc -END PGP SIGNATURE-
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 09/11/2012 09:54 AM, Ian Stakenvicius wrote: On 11/09/12 12:43 PM, Zac Medico wrote: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. I personally wonder about why this would be necessary from the perspective of the user; if the user's system at emerge time can handle X concurrent processes per emerge-job , i don't see why it would matter what phase these jobs would be launched from. Right, what matters is the system load, which is why I suggested ELOADAVG. At the ebuild level, certainly, but that's one of the reasons for EJOBS in the first place, so that it can be overridden consistently within a phase, if necessary for the ebuild (regardless of build system type), right? Right. I'm surprised that ELOADAVG wasn't proposed in tandem with EJOBS though, since overloading is not a good idea, and can happen easily any time that you doing lots of things in parallel. -- Thanks, Zac
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On Tue, Sep 11, 2012 at 1:07 PM, Zac Medico zmed...@gentoo.org wrote: On 09/11/2012 09:54 AM, Ian Stakenvicius wrote: At the ebuild level, certainly, but that's one of the reasons for EJOBS in the first place, so that it can be overridden consistently within a phase, if necessary for the ebuild (regardless of build system type), right? Right. I'm surprised that ELOADAVG wasn't proposed in tandem with EJOBS though, since overloading is not a good idea, and can happen easily any time that you doing lots of things in parallel. I tend to agree that load average matters more, although that doesn't factor in RAM use. I don't suggest that this is something that is easily remedied, but I have run into a situation where WHAT you're doing greatly influences how many jobs you can run - distcc. I once tried to implement distcc in a fairly large cluster, and then run make with VERY high levels of parallelization - such as -j32. That worked great - if the package used C. Then portage would try to build something that used ant and suddenly the host was trying to run 32 jvms in parallel - just killing the system. Ditto for python/etc. There was no way to tell the build system to go nuts with anything using distcc but not everything else. I think this is just a fundamental limitation of make - I don't suggest that we can fix this at the portage level. However, that is a use case where WHAT you're doing influences how many jobs you can run. Rich
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. One question about the suggested EJOBS variable; will it over-ride MAKEOPTS? Every so often on the Gentoo-user list, someone comes along with a mysterious build failure. The first suggestion is to reset MAKEOPTS to -j1. And on some occasions, that is indeed the solution to the mysterious build failure. Even the Gentoo manual agrees that the CPUs + 1 rule-of-thumb doesn't always work... http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#installing_portagesays... With MAKEOPTS you define how many parallel compilations should occur when you install a package. A good choice is the number of CPUs (or CPU cores) in your system plus one, ***BUT THIS GUIDELINE ISN'T ALWAYS PERFECT.*** (emphasis mine) I set -j1 and leave it that way. Yes, the builds take longer, but the resulting binary is just as fast. And the amount of time I save will be blown away the first time I end up screwing around a couple of hours trying to fix a mysterious build failure. That's why I want the user to have the option of over-riding EJOBS, should it ever be implemented. -- Walter Dnes waltd...@waltdnes.org I don't run desktop environments; I run useful applications
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 09/04/2012 04:00 AM, Walter Dnes wrote: On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. One question about the suggested EJOBS variable; will it over-ride MAKEOPTS? Every so often on the Gentoo-user list, someone comes along with a mysterious build failure. The first suggestion is to reset MAKEOPTS to -j1. And on some occasions, that is indeed the solution to the mysterious build failure. That would be due to a missing dependency in the Makefiles, and using -j1 is just a workaround. The ebuild can be hardcoded to use emake -j1 until the Makefile gets fixed. I set -j1 and leave it that way. Yes, the builds take longer, but the resulting binary is just as fast. And the amount of time I save will be blown away the first time I end up screwing around a couple of hours trying to fix a mysterious build failure. That's why I want the user to have the option of over-riding EJOBS, should it ever be implemented. You could use EXTRA_EMAKE for that. You can do per-package settings via /etc/portage/package.env. -- Thanks, Zac
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote: On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. ~harring
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Sat, Sep 1, 2012 at 8:20 PM, Brian Harring ferri...@gmail.com wrote: On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote: On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. On this point, I use dynamic load-based job counting. I'd be using distcc, too, but I don't have the spare hardware for a second build box right now. For a single box, I target a load average of $PROC_COUNT, with a max jobs in the vicinity of 2x$PROC_COUNT, to prevent I/O hangs from causing job count explosions. (Where I/O hiccups aren't a concern, I've found leaving --jobs open-ended worked fine, but builds always eventually expand to consume available I/O as upstream things get larger) I find this works very well; emerge and make both manage to lurk near the optimum point of keeping the CPU busy without spending too much time in context switches. In a distcc context, I would most likely target a load average of $LOCAL_PROC_COUNT, with a max jobs of (2*$LOCAL_PROC_COUNT+$REMOTE_PROC_COUNT). If ebuilds are try to be more aware of parallel-build contexts, I think it's important they're not limited to static job counts. -- :wq
[gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Thu, 4 Dec 2008, Diego 'Flameeyes' Pettenò wrote:
Since not all the buildsystem we support use make for the actual
build, and they don't necessarily support make-like options (-jX -s
and so on), it would be nice to be able to express a JOBS variable
that could be used for parallel build with any build systems.
Right now there are ebuilds like openoffice or some scons-based
ebuilds that parse MAKEOPTS and get out of that the number of jobs
from the -j option, but this is a) suboptimal b) error-prone.
One has to consider people might be using -l for parallel building
too, for which reasons I'd be suggesting doing something like this to
make the change transparent:
- ebuilds using non-make build systems would use JOBS;
- ebuilds using make builds systems would just use emake as usual;
- Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
- if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
- if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).
The result is that you can finally combine -l with parallel build on
OpenOffice and other packages, with a fallback number of maximum jobs
instead of using load-based decisions.
Coming back to this old topic [1]. Is there still consensus that we
should have such an EJOBS variable? (It shouldn't be called JOBS because
this name is too generic, see the old discussion.) Then we could add it
to EAPI 5.
Ulrich
[1]
http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. People using such a PM and not reading the news will get the old MAKEOPTS which will still work with makefile based build systems but will get serial builds for e.g. EAPI5 ebuilds + waf based build systems. Not a very big deal. A.
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS will have no effect for EAPI5 ebuilds, then obviously we should not be advising users to stop using MAKEOPTS until the whole tree has migrated to EAPI5. And if EJOBS will be recognized by a future version of portage for all EAPIs, then we still should allow MAKEOPTS because some users may want to use --load-average. Changing the name of MAKEOPTS in =EAPI5 makes no sense. First, because it's a standard environment variable used by gnu make. Second, because having 3 different settings for parallel building (EJOBS, MAKEOPTS, and MAKEOPTS_EAPI5) would be insane. Fancy code in the package manager would be the way to go IMHO. Ulrich's message contains a reasonable description of the algorithm. -Alexandre. signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 31/08/12 11:27 AM, Alexandre Rostovtsev wrote: On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS will have no effect for EAPI5 ebuilds, then obviously we should not be advising users to stop using MAKEOPTS until the whole tree has migrated to EAPI5. And if EJOBS will be recognized by a future version of portage for all EAPIs, then we still should allow MAKEOPTS because some users may want to use --load-average. Changing the name of MAKEOPTS in =EAPI5 makes no sense. First, because it's a standard environment variable used by gnu make. Second, because having 3 different settings for parallel building (EJOBS, MAKEOPTS, and MAKEOPTS_EAPI5) would be insane. Fancy code in the package manager would be the way to go IMHO. Ulrich's message contains a reasonable description of the algorithm. -Alexandre. I think, if i read the previous response to this correctly, that the suggestion isn't the removal of MAKEOPTS, but simply that the '-j' specification currently set in MAKEOPTS should instead be set in EJOBS in everyone's make.conf. This would then be appended to MAKEOPTS (for all EAPI) -and- be used for non-make build systems (for EAPI=5) alike. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBA4YUACgkQ2ugaI38ACPD96gD+Pu9f9SVG//0yhioO0LGP/W8o sIGpiMFIEddXvhUsDAwA/0EJkZF8jrN7zmt/LdZy3nlCGKTIkPNxp5ukUGDDWIJB =Dlem -END PGP SIGNATURE-
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 31/08/12 12:08 PM, Ian Stakenvicius wrote: On 31/08/12 11:27 AM, Alexandre Rostovtsev wrote: On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS will have no effect for EAPI5 ebuilds, then obviously we should not be advising users to stop using MAKEOPTS until the whole tree has migrated to EAPI5. And if EJOBS will be recognized by a future version of portage for all EAPIs, then we still should allow MAKEOPTS because some users may want to use --load-average. Changing the name of MAKEOPTS in =EAPI5 makes no sense. First, because it's a standard environment variable used by gnu make. Second, because having 3 different settings for parallel building (EJOBS, MAKEOPTS, and MAKEOPTS_EAPI5) would be insane. Fancy code in the package manager would be the way to go IMHO. Ulrich's message contains a reasonable description of the algorithm. -Alexandre. I think, if i read the previous response to this correctly, that the suggestion isn't the removal of MAKEOPTS, but simply that the '-j' specification currently set in MAKEOPTS should instead be set in EJOBS in everyone's make.conf. This would then be appended to MAKEOPTS (for all EAPI) -and- be used for non-make build systems (for EAPI=5) alike. ..hit send to soon... So if users stick with setting -j in MAKEOPTS, then in EAPI=5 and above this would only affect make-based builds; for parallel compilation in non-make builds they would need to convert to using EJOBS for EAPI=5 and above, otherwise those build systems will compile serially instead of in parallel. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBA4jQACgkQ2ugaI38ACPAFrAEArp7MM5w4Mv/TLKb058HzB9oN NtQeSVoCQ8X5PuxjjJ0BAKbTJXEkLlZ0hMr09RyTKzK0XtdQq6cf2fbeFFgFb5eV =+vtW -END PGP SIGNATURE-
