Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. One question about the suggested EJOBS variable; will it over-ride MAKEOPTS? Every so often on the Gentoo-user list, someone comes along with a mysterious build failure. The first suggestion is to reset MAKEOPTS to -j1. And on some occasions, that is indeed the solution to the mysterious build failure. Even the Gentoo manual agrees that the CPUs + 1 rule-of-thumb doesn't always work... http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#installing_portagesays... With MAKEOPTS you define how many parallel compilations should occur when you install a package. A good choice is the number of CPUs (or CPU cores) in your system plus one, ***BUT THIS GUIDELINE ISN'T ALWAYS PERFECT.*** (emphasis mine) I set -j1 and leave it that way. Yes, the builds take longer, but the resulting binary is just as fast. And the amount of time I save will be blown away the first time I end up screwing around a couple of hours trying to fix a mysterious build failure. That's why I want the user to have the option of over-riding EJOBS, should it ever be implemented. -- Walter Dnes waltd...@waltdnes.org I don't run desktop environments; I run useful applications
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote: On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. ~harring
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Sat, Sep 1, 2012 at 8:20 PM, Brian Harring ferri...@gmail.com wrote: On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote: On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. On this point, I use dynamic load-based job counting. I'd be using distcc, too, but I don't have the spare hardware for a second build box right now. For a single box, I target a load average of $PROC_COUNT, with a max jobs in the vicinity of 2x$PROC_COUNT, to prevent I/O hangs from causing job count explosions. (Where I/O hiccups aren't a concern, I've found leaving --jobs open-ended worked fine, but builds always eventually expand to consume available I/O as upstream things get larger) I find this works very well; emerge and make both manage to lurk near the optimum point of keeping the CPU busy without spending too much time in context switches. In a distcc context, I would most likely target a load average of $LOCAL_PROC_COUNT, with a max jobs of (2*$LOCAL_PROC_COUNT+$REMOTE_PROC_COUNT). If ebuilds are try to be more aware of parallel-build contexts, I think it's important they're not limited to static job counts. -- :wq
[gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Thu, 4 Dec 2008, Diego 'Flameeyes' Pettenò wrote:
Since not all the buildsystem we support use make for the actual
build, and they don't necessarily support make-like options (-jX -s
and so on), it would be nice to be able to express a JOBS variable
that could be used for parallel build with any build systems.
Right now there are ebuilds like openoffice or some scons-based
ebuilds that parse MAKEOPTS and get out of that the number of jobs
from the -j option, but this is a) suboptimal b) error-prone.
One has to consider people might be using -l for parallel building
too, for which reasons I'd be suggesting doing something like this to
make the change transparent:
- ebuilds using non-make build systems would use JOBS;
- ebuilds using make builds systems would just use emake as usual;
- Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
- if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
- if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).
The result is that you can finally combine -l with parallel build on
OpenOffice and other packages, with a fallback number of maximum jobs
instead of using load-based decisions.
Coming back to this old topic [1]. Is there still consensus that we
should have such an EJOBS variable? (It shouldn't be called JOBS because
this name is too generic, see the old discussion.) Then we could add it
to EAPI 5.
Ulrich
[1]
http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. People using such a PM and not reading the news will get the old MAKEOPTS which will still work with makefile based build systems but will get serial builds for e.g. EAPI5 ebuilds + waf based build systems. Not a very big deal. A.
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS will have no effect for EAPI5 ebuilds, then obviously we should not be advising users to stop using MAKEOPTS until the whole tree has migrated to EAPI5. And if EJOBS will be recognized by a future version of portage for all EAPIs, then we still should allow MAKEOPTS because some users may want to use --load-average. Changing the name of MAKEOPTS in =EAPI5 makes no sense. First, because it's a standard environment variable used by gnu make. Second, because having 3 different settings for parallel building (EJOBS, MAKEOPTS, and MAKEOPTS_EAPI5) would be insane. Fancy code in the package manager would be the way to go IMHO. Ulrich's message contains a reasonable description of the algorithm. -Alexandre. signature.asc Description: This is a digitally signed message part
