Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Walter Dnes]

On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote

> This approach is fine imo, although I'd *potentially* look at adding a 
> magic $PROC_COUNT var that is the # of cpu threads on the system; 
> either that or defaulting jobs to it.
> 
> I rather dislike requiring users to go jam a 2/4/8 in there when it's 
> easy to compute.  That said, it's minor.
> 
> Either way, yes, I think EJOBS should be in EAPI5.

  One question about the suggested EJOBS variable; will it over-ride
MAKEOPTS?  Every so often on the Gentoo-user list, someone comes along
with a mysterious build failure.  The first suggestion is to reset
MAKEOPTS to -j1.  And on some occasions, that is indeed the solution to
the mysterious build failure.  Even the Gentoo manual agrees that the
"CPUs + 1" rule-of-thumb doesn't always work...
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#installing_portagesays...

> With MAKEOPTS you define how many parallel compilations should occur
> when you install a package.  A good choice is the number of CPUs
> (or CPU cores) in your system plus one, ***BUT THIS GUIDELINE ISN'T
> ALWAYS PERFECT.*** (emphasis mine)

  I set -j1 and leave it that way.  Yes, the builds take longer, but the
resulting binary is just as fast.  And the amount of time I "save" will
be blown away the first time I end up screwing around a couple of hours
trying to fix a mysterious build failure.  That's why I want the user to
have the option of over-riding EJOBS, should it ever be implemented.

-- 
Walter Dnes 
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Michael Mol]

On Sat, Sep 1, 2012 at 8:20 PM, Brian Harring  wrote:
> On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote:
>> On Fri, 31 Aug 2012 15:45:21 +0100
>> Ciaran McCreesh  wrote:
>>
>> > On Fri, 31 Aug 2012 10:21:15 +0200
>> > Ulrich Mueller  wrote:
>> > > Coming back to this old topic [1]. Is there still consensus that we
>> > > should have such an EJOBS variable? (It shouldn't be called JOBS
>> > > because this name is too generic, see the old discussion.) Then we
>> > > could add it to EAPI 5.
>> > >
>> > > Ulrich
>> > >
>> > > [1]
>> > > 
>> >
>> > If we're doing this, do we tell users to stop setting MAKEOPTS for
>> > EAPIs 5 and greater?
>>
>> How can this work ? I cant think of any simple solution.
>>
>> > Do we change the name of MAKEOPTS for EAPIs 5 and
>> > greater instead? Do we put fancy code in the package mangler to deal
>> > with it?
>>
>> IMHO EAPI-5 compliant PMs should do MAKEOPTS="$MAKEOPTS -j$EJOBS" for
>> every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5
>> and greater.
>> This is retroactive but could be classified 'PM internals' so its fine
>> imho.
>
> This approach is fine imo, although I'd *potentially* look at adding a
> magic $PROC_COUNT var that is the # of cpu threads on the system;
> either that or defaulting jobs to it.
>
> I rather dislike requiring users to go jam a 2/4/8 in there when it's
> easy to compute.  That said, it's minor.

On this point, I use dynamic load-based job counting. I'd be using
distcc, too, but I don't have the spare hardware for a second build
box right now.

For a single box, I target a load average of $PROC_COUNT, with a max
jobs in the vicinity of 2x$PROC_COUNT, to prevent I/O hangs from
causing job count explosions. (Where I/O hiccups aren't a concern,
I've found leaving --jobs open-ended worked fine, but builds always
eventually expand to consume available I/O as upstream things get
larger) I find this works very well; emerge and make both manage to
lurk near the optimum point of keeping the CPU busy without spending
too much time in context switches.

In a distcc context, I would most likely target a load average of
$LOCAL_PROC_COUNT, with a max jobs of
(2*$LOCAL_PROC_COUNT+$REMOTE_PROC_COUNT).

If ebuilds are try to be more aware of parallel-build contexts, I
think it's important they're not limited to static job counts.

-- 
:wq

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Brian Harring]

On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote:
> On Fri, 31 Aug 2012 15:45:21 +0100
> Ciaran McCreesh  wrote:
> 
> > On Fri, 31 Aug 2012 10:21:15 +0200
> > Ulrich Mueller  wrote:
> > > Coming back to this old topic [1]. Is there still consensus that we
> > > should have such an EJOBS variable? (It shouldn't be called JOBS
> > > because this name is too generic, see the old discussion.) Then we
> > > could add it to EAPI 5.
> > > 
> > > Ulrich
> > > 
> > > [1]
> > > 
> > 
> > If we're doing this, do we tell users to stop setting MAKEOPTS for
> > EAPIs 5 and greater?
> 
> How can this work ? I cant think of any simple solution.
> 
> > Do we change the name of MAKEOPTS for EAPIs 5 and
> > greater instead? Do we put fancy code in the package mangler to deal
> > with it?
> 
> IMHO EAPI-5 compliant PMs should do MAKEOPTS="$MAKEOPTS -j$EJOBS" for
> every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5
> and greater.
> This is retroactive but could be classified 'PM internals' so its fine
> imho.

This approach is fine imo, although I'd *potentially* look at adding a 
magic $PROC_COUNT var that is the # of cpu threads on the system; 
either that or defaulting jobs to it.

I rather dislike requiring users to go jam a 2/4/8 in there when it's 
easy to compute.  That said, it's minor.

Either way, yes, I think EJOBS should be in EAPI5.
~harring

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Alexandre Rostovtsev]

On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 10:21:15 +0200
> Ulrich Mueller  wrote:
> > Coming back to this old topic [1]. Is there still consensus that we
> > should have such an EJOBS variable? (It shouldn't be called JOBS
> > because this name is too generic, see the old discussion.) Then we
> > could add it to EAPI 5.
> > 
> > Ulrich
> > 
> > [1]
> > 
> 
> If we're doing this, do we tell users to stop setting MAKEOPTS for
> EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and
> greater instead? Do we put fancy code in the package mangler to deal
> with it?

Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS will
have no effect for =EAPI5 makes no sense. First, because
it's a standard environment variable used by gnu make. Second, because
having 3 different settings for parallel building (EJOBS, MAKEOPTS, and
"MAKEOPTS_EAPI5") would be insane.

Fancy code in the package manager would be the way to go IMHO. Ulrich's
message contains a reasonable description of the algorithm.

-Alexandre.


signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Alexis Ballier]

On Fri, 31 Aug 2012 15:45:21 +0100
Ciaran McCreesh  wrote:

> On Fri, 31 Aug 2012 10:21:15 +0200
> Ulrich Mueller  wrote:
> > Coming back to this old topic [1]. Is there still consensus that we
> > should have such an EJOBS variable? (It shouldn't be called JOBS
> > because this name is too generic, see the old discussion.) Then we
> > could add it to EAPI 5.
> > 
> > Ulrich
> > 
> > [1]
> > 
> 
> If we're doing this, do we tell users to stop setting MAKEOPTS for
> EAPIs 5 and greater?

How can this work ? I cant think of any simple solution.

> Do we change the name of MAKEOPTS for EAPIs 5 and
> greater instead? Do we put fancy code in the package mangler to deal
> with it?

IMHO EAPI-5 compliant PMs should do MAKEOPTS="$MAKEOPTS -j$EJOBS" for
every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5
and greater.
This is retroactive but could be classified 'PM internals' so its fine
imho.

People using such a PM and not reading the news will get the old
MAKEOPTS which will still work with makefile based build systems but
will get serial builds for e.g. EAPI5 ebuilds + waf based build systems.
Not a very big deal.

A.

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Ciaran McCreesh]

On Fri, 31 Aug 2012 10:21:15 +0200
Ulrich Mueller  wrote:
> Coming back to this old topic [1]. Is there still consensus that we
> should have such an EJOBS variable? (It shouldn't be called JOBS
> because this name is too generic, see the old discussion.) Then we
> could add it to EAPI 5.
> 
> Ulrich
> 
> [1]
> 

If we're doing this, do we tell users to stop setting MAKEOPTS for
EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and
greater instead? Do we put fancy code in the package mangler to deal
with it?

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

[gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Ulrich Mueller]

> On Thu, 4 Dec 2008, Diego 'Flameeyes' Pettenò wrote:

> Since not all the buildsystem we support use make for the actual
> build, and they don't necessarily support make-like options (-jX -s
> and so on), it would be nice to be able to express a JOBS variable
> that could be used for parallel build with any build systems.

> Right now there are ebuilds like openoffice or some scons-based
> ebuilds that parse MAKEOPTS and get out of that the number of jobs
> from the -j option, but this is a) suboptimal b) error-prone.

> One has to consider people might be using -l for parallel building
> too, for which reasons I'd be suggesting doing something like this to
> make the change transparent:

>  - ebuilds using non-make build systems would use JOBS;
>  - ebuilds using make builds systems would just use emake as usual;
>  - Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
>  - if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
>  - if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).

> The result is that you can finally combine -l with parallel build on
> OpenOffice and other packages, with a fallback number of maximum jobs
> instead of using load-based decisions.

Coming back to this old topic [1]. Is there still consensus that we
should have such an EJOBS variable? (It shouldn't be called JOBS because
this name is too generic, see the old discussion.) Then we could add it
to EAPI 5.

Ulrich

[1]