Re: [gentoo-user] per package parallel build

[Khaosgrille]

Hi,

since some people recommended distcc i want to add icecream (or icecc) by SUSE 
https://github.com/icecc/icecream

It is based on distcc but can automaticly schedule things, do some networking 
and is system agnostic. Since it works perfectly fine with portage i have it as 
my default. When i dont have access to my compile machines it just uses my 
laptop and works fine.

Another thing that speeds up compile time significantly is putting your tmpdir 
in the RAM. https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs

Khaosgrille

publickey - Khaosgrille@protonmail.com - 0xE78BC986.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: [gentoo-user] per package parallel build

[Neil Bothwick]

On Tue, 3 Dec 2019 16:26:58 -0500, John Blinka wrote:

> > But if you emerge --update libreoffice before the package that is
> > forcing the rebuild, why would libreoffice rebuild? I would expect it
> > to only rebuild libreoffice after the dependency had been changed.  
> 
> That’s exactly what happened.  I issued an emerge -DuNv —changed-deps
> libreoffice first.  That had the effect of 1) first upgrading several
> libreoffice dependencies, and 2) subsequently rebuilding libreoffice
> once the dependencies changed.  I’m guessing emerge is smart enough to
> trigger both activities and sequence them appropriately.  Operative
> word is guess - I don’t pretend to understand the inner workings.

That's good to know - that it works, i mean, not that you don't
understand how ;-)
> 
> > I'm not saying out wouldn't work some of the time, but I can see
> > situations where it wouldn't. Whereas
> >
> > emerge --opts @world --exclude memory-hogs...
> > emerge --opts --jobs 1 @world
> >
> > should always isolate them.  
> 
> Agreed that this technique should always work.  And it has the
> advantage of fewer invocations of emerge.  But at the cost, I suspect,
> of serializing the building of any memory hog dependencies that were
> excluded from consideration by the first invocation of emerge.

That's a fair point, but it would only apply if the hog had a new
dependency. Even then, it would only be significant if the dependency was
large.

> What situations do you see as not working?

The ones I mentioned before, but it seems that portage may be intelligent
to deal with them. It will be interesting to see if it continues to work
over time.


-- 
Neil Bothwick

Q: Why do PCs - even modern ones - have reset buttons on the front?
A: Because they come with Microsoft operating systems.


pgperF5jy_u5d.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] per package parallel build

[John Blinka]

On Tue, Dec 3, 2019 at 3:46 PM Neil Bothwick  wrote:

> On Tue, 3 Dec 2019 10:43:36 -0500, John Blinka wrote:
>
> > > > Couldn't you just have a script that "emerge --update"s each
> > > > package in sequence? If the package isn't due for update nothing
> > > > will happen. And then you could follow that with an "emerge world"
> > > > knowing that your hogs are already done.
> > >
> > > Sometimes the packages are rebuilt without an update, especially if
> > > you use --changed-use or --changed-deps, so it's not quite that
> > > simple.
> >
> >
> > But still pretty simple.  I’ve just used the “build in sequence” idea
> > for an update that forced a libreoffice rebuild.  It first upgraded a
> > few of libreoffice’s dependencies in parallel, and then rebuilt
> > libreoffice by itself afterwards. A subsequent emerge @world upgraded a
> > bunch of minor kde stuff.  I like this idea - seems to isolate the
> > “hogs” so they build one at a time, and it does so without any
> > intervention on my part.  Thanks!
>
> But if you emerge --update libreoffice before the package that is forcing
> the rebuild, why would libreoffice rebuild? I would expect it to only
> rebuild libreoffice after the dependency had been changed.


That’s exactly what happened.  I issued an emerge -DuNv —changed-deps
libreoffice first.  That had the effect of 1) first upgrading several
libreoffice dependencies, and 2) subsequently rebuilding libreoffice once
the dependencies changed.  I’m guessing emerge is smart enough to trigger
both activities and sequence them appropriately.  Operative word is guess -
I don’t pretend to understand the inner workings.

>
> I'm not saying out wouldn't work some of the time, but I can see
> situations where it wouldn't. Whereas
>
> emerge --opts @world --exclude memory-hogs...
> emerge --opts --jobs 1 @world
>
> should always isolate them.


Agreed that this technique should always work.  And it has the advantage of
fewer invocations of emerge.  But at the cost, I suspect, of serializing
the building of any memory hog dependencies that were excluded from
consideration by the first invocation of emerge.

What situations do you see as not working?

John Blinka

Re: [gentoo-user] per package parallel build

[Neil Bothwick]

On Tue, 3 Dec 2019 10:43:36 -0500, John Blinka wrote:

> > > Couldn't you just have a script that "emerge --update"s each
> > > package in sequence? If the package isn't due for update nothing
> > > will happen. And then you could follow that with an "emerge world"
> > > knowing that your hogs are already done.  
> >
> > Sometimes the packages are rebuilt without an update, especially if
> > you use --changed-use or --changed-deps, so it's not quite that
> > simple.  
> 
> 
> But still pretty simple.  I’ve just used the “build in sequence” idea
> for an update that forced a libreoffice rebuild.  It first upgraded a
> few of libreoffice’s dependencies in parallel, and then rebuilt
> libreoffice by itself afterwards. A subsequent emerge @world upgraded a
> bunch of minor kde stuff.  I like this idea - seems to isolate the
> “hogs” so they build one at a time, and it does so without any
> intervention on my part.  Thanks!

But if you emerge --update libreoffice before the package that is forcing
the rebuild, why would libreoffice rebuild? I would expect it to only
rebuild libreoffice after the dependency had been changed.

I'm not saying out wouldn't work some of the time, but I can see
situations where it wouldn't. Whereas 

emerge --opts @world --exclude memory-hogs...
emerge --opts --jobs 1 @world

should always isolate them.


-- 
Neil Bothwick

Hello, this is an extension to the famous signature virus, called spymail.
Could you please copy me into your signature and send back what you were
doing last night between 10pm and 3am?


pgp5pCpb7diPU.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] per package parallel build

[Wols Lists]

On 03/12/19 15:43, John Blinka wrote:
> 
> 
> On Sat, Nov 30, 2019 at 5:35 PM Neil Bothwick  > wrote:
> 
> On Sat, 30 Nov 2019 16:47:35 +, Wols Lists wrote:
> 
> > > There's no need to mess around adding and removing masks, just use
> > > the -
> > > - exclude option.
> > >
> > > Yep!  For some reason, that option doesn’t always occur to me, but
> > > that’s clearly a simpler way to do it.  Thanks for reminding me!
> > >   
> > Couldn't you just have a script that "emerge --update"s each
> package in
> > sequence? If the package isn't due for update nothing will happen. And
> > then you could follow that with an "emerge world" knowing that
> your hogs
> > are already done.
> 
> Sometimes the packages are rebuilt without an update, especially if you
> use --changed-use or --changed-deps, so it's not quite that simple.
> 
> 
> But still pretty simple.  I’ve just used the “build in sequence” idea
> for an update that forced a libreoffice rebuild.  It first upgraded a
> few of libreoffice’s dependencies in parallel, and then rebuilt
> libreoffice by itself afterwards. A subsequent emerge @world upgraded a
> bunch of minor kde stuff.  I like this idea - seems to isolate the
> “hogs” so they build one at a time, and it does so without any
> intervention on my part.  Thanks!
> 
Eggsackerley

If you do an "emerge memory-hog --normal-options" (whatever your normal
options are), then when you do an "emerge world" with those same
options, it should not see the need to emerge said memory-hog.

Cheers,
Wol

Re: [gentoo-user] per package parallel build

[John Blinka]

On Sat, Nov 30, 2019 at 5:35 PM Neil Bothwick  wrote:

> On Sat, 30 Nov 2019 16:47:35 +, Wols Lists wrote:
>
> > > There's no need to mess around adding and removing masks, just use
> > > the -
> > > - exclude option.
> > >
> > > Yep!  For some reason, that option doesn’t always occur to me, but
> > > that’s clearly a simpler way to do it.  Thanks for reminding me!
> > >
> > Couldn't you just have a script that "emerge --update"s each package in
> > sequence? If the package isn't due for update nothing will happen. And
> > then you could follow that with an "emerge world" knowing that your hogs
> > are already done.
>
> Sometimes the packages are rebuilt without an update, especially if you
> use --changed-use or --changed-deps, so it's not quite that simple.


But still pretty simple.  I’ve just used the “build in sequence” idea for
an update that forced a libreoffice rebuild.  It first upgraded a few of
libreoffice’s dependencies in parallel, and then rebuilt libreoffice by
itself afterwards. A subsequent emerge @world upgraded a bunch of minor kde
stuff.  I like this idea - seems to isolate the “hogs” so they build one at
a time, and it does so without any intervention on my part.  Thanks!

John Blinka

Re: [gentoo-user] per package parallel build

[Neil Bothwick]

On Sat, 30 Nov 2019 16:47:35 +, Wols Lists wrote:

> > There's no need to mess around adding and removing masks, just use
> > the -
> > - exclude option. 
> > 
> > Yep!  For some reason, that option doesn’t always occur to me, but
> > that’s clearly a simpler way to do it.  Thanks for reminding me!
> >   
> Couldn't you just have a script that "emerge --update"s each package in
> sequence? If the package isn't due for update nothing will happen. And
> then you could follow that with an "emerge world" knowing that your hogs
> are already done.

Sometimes the packages are rebuilt without an update, especially if you
use --changed-use or --changed-deps, so it's not quite that simple.


-- 
Neil Bothwick

I am Flatulus of Borg.  You will be asphixiated.


pgpVAqzy6WIHV.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] per package parallel build

[Wols Lists]

On 26/10/19 14:32, John Blinka wrote:
> There's no need to mess around adding and removing masks, just use the -
> - exclude option. 
> 
> Yep!  For some reason, that option doesn’t always occur to me, but
> that’s clearly a simpler way to do it.  Thanks for reminding me!
> 
Couldn't you just have a script that "emerge --update"s each package in
sequence? If the package isn't due for update nothing will happen. And
then you could follow that with an "emerge world" knowing that your hogs
are already done.

Cheers,
Wol

Re: [gentoo-user] per package parallel build

[Dale]

Mick wrote:
> On Friday, 25 October 2019 18:01:57 BST Dale wrote:
>> Mick wrote:
>>> PS.  In an ideal AI world, portage would know how much memory is necessary
>>> for a given package and would auto-adjust the number of jobs to minimise
>>> swapping given any amount of RAM.  In an even more ideal world, it would
>>> be able to do this in real time.  :-)
>> I agree that it would be nice if emerge could do that automatically,
>> although I have no clue how to do that or even if it can be done at
>> all.  Back when I had less memory, I could let FF, LOo or another
>> package run at full speed but only if it was only one of those packages
>> at a time.  Thing is, on occasion two or more of those updates would hit
>> and due to the long compile times, end up compiling at the same time. 
>> Do you think there is a way for the devs to set up a method to tell
>> emerge not to emerge certain packages at the same time?  In other words,
>> if Firefox is emerging, LOo is held until it is done or vice versa. 
>> Maybe even have it so others can be listed.  The list of large packages
>> are likely small but they can have a huge impact on systems with less
>> memory. 
>>
>> You think that a feature worth asking the devs about?  Maybe they can
>> figure out a way to implement that??
>>
>> Dale
>>
>> :-)  :-) 
> Well, ebuilds with known large compiles have specific tests at present, e.g. 
> chromium, FF, et al., run checks on available disk space for /var/tmp/portage 
> and warn the user accordingly.  In firefox-70.0.ebuild we see this:
>
> ...
> pkg_pretend() {
> # Ensure we have enough disk space to compile
> if use pgo || use lto || use debug || use test ; then
> CHECKREQS_DISK_BUILD="8G"
> else
> CHECKREQS_DISK_BUILD="4G"
> fi
>
> check-reqs_pkg_setup
> }
>
> I guess a lot can be included in such checks, but I have no idea how to do 
> it, 
> or how complicated an ebuild would get as a result.  The problem with RAM 
> usage is that it changes continuously.  You open a browser, or some large 
> PDF, 
> or LO and more RAM is being used, just at the moment an ebuild decides to run 
> a check on available RAM.  It can't guess if you will shut down your 
> applications and log out of your desktop the next moment to free up more 
> memory, if you will shut down web servers, databases or whatever else you may 
> be running in the background and drop to a console.  For this reason I assume 
> such a measure won't be implemented in rush and even if it did, it would 
> annoy 
> people, especially if they are asked to add USE="I-really-know-what-I-am-
> doing" to allow the compile to start.


I was hoping for something more simple.  Nothing in the ebuild just a
list of packages that emerge won't run at the same time.  Let's say the
big ones are included in the list, Seamonkey, Firefox, LOo and maybe a
couple others.  When emerge is run, Firefox and LOo is to be updated. 
Emerge itself knows not to update both so if one is updating and the
other is next in line, then it waits until the other is finished before
starting the next big one.  I would think the code should be in emerge
itself not the ebuild.  That way the user can determine what their
particular machine can handle. 

Like you, I'm not sure how to code that.  Of course, I've read about the
devs doing some complicated things that I don't grasp.  After all, it
calculates dependencies a lot faster than I could do on paper.  lol 

Dale

:-)  :-) 

Re: [gentoo-user] per package parallel build

[John Blinka]

There's no need to mess around adding and removing masks, just use the - -
exclude option.

Yep!  For some reason, that option doesn’t always occur to me, but that’s
clearly a simpler way to do it.  Thanks for reminding me!

John

Re: [gentoo-user] per package parallel build

[Neil Bothwick]

On 26 October 2019 12:16:37 BST, John Blinka  wrote:
>>
>> I agree that it would be nice if emerge could do that automatically,
>> although I have no clue how to do that or even if it can be done at
>> all.  Back when I had less memory, I could let FF, LOo or another
>> package run at full speed but only if it was only one of those
>packages
>> at a time.  Thing is, on occasion two or more of those updates would
>hit
>> and due to the long compile times, end up compiling at the same time.
>> Do you think there is a way for the devs to set up a method to tell
>> emerge not to emerge certain packages at the same time?  In other
>words,
>> if Firefox is emerging, LOo is held until it is done or vice versa.
>> Maybe even have it so others can be listed.  The list of large
>packages
>> are likely small but they can have a huge impact on systems with less
>> memory.
>>
>> You think that a feature worth asking the devs about?  Maybe they can
>> figure out a way to implement that??
>
>
>There already is a mechanism you can use, but it’s not the automatic
>type
>that you (and, admittedly I) would like.
>
>I have 3 old 2 core machines, and I use distcc heavily to reduce emerge
>times.  The “fastest” (not really) and best equipped has 16 gb memory. 
>I
>do updates on this machine (with distcc help from the others) and
>distribute packages to the rest.  After a lot of experimenting, I find
>that
>MAKEOPTS=“-j13 -l5” works the best on this fastest machine.  That
>setting
>allows it to attempt a workload that it alone doesn’t have the
>resources to
>accomplish, but successfully distributes to the other machines.  I use
>firefox, chromium, and libreoffice.  Occasionally portage wants to
>upgrade
>more than one of these at a time, which I discover by running emerge
>—pretend.  On those occasions,  I’ve learned that I run out of
>resources
>and builds fail.  So I just temporarily mask all but one of those
>updates,
>do the upgrade, unmask one of the masked updates, do another upgrade,
>and
>so on.  Works well for me.  No builds crash, essentially no swap gets
>used,
>and I have substantially accelerated compile and ebuild times.
>
>The tools exist to do what you want to do.  If you were so inclined,
>you
>might even contemplate writing a script to automate what I just
>described.
>
>John Blinka

There's no need to mess around adding and removing masks, just use the - - 
exclude option. 
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] per package parallel build

[John Blinka]

>
> I agree that it would be nice if emerge could do that automatically,
> although I have no clue how to do that or even if it can be done at
> all.  Back when I had less memory, I could let FF, LOo or another
> package run at full speed but only if it was only one of those packages
> at a time.  Thing is, on occasion two or more of those updates would hit
> and due to the long compile times, end up compiling at the same time.
> Do you think there is a way for the devs to set up a method to tell
> emerge not to emerge certain packages at the same time?  In other words,
> if Firefox is emerging, LOo is held until it is done or vice versa.
> Maybe even have it so others can be listed.  The list of large packages
> are likely small but they can have a huge impact on systems with less
> memory.
>
> You think that a feature worth asking the devs about?  Maybe they can
> figure out a way to implement that??


There already is a mechanism you can use, but it’s not the automatic type
that you (and, admittedly I) would like.

I have 3 old 2 core machines, and I use distcc heavily to reduce emerge
times.  The “fastest” (not really) and best equipped has 16 gb memory.  I
do updates on this machine (with distcc help from the others) and
distribute packages to the rest.  After a lot of experimenting, I find that
MAKEOPTS=“-j13 -l5” works the best on this fastest machine.  That setting
allows it to attempt a workload that it alone doesn’t have the resources to
accomplish, but successfully distributes to the other machines.  I use
firefox, chromium, and libreoffice.  Occasionally portage wants to upgrade
more than one of these at a time, which I discover by running emerge
—pretend.  On those occasions,  I’ve learned that I run out of resources
and builds fail.  So I just temporarily mask all but one of those updates,
do the upgrade, unmask one of the masked updates, do another upgrade, and
so on.  Works well for me.  No builds crash, essentially no swap gets used,
and I have substantially accelerated compile and ebuild times.

The tools exist to do what you want to do.  If you were so inclined, you
might even contemplate writing a script to automate what I just described.

John Blinka

Re: [gentoo-user] per package parallel build

[Mick]

On Friday, 25 October 2019 18:01:57 BST Dale wrote:
> Mick wrote:

> > PS.  In an ideal AI world, portage would know how much memory is necessary
> > for a given package and would auto-adjust the number of jobs to minimise
> > swapping given any amount of RAM.  In an even more ideal world, it would
> > be able to do this in real time.  :-)
> 
> I agree that it would be nice if emerge could do that automatically,
> although I have no clue how to do that or even if it can be done at
> all.  Back when I had less memory, I could let FF, LOo or another
> package run at full speed but only if it was only one of those packages
> at a time.  Thing is, on occasion two or more of those updates would hit
> and due to the long compile times, end up compiling at the same time. 
> Do you think there is a way for the devs to set up a method to tell
> emerge not to emerge certain packages at the same time?  In other words,
> if Firefox is emerging, LOo is held until it is done or vice versa. 
> Maybe even have it so others can be listed.  The list of large packages
> are likely small but they can have a huge impact on systems with less
> memory. 
> 
> You think that a feature worth asking the devs about?  Maybe they can
> figure out a way to implement that??
> 
> Dale
> 
> :-)  :-) 

Well, ebuilds with known large compiles have specific tests at present, e.g. 
chromium, FF, et al., run checks on available disk space for /var/tmp/portage 
and warn the user accordingly.  In firefox-70.0.ebuild we see this:

...
pkg_pretend() {
# Ensure we have enough disk space to compile
if use pgo || use lto || use debug || use test ; then
CHECKREQS_DISK_BUILD="8G"
else
CHECKREQS_DISK_BUILD="4G"
fi

check-reqs_pkg_setup
}

I guess a lot can be included in such checks, but I have no idea how to do it, 
or how complicated an ebuild would get as a result.  The problem with RAM 
usage is that it changes continuously.  You open a browser, or some large PDF, 
or LO and more RAM is being used, just at the moment an ebuild decides to run 
a check on available RAM.  It can't guess if you will shut down your 
applications and log out of your desktop the next moment to free up more 
memory, if you will shut down web servers, databases or whatever else you may 
be running in the background and drop to a console.  For this reason I assume 
such a measure won't be implemented in rush and even if it did, it would annoy 
people, especially if they are asked to add USE="I-really-know-what-I-am-
doing" to allow the compile to start.
-- 
Regards,

Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] per package parallel build

[Dale]

Mick wrote:
> On Friday, 25 October 2019 06:31:03 BST Raffaele BELARDI wrote:
>> 8-core CPU:
>>
>> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
>>
>> MAKEOPTS="-j9 -l9"
>>
>> Works fine except when both Firefox and Thunderbird need update, in that
>> case emerge typically tries to build them in parallel and one gets OOM
>> killed due to insufficient swap space (1G swap, 16G RAM). I will increase
>> the swap but I'd like to know: Is there a way to tell emerge to normally
>> run 9 parallel jobs but limit to 1 when it is building one of the two
>> monsters?
>>
>> Thanks,
>>
>> raffaele
> Good advice has already been given, but here are some supporting thoughts 
> based on my experience of trying to emerge mammoth builds on old systems with 
> low RAM.
>
> I can't recall how much memory each thread of a FF or TB compile chews up, 
> but 
> if you allow for up to 2G per thread, with -j9 your 16G RAM *will* be 
> exhausted at some point.  Swapping will commence then in earnest and a I/O 
> offloading of RAM into swap and back again as threads gasp for adequate 
> memory 
> will be sustained, while demand for more RAM than available is present. This 
> creates a bottleneck which will prolong the emerge duration.  With enough 
> swap 
> you won't suffer any more OOMs, but you could find emerge times increase 
> exponentially.  Therefore reducing the total number of jobs for these two 
> packages will help in both cases.
>
> Setting up an env directive as already advised in previous responses to 
> restrict both packages to fewer jobs will work, but in some cases it can be 
> suboptimal.  When only one package needs updating, why should it be emerged 
> at 
> a lesser speed than your system can support?  Even when both packages are 
> emerged at the same time, the memory crunch may only take place for a limited 
> period, at a time when both packages will be using maximum memory per thread. 
>  
> To make matters worse ebuilds and source code changes, compilers change and 
> trying to optimise your emerge soon becomes a moving beast, so some 
> experimentation will be necessary.
>
> I think reducing your job/load count to 1 would be excessive.  Reducing it 
> down to 5 ought to be adequate, but I suggest to increase your swap.  You 
> could just use a swap file for this purpose.
>
> PS.  In an ideal AI world, portage would know how much memory is necessary 
> for 
> a given package and would auto-adjust the number of jobs to minimise swapping 
> given any amount of RAM.  In an even more ideal world, it would be able to do 
> this in real time.  :-)
>


I agree that it would be nice if emerge could do that automatically,
although I have no clue how to do that or even if it can be done at
all.  Back when I had less memory, I could let FF, LOo or another
package run at full speed but only if it was only one of those packages
at a time.  Thing is, on occasion two or more of those updates would hit
and due to the long compile times, end up compiling at the same time. 
Do you think there is a way for the devs to set up a method to tell
emerge not to emerge certain packages at the same time?  In other words,
if Firefox is emerging, LOo is held until it is done or vice versa. 
Maybe even have it so others can be listed.  The list of large packages
are likely small but they can have a huge impact on systems with less
memory. 

You think that a feature worth asking the devs about?  Maybe they can
figure out a way to implement that??

Dale

:-)  :-) 

Re: [gentoo-user] per package parallel build

[Mick]

On Friday, 25 October 2019 06:31:03 BST Raffaele BELARDI wrote:
> 8-core CPU:
> 
> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
> 
> MAKEOPTS="-j9 -l9"
> 
> Works fine except when both Firefox and Thunderbird need update, in that
> case emerge typically tries to build them in parallel and one gets OOM
> killed due to insufficient swap space (1G swap, 16G RAM). I will increase
> the swap but I'd like to know: Is there a way to tell emerge to normally
> run 9 parallel jobs but limit to 1 when it is building one of the two
> monsters?
> 
> Thanks,
> 
> raffaele

Good advice has already been given, but here are some supporting thoughts 
based on my experience of trying to emerge mammoth builds on old systems with 
low RAM.

I can't recall how much memory each thread of a FF or TB compile chews up, but 
if you allow for up to 2G per thread, with -j9 your 16G RAM *will* be 
exhausted at some point.  Swapping will commence then in earnest and a I/O 
offloading of RAM into swap and back again as threads gasp for adequate memory 
will be sustained, while demand for more RAM than available is present. This 
creates a bottleneck which will prolong the emerge duration.  With enough swap 
you won't suffer any more OOMs, but you could find emerge times increase 
exponentially.  Therefore reducing the total number of jobs for these two 
packages will help in both cases.

Setting up an env directive as already advised in previous responses to 
restrict both packages to fewer jobs will work, but in some cases it can be 
suboptimal.  When only one package needs updating, why should it be emerged at 
a lesser speed than your system can support?  Even when both packages are 
emerged at the same time, the memory crunch may only take place for a limited 
period, at a time when both packages will be using maximum memory per thread.  
To make matters worse ebuilds and source code changes, compilers change and 
trying to optimise your emerge soon becomes a moving beast, so some 
experimentation will be necessary.

I think reducing your job/load count to 1 would be excessive.  Reducing it 
down to 5 ought to be adequate, but I suggest to increase your swap.  You 
could just use a swap file for this purpose.

PS.  In an ideal AI world, portage would know how much memory is necessary for 
a given package and would auto-adjust the number of jobs to minimise swapping 
given any amount of RAM.  In an even more ideal world, it would be able to do 
this in real time.  :-)

-- 
Regards,

Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] per package parallel build

[Peter Humphrey]

On Friday, 25 October 2019 06:31:03 BST Raffaele BELARDI wrote:
> 8-core CPU:
> 
> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
> 
> MAKEOPTS="-j9 -l9"
> 
> Works fine except when both Firefox and Thunderbird need update, in that
> case emerge typically tries to build them in parallel and one gets OOM
> killed due to insufficient swap space (1G swap, 16G RAM). I will increase
> the swap but I'd like to know: Is there a way to tell emerge to normally
> run 9 parallel jobs but limit to 1 when it is building one of the two
> monsters?

Apart from increasing the swap space, I think the only way to avoid this 
condition is to spot it in advance and not allow portage to work on them both 
at the same time: either --exclude one of them or specify it.

-- 
Regards,
Peter.

Re: [gentoo-user] per package parallel build

[Dale]

Raffaele BELARDI wrote:
>
> 8-core CPU:
>
> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
>
> MAKEOPTS="-j9 -l9"
>
>  
>
> Works fine except when both Firefox and Thunderbird need update, in
> that case emerge typically tries to build them in parallel and one
> gets OOM killed due to insufficient swap space (1G swap, 16G RAM). I
> will increase the swap but I’d like to know:
>
> Is there a way to tell emerge to normally run 9 parallel jobs but
> limit to 1 when it is building one of the two monsters?
>
>  
>
> Thanks,
>
>  
>
> raffaele
>


This may help, along with Adams info.

https://wiki.gentoo.org/wiki//etc/portage/package.env

This is a bit of mine that may also help.  I hope you can follow what
that means.  The one for single may interest you the most.



root@fireball / # cat /etc/portage/package.env/package.env
#www-client/seamonkey  ../env/single.conf
#www-client/firefox  ../env/single.conf
#www-client/firefox  ../env/notmpfs.conf
#www-client/seamonkey ../env/notmpfs.conf
app-office/libreoffice ../env/notmpfs.conf
#sys-devel/gcc ../env/notmpfs.conf
dev-qt/qtwebengine ../env/notmpfs.conf
#dev-qt/qtwebkit ../env/notmpfs.conf
#sci-electronics/kicad ../env/notmpfs.conf


root@fireball / # cat /etc/portage/env/notmpfs.conf
PORTAGE_TMPDIR="/var/tmp/notmpfs"
root@fireball / # cat /etc/portage/env/single.conf
#EMERGE_DEFAULT_OPTS="-j1"
 
 
root@fireball / #


Note, some of mine are commented out so don't duplicate that.  I
upgraded from 16GBs to 32GBs and no longer need some exceptions.

Dale

:-)  :-) 

Re: [gentoo-user] per package parallel build

[Adam Carter]

On Fri, Oct 25, 2019 at 4:31 PM Raffaele BELARDI 
wrote:

> 8-core CPU:
>
> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
>
> MAKEOPTS="-j9 -l9"
>
>
>
> Works fine except when both Firefox and Thunderbird need update, in that
> case emerge typically tries to build them in parallel and one gets OOM
> killed due to insufficient swap space (1G swap, 16G RAM). I will increase
> the swap but I’d like to know:
>
> Is there a way to tell emerge to normally run 9 parallel jobs but limit to
> 1 when it is building one of the two monsters?
>

IIRC;

/etc/portage/env/smallbuild.conf

EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 1 --load-average 1"

MAKEOPTS="-j1 -l1"

/etc/portage/package.env
mail-client/thunderbird smallbuild.conf
www-client/firefox smallbuild.conf

[gentoo-user] per package parallel build

[Raffaele BELARDI]

8-core CPU:

EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"

MAKEOPTS="-j9 -l9"

Works fine except when both Firefox and Thunderbird need update, in that case 
emerge typically tries to build them in parallel and one gets OOM killed due to 
insufficient swap space (1G swap, 16G RAM). I will increase the swap but I'd 
like to know:
Is there a way to tell emerge to normally run 9 parallel jobs but limit to 1 
when it is building one of the two monsters?

Thanks,

raffaele