> From: Alan McKinnon alan.mckin...@gmail.com<mailto:alan.mckin...@gmail.com> > Sent: Monday, September 18, 2023 9:13 AM > To: gentoo-user@lists.gentoo.org<mailto:gentoo-user@lists.gentoo.org> > Subject: Re: [gentoo-user] Controlling emerges > > > > On Mon, Sep 18, 2023 at 6:03 PM Peter Humphrey > pe...@prh.myzen.co.uk<mailto:pe...@prh.myzen.co.uk> wrote: > On Monday, 18 September 2023 14:48:46 BST Alan McKinnon wrote: > > On Mon, Sep 18, 2023 at 3:44 PM Peter Humphrey > > pe...@prh.myzen.co.uk<mailto:pe...@prh.myzen.co.uk> > > > > wrote: > > > It may be less complex than you think, Jack. I envisage a package being > > > marked > > > as solitary, and when portage reaches that package, it waits until all > > > current > > > jobs have finished, then it starts the solitary package with the > > > environment > > > specified for it, and it doesn't start the next one until that one has > > > finished. > > > The dependency calculation shouldn't need to be changed. > > > > > > It seems simple the way I see it. > > > > How does that improve emerge performance overall? > > By allocating all the system resources to huge packages while not flooding the > system with lesser ones. For example, I can set -j20 for webkit-gtk today > without overflowing the 64GB RAM, and still have 4 CPU threads available to > other tasks. The change I've proposed should make the whole operation more > efficient overall and take less time. > > As things stand today, I have to make do with -j12 or so, wasting time and > resources. I have load-average set at 32, so if I were to set -j20 generally > I'd run out of RAM in no time. I've had many instances of packages failing to > compile in a large update, but going just fine on their own; and I've had > mysterious operational errors resulting, I suspect, from otherwise undetected > miscompilation. > > Previous threads have more detail of what I've tried already. > > I did read all those but no matter how you move things around you still have > only X resources available all the time. > Whether you just let emerge do it's thing or try get it to do big packages on > their own, everything is still going to use the same number of cpu cycles > overall and you will save nothing. > > If webkit-gtk is the only big package, have you considered: > > emerge -1v webkit-gtk && emerge -avuND @world? > > > What you have is not a portage problem. It is a orthodox parallelism problem, > and I think you are thinking your constraint is unique in the work - it isn't. > With parallelism, trying to fiddle single nodes to improve things overall > never really works out. > > Just my $0.02 > > > Alan > > -- > Alan McKinnon > alan dot mckinnon at gmail dot com >
Note that on my systems I just make heavy use of the various load-average limiting options and as long as two of the big packages don't start within seconds of each other it does a pretty good job of letting them run by themselves. If things do get in a snarl, you can always use kill -18/19 to suspend a few compile jobs until the system stops thrashing and resume them as capacity permits. LMP
