On Monday, 18 September 2023 17:13:04 BST Alan McKinnon wrote: > On Mon, Sep 18, 2023 at 6:03 PM Peter Humphrey <pe...@prh.myzen.co.uk> > > wrote: > > On Monday, 18 September 2023 14:48:46 BST Alan McKinnon wrote: > > > On Mon, Sep 18, 2023 at 3:44 PM Peter Humphrey <pe...@prh.myzen.co.uk> > > > > > > wrote: > > > > It may be less complex than you think, Jack. I envisage a package > > > > being > > > > marked > > > > as solitary, and when portage reaches that package, it waits until all > > > > current > > > > jobs have finished, then it starts the solitary package with the > > > > environment > > > > specified for it, and it doesn't start the next one until that one has > > > > finished. > > > > The dependency calculation shouldn't need to be changed. > > > > > > > > It seems simple the way I see it. > > > > > > How does that improve emerge performance overall? > > > > By allocating all the system resources to huge packages while not flooding > > the > > system with lesser ones. For example, I can set -j20 for webkit-gtk today > > without overflowing the 64GB RAM, and still have 4 CPU threads available > > to > > other tasks. The change I've proposed should make the whole operation more > > efficient overall and take less time. > > > > As things stand today, I have to make do with -j12 or so, wasting time and > > resources. I have load-average set at 32, so if I were to set -j20 > > generally > > I'd run out of RAM in no time. I've had many instances of packages failing > > to > > compile in a large update, but going just fine on their own; and I've had > > mysterious operational errors resulting, I suspect, from otherwise > > undetected > > miscompilation. > > > > Previous threads have more detail of what I've tried already. > > > > > > I did read all those but no matter how you move things around you still > > have only X resources available all the time. > Whether you just let emerge do it's thing or try get it to do big packages > on their own, everything is still going to use the same number of cpu > cycles overall and you will save nothing. > > If webkit-gtk is the only big package, have you considered: > > emerge -1v webkit-gtk && emerge -avuND @world? > > > What you have is not a portage problem. It is a orthodox parallelism > problem, and I think you are thinking your constraint is unique in the work > - it isn't. > With parallelism, trying to fiddle single nodes to improve things overall > never really works out. > > Just my $0.02 > > > Alan
I think there is a level of complexity involved which will make (m)any attempts on optimisation difficult, because EMERGE_DEFAULT_OPTS competes for resources against MAKEOPTS, resulting in a trade-off between their optimal settings. Parallelisation becomes difficult to maximise on the basis of some presets when not all updates have the same combination of small Vs large packages, dependent packages queue up before dependencies are built, various emerge stages are processed linearly, some versions of gcc may get hungrier for RAM and whatever else I haven't accounted for. Someone with a PhD on multivariate stochastic analysis could probably come up with some nifty code to include in portage? ;-)
signature.asc
Description: This is a digitally signed message part.
