On Thursday, 14 September 2023 10:49:35 BST Peter Humphrey wrote: > On Wednesday, 13 September 2023 16:14:09 BST Michael wrote: > > I recall this being discussed in a previous thread, but if your CPU has 24 > > threads and you've set: > > > > EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32 > > MAKEOPTS="-j14" > > > > You will be asking emerge to run up to 4 x 14 = 56 threads, which could > > potentially eat up to 2G of RAM each, i.e. 112G of RAM. This will exhaust > > your 64G of RAM, not taking into account whatever else the OS will be > > trying to run at the time. > > Yes, I understand that, but I've spent a lot of time watching, and > instrumenting, and in practice the load doesn't exceed about 33. > > > The --load-average=32 is normally expected to be a floating point number > > That stipulation has only appeared recently. I have tried adding a '.0' to > the number, and it's made no noticeaible difference. Besides, I could > attempt pedantry and declare that the set of all integers is a proper > subset of all real numbers. ;-) > > > Of course, not all emerges use make and you may never or rarely emerge 4 x > > monster packages in parallel to need 2G of RAM for each thread at the same > > time. > > I have actually had three or four big packages running together, but my > observation is that they don't pump the load up too far. > > > If only we had at our disposal some AI algorithm to calculate dynamically > > each time we run emerge the optimal combo of parallel emerge jobs and > > number of make tasks, so as to achieve the highest total time saving Vs > > energy spent! Or just the highest total time saving. ;-) > > Yes, that's what we need, all right. > > > I haven't performed any meaningful comparisons to determine where the > > greatest gains are to be achieved. Parallel emerges of many small > > packages, or a large number of make jobs for big packages. The > > combination > > would change each time according to the individual packages waiting for an > > update. In my use case, instinctively feels more beneficial reducing the > > time I have to wait for huge packages like qtwebengine to finish, than > > accelerating the updates of half a dozen smaller packages. > > That is the difficulty. I do often rebuild a new system, not trusting the > existing one any longer, and a lot of time is spent fiddling with four tiny > packages at a time in the early and middle stages, then benefitting from the > limit of 4 once the desktop packages begin.
I very rarely reinstall. So rarely, I can't remember the last time I did it. > > Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset. I set MAKEOPTS > > jobs to the number of CPU threads +1 and the load average at 95%, so I > > can continue using the PC without any noticeable latency. On PCs where > > RAM is constrained I reduce the MAKEOPTS in /etc/portage/ package.env for > > any large packages which are bound to start swapping and thrashing the > > disk. > Interesting. Do you mean 95% of the jobs figure? I'll do some more > experimenting. Yes. Example: A CPU with 4 cores and 8 threads is set like this: MAKEOPTS="-j8 -l7.2" This allows up to 8 make jobs at a time, with an average load being kept at 90% to leave some breathing space for day to day desktop work and minimise too much/frequent swapping: 8 x 0.9 = 7.2 Looking at top shows 7 to 8 compiling jobs at a time. The PC in question has 16G of RAM. I've observed small packages rarely if ever reach 2G of RAM per job. So, I set the above MAKEOPTS in make.conf, but bypass it for large packages in /etc/portage/env/ by setting, e.g.: /etc/portage/env/j4.conf with MAKEOPTS="-j4" in it and specify this for each large package in /etc/ portage/package.env, e.g.: www-client/qtwebengine -j4.conf www-client/firefox -j4.conf dev-lang/rust j4.conf etc. > Meanwhile, perhaps I'll run new builds in two stages: the first without any > desktop packages - I do have sets defined to enable that - and the second > with them. I can't do that with emerge -e though. Unlike MAKEOPTS, the EMERGE_DEFAULT_OPTS variable cannot be set in /etc/ portage/package.env. Therefore, I leave the latter unset in make.conf and specify it on the CLI when I'm about to update a lot of smaller packages and remember to do it. ;-)
signature.asc
Description: This is a digitally signed message part.
