On Wednesday, 13 September 2023 16:14:09 BST Michael wrote: > I recall this being discussed in a previous thread, but if your CPU has 24 > threads and you've set: > > EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32 > MAKEOPTS="-j14" > > You will be asking emerge to run up to 4 x 14 = 56 threads, which could > potentially eat up to 2G of RAM each, i.e. 112G of RAM. This will exhaust > your 64G of RAM, not taking into account whatever else the OS will be trying > to run at the time.
Yes, I understand that, but I've spent a lot of time watching, and instrumenting, and in practice the load doesn't exceed about 33. > The --load-average=32 is normally expected to be a floating point number That stipulation has only appeared recently. I have tried adding a '.0' to the number, and it's made no noticeaible difference. Besides, I could attempt pedantry and declare that the set of all integers is a proper subset of all real numbers. ;-) > Of course, not all emerges use make and you may never or rarely emerge 4 x > monster packages in parallel to need 2G of RAM for each thread at the same > time. I have actually had three or four big packages running together, but my observation is that they don't pump the load up too far. > If only we had at our disposal some AI algorithm to calculate dynamically > each time we run emerge the optimal combo of parallel emerge jobs and > number of make tasks, so as to achieve the highest total time saving Vs > energy spent! Or just the highest total time saving. ;-) Yes, that's what we need, all right. > I haven't performed any meaningful comparisons to determine where the > greatest gains are to be achieved. Parallel emerges of many small > packages, or a large number of make jobs for big packages. The combination > would change each time according to the individual packages waiting for an > update. In my use case, instinctively feels more beneficial reducing the > time I have to wait for huge packages like qtwebengine to finish, than > accelerating the updates of half a dozen smaller packages. That is the difficulty. I do often rebuild a new system, not trusting the existing one any longer, and a lot of time is spent fiddling with four tiny packages at a time in the early and middle stages, then benefitting from the limit of 4 once the desktop packages begin. > Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset. I set MAKEOPTS jobs > to the number of CPU threads +1 and the load average at 95%, so I can > continue using the PC without any noticeable latency. On PCs where RAM is > constrained I reduce the MAKEOPTS in /etc/portage/ package.env for any large > packages which are bound to start swapping and thrashing the disk. Interesting. Do you mean 95% of the jobs figure? I'll do some more experimenting. Meanwhile, perhaps I'll run new builds in two stages: the first without any desktop packages - I do have sets defined to enable that - and the second with them. I can't do that with emerge -e though. -- Regards, Peter.
