Re: [gentoo-user] Jobs and load-average
On Wednesday, February 15, 2023 10:56:22 AM CET Peter Humphrey wrote: > Hello list, > > Not long ago I read that we should allow 2GB RAM for every emerge job - that > is, we should divide our RAM size by 2 to get the maximum number of > simultaneous jobs. I'm trying to get that right, but I'm not there yet. > > I have these entries in make.conf: > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > unmerge-warn --ke> > MAKEOPTS="-j16" > > Today, though, I saw load averages going up to 72. Can anyone suggest better > values to suit my 24 threads and 64GB RAM? One other item I missed in the replies: "--load-average" is also a valid option for make. If you want to keep the load down, I would suggest adding this to MAKEOPTS as well: MAKEOPTS="--jobs=16 --load-average=32" I write the options out full because I had some weird errors in the past because the "-j" wasn't handled correctly at some point. -- Joost
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
On Wed, Feb 15, 2023 at 07:20:37PM +, Wol wrote > what are they going to do about "eselect kernel set ..." then? > > It's bad enough depclean deleting the active kernel if you don't watch > out, without something deciding to install a non-existent kernel and > deleting the live one :-) I have my own hand-coded script that runs "emerge --pretend --depclean" and tweaks/filters the output into another script called "cleanscript". I've set it to filter out "gentoo-sources". I then inspect "cleanscript" before running it. And, oh yeah, depclean wants to remove nano. I had to "emerge -n nano" to protect it. -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
Hi, Wol writes: > On 15/02/2023 15:51, Daniel Frey wrote: >> On 2/15/23 06:10, Michael Orlitzky wrote: >>> On 2023-02-15 08:11:46, Neil Bothwick wrote: If, as you say, it will eventually replace eselect, there is no more bloat, just different bloat. It's still just a bunch of symlinks, but managed differently. >>> >>> Should be less, since you already have portage installed but not >>> necessarily eselect-whatever. >>> >> I didn't even know eselect-whatever was even an option until this >> post... It's not something I've ever used. >> It does (at least to me) make sense for the package manager to enforce these >> selections rather than some optional tool though. >> > what are they going to do about "eselect kernel set ..." then? > > It's bad enough depclean deleting the active kernel if you don't watch out, > without something deciding to install a non-existent kernel and deleting the > live one :-) This is part of the motivation behind the dist-kernel project. See: https://wiki.gentoo.org/wiki/Project:Distribution_Kernel As an anecdote, I haven't thought about what my kernel and modules are doing in a very long time, and I use multiple out of tree modules. Happy hacking and have a great day. -- Arsen Arsenović signature.asc Description: PGP signature
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
On 15/02/2023 15:51, Daniel Frey wrote: On 2/15/23 06:10, Michael Orlitzky wrote: On 2023-02-15 08:11:46, Neil Bothwick wrote: If, as you say, it will eventually replace eselect, there is no more bloat, just different bloat. It's still just a bunch of symlinks, but managed differently. Should be less, since you already have portage installed but not necessarily eselect-whatever. I didn't even know eselect-whatever was even an option until this post... It's not something I've ever used. It does (at least to me) make sense for the package manager to enforce these selections rather than some optional tool though. what are they going to do about "eselect kernel set ..." then? It's bad enough depclean deleting the active kernel if you don't watch out, without something deciding to install a non-existent kernel and deleting the live one :-) Cheers, Wol
[gentoo-user] Re: my 5.15.93 kernel keeps rebooting
On 2023-02-15, Grant Edwards wrote: > [1] For this purpose you want a plain old UART on the motherboard type > seial port. You'd be surprised how many motherboards still have > them. Even though they're never brought out to a DB9 connector on > the back panel, there's often an 8-pin header on the edge of the > board somewhere, so you'd need one of these: Oops, it's a 10pin (2x5) header not an 8-pin header, as I'm sure you'd have figured out.
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
On 2/15/23 06:10, Michael Orlitzky wrote: On 2023-02-15 08:11:46, Neil Bothwick wrote: If, as you say, it will eventually replace eselect, there is no more bloat, just different bloat. It's still just a bunch of symlinks, but managed differently. Should be less, since you already have portage installed but not necessarily eselect-whatever. I didn't even know eselect-whatever was even an option until this post... It's not something I've ever used. It does (at least to me) make sense for the package manager to enforce these selections rather than some optional tool though. Dan
Re: [gentoo-user] Jobs and load-average
On Wednesday, 15 February 2023 15:12:28 GMT Michael wrote: > You can have both a generic MAKEOPTS in make.conf, which suits your base > case of emerge operations and will not cause your PC to explode when > combined with EMERGE_DEFAULT_OPTS, as well as package specific MAKEOPTS in > package.env to finely tune individual package requirements. Yes, I assumed so, and I've now set it up that way. -- Regards, Peter.
Re: [gentoo-user] Re: my 5.15.93 kernel keeps rebooting
On Wed, 15 Feb 2023 09:50:27 -0500, Grant Edwards wrote: > > On 2023-02-14, Rich Freeman wrote: > > > Where are you getting this from, the system log/journal? This doesn't > > seem like a clean shutdown, so if it is a kernel PANIC I wouldn't > > expect the most critical info to be in the log (since it will stop > > syncing to protect the filesystem). The details you need probably > > will be displayed on the console briefly. You can also enable a > > network console, which will send the dmesg output continuously over > > UDP to another device. This won't be interrupted by a PANIC unless > > there is some issue with the hardware or networking stack. > > If you've got a serial port[1], you could also set up serial > logging. Though using serial ports have become a bit of a lost art, > the serial console code in the kernel is pretty carefully designed to > be the last man standing when things start to die. It's possible > (though I wouldn't say probable) that a serial console will be able to > show you stuff closer to the event horizon than a network console can. > > Anyway, since still I'm in the serial port business (yes, there are > still plenty of people using serial ports in industrial settings) I > had to mention it... > > [1] For this purpose you want a plain old UART on the motherboard type > seial port. You'd be surprised how many motherboards still have > them. Even though they're never brought out to a DB9 connector on > the back panel, there's often an 8-pin header on the edge of the > board somewhere, so you'd need one of these: > > > https://www.amazon.com/C2G-27550-Adapter-Bracket-Motherboards/dp/B0002J27R8/ I do have one which I use for my speech synthesizer. I also have one on my other box which I could hook up -- if I can find my null modem cable. I think I will try the netconsole first and the serial console if that does not work. Thanks for the hint. -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici wb2una cov...@ccs.covici.com
Re: [gentoo-user] Jobs and load-average
On Wednesday, 15 February 2023 14:31:25 GMT Peter Humphrey wrote: > On Wednesday, 15 February 2023 13:18:24 GMT Rich Freeman wrote: > > On Wed, Feb 15, 2023 at 4:56 AM Peter Humphrey > > wrote: > > > Not long ago I read that we should allow 2GB RAM for every emerge job - > > > that is, we should divide our RAM size by 2 to get the maximum number of > > > simultaneous jobs. I'm trying to get that right, but I'm not there yet. > > > > > > I have these entries in make.conf: > > > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > > > unmerge-warn --ke> > > > MAKEOPTS="-j16" > > > > > > Today, though, I saw load averages going up to 72. Can anyone suggest > > > better values to suit my 24 threads and 64GB RAM? > > > > First, keep in mind that --jobs=16 + -j16 can result in up to 256 > > (16*16) tasks running at once. Of course, that is worst case and most > > of the time you'll have way less than that. > > > > Keep in mind that you need to consider available RAM and not just > > total RAM. Run free under the conditions where you typically run > > emerge and see how much available memory it displays. Depending on > > what you have running it could be much lower than 64GB. > > > > Beyond that, unfortunately this is hard to deal with beyond just > > figuring out what needs more RAM and making exceptions in package.env. > > > > Also, RAM pressure could also come from the build directory if it is > > on tmpfs, which of course many of us use. > > > > Some packages that I build with either a greatly reduced -j setting or > > a non-tmpfs build directory are: > > sys-cluster/ceph > > dev-python/scipy > > dev-python/pandas > > app-office/calligra > > net-libs/nodejs > > dev-qt/qtwebengine > > dev-qt/qtwebkit > > dev-lang/spidermonkey > > www-client/chromium > > app-office/libreoffice > > sys-devel/llvm > > dev-lang/rust (I use the rust binary these days as this has gotten > > really out of hand) > > x11-libs/gtk+ > > > > These are just packages I've had issues with at some point, and it is > > possible that some of these packages no longer use as much memory > > today. > > Thank you all. I can see what I'm doing better now. (Politicians aren't the > only ones who can be ambiguous!) > > I'll start by picking up the point I'd missed - putting MAKEOPTS in > package.env. You can have both a generic MAKEOPTS in make.conf, which suits your base case of emerge operations and will not cause your PC to explode when combined with EMERGE_DEFAULT_OPTS, as well as package specific MAKEOPTS in package.env to finely tune individual package requirements. signature.asc Description: This is a digitally signed message part.
[gentoo-user] Re: my 5.15.93 kernel keeps rebooting
On 2023-02-14, Rich Freeman wrote: > Where are you getting this from, the system log/journal? This doesn't > seem like a clean shutdown, so if it is a kernel PANIC I wouldn't > expect the most critical info to be in the log (since it will stop > syncing to protect the filesystem). The details you need probably > will be displayed on the console briefly. You can also enable a > network console, which will send the dmesg output continuously over > UDP to another device. This won't be interrupted by a PANIC unless > there is some issue with the hardware or networking stack. If you've got a serial port[1], you could also set up serial logging. Though using serial ports have become a bit of a lost art, the serial console code in the kernel is pretty carefully designed to be the last man standing when things start to die. It's possible (though I wouldn't say probable) that a serial console will be able to show you stuff closer to the event horizon than a network console can. Anyway, since still I'm in the serial port business (yes, there are still plenty of people using serial ports in industrial settings) I had to mention it... [1] For this purpose you want a plain old UART on the motherboard type seial port. You'd be surprised how many motherboards still have them. Even though they're never brought out to a DB9 connector on the back panel, there's often an 8-pin header on the edge of the board somewhere, so you'd need one of these: https://www.amazon.com/C2G-27550-Adapter-Bracket-Motherboards/dp/B0002J27R8/ -- Grant
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
On Wed, Feb 15, 2023 at 9:10 AM Michael Orlitzky wrote: > > On 2023-02-15 08:11:46, Neil Bothwick wrote: > > > > If, as you say, it will eventually replace eselect, there is no more > > bloat, just different bloat. It's still just a bunch of symlinks, but > > managed differently. > > > > Should be less, since you already have portage installed but not > necessarily eselect-whatever. > The symlinks are all associated with packages as well, which means that when you uninstall things that will get rid of the symlinks as well. This is really just a best practice all-around. I have a Gentoo system I've been maintaining for a while and I occasionally find orphaned stuff poking around because of special cases of things that weren't managed by the package manager, and so when things were obsoleted they stuck around. The news is needed precisely because the migration involves having the package manager install a bunch of stuff over files not owned by any package. That triggers a warning, but only because the files were in a less than ideal state to start. Things like this and the new user/group packages also reduce the complexity of dependency management because they just turn everything into a package dependency. Less special cases. -- Rich
Re: [gentoo-user] Jobs and load-average
On Wednesday, 15 February 2023 13:18:24 GMT Rich Freeman wrote: > On Wed, Feb 15, 2023 at 4:56 AM Peter Humphrey wrote: > > Not long ago I read that we should allow 2GB RAM for every emerge job - > > that is, we should divide our RAM size by 2 to get the maximum number of > > simultaneous jobs. I'm trying to get that right, but I'm not there yet. > > > > I have these entries in make.conf: > > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > > unmerge-warn --ke> > > MAKEOPTS="-j16" > > > > Today, though, I saw load averages going up to 72. Can anyone suggest > > better values to suit my 24 threads and 64GB RAM? > > First, keep in mind that --jobs=16 + -j16 can result in up to 256 > (16*16) tasks running at once. Of course, that is worst case and most > of the time you'll have way less than that. > > Keep in mind that you need to consider available RAM and not just > total RAM. Run free under the conditions where you typically run > emerge and see how much available memory it displays. Depending on > what you have running it could be much lower than 64GB. > > Beyond that, unfortunately this is hard to deal with beyond just > figuring out what needs more RAM and making exceptions in package.env. > > Also, RAM pressure could also come from the build directory if it is > on tmpfs, which of course many of us use. > > Some packages that I build with either a greatly reduced -j setting or > a non-tmpfs build directory are: > sys-cluster/ceph > dev-python/scipy > dev-python/pandas > app-office/calligra > net-libs/nodejs > dev-qt/qtwebengine > dev-qt/qtwebkit > dev-lang/spidermonkey > www-client/chromium > app-office/libreoffice > sys-devel/llvm > dev-lang/rust (I use the rust binary these days as this has gotten > really out of hand) > x11-libs/gtk+ > > These are just packages I've had issues with at some point, and it is > possible that some of these packages no longer use as much memory > today. Thank you all. I can see what I'm doing better now. (Politicians aren't the only ones who can be ambiguous!) I'll start by picking up the point I'd missed - putting MAKEOPTS in package.env. -- Regards, Peter.
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
On 2023-02-15 08:11:46, Neil Bothwick wrote: > > If, as you say, it will eventually replace eselect, there is no more > bloat, just different bloat. It's still just a bunch of symlinks, but > managed differently. > Should be less, since you already have portage installed but not necessarily eselect-whatever.
Re: [gentoo-user] Jobs and load-average
On Wed, Feb 15, 2023 at 4:56 AM Peter Humphrey wrote: > > Not long ago I read that we should allow 2GB RAM for every emerge job - that > is, we should divide our RAM size by 2 to get the maximum number of > simultaneous jobs. I'm trying to get that right, but I'm not there yet. > > I have these entries in make.conf: > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > unmerge-warn --ke> > MAKEOPTS="-j16" > > Today, though, I saw load averages going up to 72. Can anyone suggest better > values to suit my 24 threads and 64GB RAM? First, keep in mind that --jobs=16 + -j16 can result in up to 256 (16*16) tasks running at once. Of course, that is worst case and most of the time you'll have way less than that. Keep in mind that you need to consider available RAM and not just total RAM. Run free under the conditions where you typically run emerge and see how much available memory it displays. Depending on what you have running it could be much lower than 64GB. Beyond that, unfortunately this is hard to deal with beyond just figuring out what needs more RAM and making exceptions in package.env. Also, RAM pressure could also come from the build directory if it is on tmpfs, which of course many of us use. Some packages that I build with either a greatly reduced -j setting or a non-tmpfs build directory are: sys-cluster/ceph dev-python/scipy dev-python/pandas app-office/calligra net-libs/nodejs dev-qt/qtwebengine dev-qt/qtwebkit dev-lang/spidermonkey www-client/chromium app-office/libreoffice sys-devel/llvm dev-lang/rust (I use the rust binary these days as this has gotten really out of hand) x11-libs/gtk+ These are just packages I've had issues with at some point, and it is possible that some of these packages no longer use as much memory today. -- Rich
Re: [gentoo-user] Jobs and load-average
On Wednesday, 15 February 2023 11:31:49 GMT Peter Böhm wrote: > Am Mittwoch, 15. Februar 2023, 10:56:22 CET schrieb Peter Humphrey: > > Hello list, > > > > Not long ago I read that we should allow 2GB RAM for every emerge job - > > that is, we should divide our RAM size by 2 to get the maximum number of > > simultaneous jobs. I'm trying to get that right, but I'm not there yet. > > > > I have these entries in make.conf: > > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > > unmerge-warn --ke> The above determine how may ebuilds will be emerged in parallel. If you are rebuilding your whole system with hundreds of packages stacking up to be emerged, then having as high as 16 packages being emerged in parallel could be advantageous. > > MAKEOPTS="-j16" This determines how many MAKE jobs will run in parallel in any one emerge. Large packages like chromium will benefit from maximising the number of jobs here, as long as you have enough RAM. Given you have 24 threads and your RAM is 64GB, you should be able to ratchet this up to -j24, but not if you specify a high EMERGE_DEFAULT_OPTS at the same time, or if the compiler is eating up more than 2G per process. > > Today, though, I saw load averages going up to 72. Can anyone suggest > > better values to suit my 24 threads and 64GB RAM? Since you have specified up to 16 parallel emerges and each one could run up to 16 MAKE jobs, you can understand why you would soon find loads escalating and your machine becoming unresponsive. You should consider what is more important for you, emerging as many packages in parallel as possible, or emerging any one large package as fast as possible. Two extreme examples would be setting EMERGE_DEFAULT_OPTS at "--jobs 24", with MAKEOPTS at "-j1", or conversely setting EMERGE_DEFAULT_OPTS at "--jobs 1", with MAKEOPTS at "-j24". On my old and slow laptop with only 4 threads and 16G of RAM my priority is to finish large packages faster. I leave EMERGE_DEFAULT_OPTS unset, while specifying MAKEOPTS="-j5 -l4.8". This uses a job number I determined by trial and error, building ffmpeg repeatedly by progressively increasing the number for -j from 1 to 12. The two faster times were achieved with -j5 and -j10, which aligns with the old myth of using CPU+1. Regarding RAM being used being ~2G per MAKE job, this fluctuates with successive compiler versions. I have seen up to 3.4G of RAM per process, while emerging chromium. For such huge packages which cause excessive swapping, unresponsiveness and thrashing of disk, I limit the MAKEOPTS to 3 in package.env. > Maybe you are interested in this wiki article: > > https://wiki.gentoo.org/wiki/User:Pietinger/Tutorials/Optimize_compile_times > > Regards, > Peter I'd start by reading the suggestions in this article first, which is a good introduction to the concepts involved: https://wiki.gentoo.org/wiki/MAKEOPTS signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Jobs and load-average
On Wednesday, 15 February 2023 09:56:22 GMT Peter Humphrey wrote: > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > unmerge-warn --ke> That should have been: EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- unmerge-warn --keep-going --nospinner" -- Regards, Peter.
Re: [gentoo-user] Jobs and load-average
Am Mittwoch, 15. Februar 2023, 10:56:22 CET schrieb Peter Humphrey: > Hello list, > > Not long ago I read that we should allow 2GB RAM for every emerge job - that > is, we should divide our RAM size by 2 to get the maximum number of > simultaneous jobs. I'm trying to get that right, but I'm not there yet. > > I have these entries in make.conf: > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- > unmerge-warn --ke> > MAKEOPTS="-j16" > > Today, though, I saw load averages going up to 72. Can anyone suggest better > values to suit my 24 threads and 64GB RAM? Maybe you are interested in this wiki article: https://wiki.gentoo.org/wiki/User:Pietinger/Tutorials/Optimize_compile_times Regards, Peter
[gentoo-user] Jobs and load-average
Hello list, Not long ago I read that we should allow 2GB RAM for every emerge job - that is, we should divide our RAM size by 2 to get the maximum number of simultaneous jobs. I'm trying to get that right, but I'm not there yet. I have these entries in make.conf: EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet- unmerge-warn --ke> MAKEOPTS="-j16" Today, though, I saw load averages going up to 72. Can anyone suggest better values to suit my 24 threads and 64GB RAM? -- Regards, Peter.
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
Hi, "Walter Dnes" writes: > A whole bunch of busy-work for emerge, and nothing in the news item > indicates it's really necessary for the average user. Howsabout... It's definitely necessary. Those packages provide links for vital parts of the filesystem, like /bin/sh. Why do you want to remove them? If there's something we failed to consider when implementing app-alternatives, please let us know and we'll try to rectify the issue. > * manually zapping with "rm -rf /var/db/repos/gentoo/app-alternatives" > * and then include "app-alternatives" in the file pointed to by > PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=" > > Am I missing something obvious that would cause problems? Have a great day. -- Arsen Arsenović signature.asc Description: PGP signature
Re: [gentoo-user] Is it OK to get rid of app-alternatives/* ?
On Wed, 15 Feb 2023 02:44:47 -0500, Walter Dnes wrote: > After thoroughly reading the docs at... > https://www.gentoo.org/support/news-items/2022-12-27-alternatives-introduction.html > it looks like the hand of him-who-must-not-be-named. Rather than > provide special support for the 1% extreme edge cases, the remaining 99% > of regular users will be dragged through the change. More bloat; and > eselect is on the road to eventual deprecation. "Systems will be more robust and desired system configuration can be achieved using the package manager rather than manual steps outside of it." Sounds quite reasonable to me. As does being able to control everything from within Portage, with USE flags, rather than messing around with eselect. If, as you say, it will eventually replace eselect, there is no more bloat, just different bloat. It's still just a bunch of symlinks, but managed differently. -- Neil Bothwick Inland Revenue: We've got what it takes to take what you've got! pgpDu0Q1pEoZU.pgp Description: OpenPGP digital signature
