Re: [gentoo-user] Re: how did i get ~/26H1MJ8.txt?
thanks. that's it probably (gcc upgrade). (didn't record the date, deleted it on spot, but i did a gcc upgrade recently) rgrds, cm. ‐‐‐ Original Message ‐‐‐ On Sunday, October 20, 2019 9:10 PM, Nikos Chantziaras wrote: > On 20/10/2019 12:57, Caveman Al Toraboran wrote: > > > today, i did `ls -a ~` and found the file in title. > > its content is here: > > https://gist.github.com/2eb82a8e31a3e560abd28a2c8102865f > > any idea what is this? and how did i get it? > > (i started to worry that it might be related to the > > sks key server poisoning. specially that i get no > > gpg warnings. could it be that i am poisoning real > > good?) > > That's an automated GCC upgrade and full system rebuild script. I'd > expect something like this to be used on a managed server by automated > tooling. The filename looks like a typical random temporary filename. > > What's the date of the file?
Re: [gentoo-user] Memory manager
Mick wrote: > On Sunday, 20 October 2019 16:03:42 BST Dale wrote: >> Here's the >> thing about using swap on my rig, once it does, the system gets >> extremely slow. Even switching desktops can take a minute or longer. >> Other than trying to get to what is eating up memory and killing it, the >> system is virtually useless. Even my video stops playing. > Swapping can bring the system to its knees, but only under certain operating > scenarios. This is how I understand it works: > > Say you're browsing and keep opening tabs. The browser application will > preemptively allocate memory for more tabs, in case you carry on opening even > more tabs. Then you open yet another big application in terms of memory > usage > and start running it. The kernel will reallocate some of the browser memory > not currently utilised to the other application and keep things working > smoothly. With more applications/tabs being opened you will eventually run > out of RAM and the kernel will swap some of the memory pages to disk. The > swapping is meant to be selective, i.e. things you haven't used in a while > will be taken out of RAM and saved onto your disk. > > Under the above scenario you may notice a momentary latency on your desktop > as > data is swapped onto the disk, but afterwards the desktop should be > responsive > once more - unless more swapping is again demanded by your actions. If you > try to access an application which has had parts of its memory allocation > swapped out to disk you will notice a delay in its reactions. > > Now, in a gentoo scenario, say a mammoth compile like Chromium, with a large > count of jobs specified for it, you could end up swapping part or all of one > or more jobs into memory, only to swap it out again in order to process it. > The compile keeps swapping in and out a job at a time in order to carry on > compiling. The disk thrashing is now continuous and indeed interacting with > your desktop will be painful - potentially waiting for minutes at a time > before an application responds. The way out of this bottleneck is to either > increase your RAM, or minimise the use of memory by reducing the job count in > MAKEOPTS. Shutting down desktop applications and login out of any desktop > sessions to release RAM will also help. > > On a laptop with 4G RAM compiling Chromium is quite challenging when even a > single gcc job could grow to 3G or more. Swapping and a disk I/O bottleneck > becomes unavoidable and moving the compile of binaries to a bigger PC becomes > a rather wise solution. > > Another occasion when swapping can cause havoc is when you have a memory leak > due to some buggy application and all your RAM followed by swap is chewed up > until an OOM ensues. > > For these reasons I always set up swap on my gentoo systems. What you describe in your first scenario, that is when it is so slow and virtually unresponsive. I don't generally run out of memory when compiling since I close other stuff, like Firefox, to free up memory, even in the 16GB days. What I run into is a tab, or tabs, in Firefox that are chewing through memory like a hungry junk yard dog on a crook. When Firefox does that, it is slow, extremely slow. When I get to where I can see what is going on, it is still slow to even close or kill that process. If I get to a Konsole, I usually run htop and kill it from there. Thing is, it may take a minute even for htop to start and show the problem. That's a fairly small program but it can take a minute or two to even load and show its screen. I have two sites in particular that does this. They run for days with no problem and I may not even be using that tab or that site. Then it's like it gets mad and starts using more and more memory. I have caught it in time to just refresh the tab and it go back to normal. Once it starts using swap tho, it's very slow. I wish Firefox had a way to fix that out of control memory usage. It may not be Firefox itself doing it but it would seem it can see something isn't right and put a stop to it. It's sad when a Linux desktop has 32GBs of memory eat up like that, usually by one program at that. Still, no swap at all would result in a crash or reset. It's better than nothing but I wish the root cause could be fixed. Dale :-) :-)
[gentoo-user] Re: how did i get ~/26H1MJ8.txt?
On 20/10/2019 12:57, Caveman Al Toraboran wrote: today, i did `ls -a ~` and found the file in title. its content is here: https://gist.github.com/2eb82a8e31a3e560abd28a2c8102865f any idea what is this? and how did i get it? (i started to worry that it might be related to the sks key server poisoning. specially that i get no gpg warnings. could it be that i am poisoning real good?) That's an automated GCC upgrade and full system rebuild script. I'd expect something like this to be used on a managed server by automated tooling. The filename looks like a typical random temporary filename. What's the date of the file?
Re: [gentoo-user] Memory manager
On Sunday, 20 October 2019 16:03:42 BST Dale wrote: > Here's the > thing about using swap on my rig, once it does, the system gets > extremely slow. Even switching desktops can take a minute or longer. > Other than trying to get to what is eating up memory and killing it, the > system is virtually useless. Even my video stops playing. Swapping can bring the system to its knees, but only under certain operating scenarios. This is how I understand it works: Say you're browsing and keep opening tabs. The browser application will preemptively allocate memory for more tabs, in case you carry on opening even more tabs. Then you open yet another big application in terms of memory usage and start running it. The kernel will reallocate some of the browser memory not currently utilised to the other application and keep things working smoothly. With more applications/tabs being opened you will eventually run out of RAM and the kernel will swap some of the memory pages to disk. The swapping is meant to be selective, i.e. things you haven't used in a while will be taken out of RAM and saved onto your disk. Under the above scenario you may notice a momentary latency on your desktop as data is swapped onto the disk, but afterwards the desktop should be responsive once more - unless more swapping is again demanded by your actions. If you try to access an application which has had parts of its memory allocation swapped out to disk you will notice a delay in its reactions. Now, in a gentoo scenario, say a mammoth compile like Chromium, with a large count of jobs specified for it, you could end up swapping part or all of one or more jobs into memory, only to swap it out again in order to process it. The compile keeps swapping in and out a job at a time in order to carry on compiling. The disk thrashing is now continuous and indeed interacting with your desktop will be painful - potentially waiting for minutes at a time before an application responds. The way out of this bottleneck is to either increase your RAM, or minimise the use of memory by reducing the job count in MAKEOPTS. Shutting down desktop applications and login out of any desktop sessions to release RAM will also help. On a laptop with 4G RAM compiling Chromium is quite challenging when even a single gcc job could grow to 3G or more. Swapping and a disk I/O bottleneck becomes unavoidable and moving the compile of binaries to a bigger PC becomes a rather wise solution. Another occasion when swapping can cause havoc is when you have a memory leak due to some buggy application and all your RAM followed by swap is chewed up until an OOM ensues. For these reasons I always set up swap on my gentoo systems. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] how did i get ~/26H1MJ8.txt?
On 10/20/19 3:57 AM, Caveman Al Toraboran wrote: any idea what is this? and how did i get it? At quick glance, it looks like a script to upgrade GCC across otherwise not quite as compatible versions as possible. Nothing in it concerns me. Warning: I am relying on my uncaffeinated memory, which is known to be forgetful at times. -- Grant. . . . unix || die
Re: [gentoo-user] Re: syslog-ng 10s pause during startup
On 10/19/19 8:09 AM, Grant Edwards wrote: On 2019-10-19, Daniel Frey wrote: On 10/18/19 5:47 PM, Grant Edwards wrote: On 2019-10-18, Daniel Frey wrote: It is waiting for entropy to build. Interesting -- what does syslog-ng need entropy for? Moving mouse or typing on keyboard will speed it up but I have machines only controlled by IR so this was not helpful. Thanks, I'll try that. That doesn't seem to make any difference. It is a boot service and needs to start very early in the boot process. When I first installed it, syslog-ng was starting before it, negating any usefulness. This problem affected 4 out of 5 machines after that update. I'm not sure actually. I only found it after 15 minutes of troubleshooting while noting the time when things started/stopped. I came across a log entry with something saying "waiting for entropy" and when I noted when boot continued another entry like "entropy gathered." Hmm. No messages like that in demsg output. Check /var/log/messages too - I don't remember where I came across that in the logs. Waiting 20 minutes for a MythTV appliance to start is pretty silly though. Yea, that's definitely beyond the pale. How's MythTV these days? I ran Myth for 10+ years, but got frustrated at the lack of a small cheap silent front-end. Does the server still refuse to run on a headless machine without X11 and demand that you configure it via a MythTV client on a UI layed out for a TV? When I gave up on Myth I switched to SageTV. When Google bought that and shut it down, I switched to to Plex. Both SageTV and Plex can run on a server without X11 can be configured via a web UI. I haven't had to run the setup in like 8 years but I suspect it hasn't changed. Some frontends use the kodi plugin to access the recordings. Dan
Re: [gentoo-user] Memory manager
Mick wrote: > On Sunday, 20 October 2019 12:59:03 BST Wols Lists wrote: > >> Well, I do all my emerges on tmpfs, so if things like LO, firefox et al >> need maybe 10GB, I need at least that available ... (that said, 16GB ram >> could probably do it without needing swap :-) > Anecdotal evidence suggests 16G RAM may not be enough: > > A number of times I was emerging chromium with USE="jumbo-build" enabled on a > PC with 16G RAM, while a user was running Kmail, firefox and some other apps. > > FF in particular is a hog, as it pre-emptively allocates more RAM as you keep > opening more tabs. With 15-20 FF tabs open and MAKEOPTS="-j5", all RAM was > exhausted and swap started being used heavily by portage, thrashing the disk > as jobs were being swapped I/O. With no swap the emerge would have been > killed with an OOM. > > >> But seeing as I try to fill up my mobo ram, my disks are mirrored, and I >> still try and stick to the "twice ram" rule, this setup means any >> upgrades/changes to the computer means I don't break that rule. If it's >> overkill, well disk is cheap (and I can always nick a swap partition and >> repurpose it temporarily if needs be :-) >> >> Cheers, >> Wol > Quite, disks are cheap and having more swap won't break the bank. On PCs > with > limited RAM I often end up adding more swap space and reducing job counts to > get big emerges through. > This is some things I've noticed. I have swap but set it to not be used unless something bad happens. I think swappiness is set to like 10 or 20. It's rare that any swap is ever used. When it does, it is usually some tab in Firefox that has went wild eating up memory. Here's the thing about using swap on my rig, once it does, the system gets extremely slow. Even switching desktops can take a minute or longer. Other than trying to get to what is eating up memory and killing it, the system is virtually useless. Even my video stops playing. For me and my experience, I only want swap available to prevent the system from crashing and resetting itself. It gives me a little time to kill the memory hog and not have a crash. Other than that, I don't want swap used because it is just to slow. I have 32GBs so it's rare that I run out anymore. When I had 16GBs tho, Firefox was a regular abuser. Seamonkey never runs wild exactly but sometimes I close and restart it to force it to release memory. What I wish, Firefox and even Seamonkey would stop running wild with memory. Sometimes it gets really hungry and takes up several GBs when there is no reason to do so. Like with Mick tho, 16GBs got to where it wasn't enough. I had to close web browsers during large compiles. LOo, Firefox, Seamonkey etc etc. I have emerge's work directory on tmpfs here. That's my experience with swap, real world. Dale :-) :-)
[gentoo-user] OCR for music (OMR)
Hi All, I'm on the lookout for an application which can convert musical notation from scanned bitmap copies/pdf files to midi files. Apparently there are some apps in the Apple store, but I have not yet found anything in portage. Do you have experience using any tools for this purpose? Are there any in portage/overlays? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Memory manager
On Sunday, 20 October 2019 12:59:03 BST Wols Lists wrote: > Well, I do all my emerges on tmpfs, so if things like LO, firefox et al > need maybe 10GB, I need at least that available ... (that said, 16GB ram > could probably do it without needing swap :-) Anecdotal evidence suggests 16G RAM may not be enough: A number of times I was emerging chromium with USE="jumbo-build" enabled on a PC with 16G RAM, while a user was running Kmail, firefox and some other apps. FF in particular is a hog, as it pre-emptively allocates more RAM as you keep opening more tabs. With 15-20 FF tabs open and MAKEOPTS="-j5", all RAM was exhausted and swap started being used heavily by portage, thrashing the disk as jobs were being swapped I/O. With no swap the emerge would have been killed with an OOM. > But seeing as I try to fill up my mobo ram, my disks are mirrored, and I > still try and stick to the "twice ram" rule, this setup means any > upgrades/changes to the computer means I don't break that rule. If it's > overkill, well disk is cheap (and I can always nick a swap partition and > repurpose it temporarily if needs be :-) > > Cheers, > Wol Quite, disks are cheap and having more swap won't break the bank. On PCs with limited RAM I often end up adding more swap space and reducing job counts to get big emerges through. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Memory manager
On 20/10/19 10:59, Mick wrote: > On Sunday, 20 October 2019 00:35:56 BST Wol's lists wrote: > >> The original swap algorithm NEEDED twice ram as swap. And when Linus >> ripped out all the "optimisation", the vanilla kernels only needed to >> touch swap, and if they didn't have twice ram they would crash. > > Was this also the time when the default swappiness was set at 60? > > >> At that point, the recommendation changed to "no swap is fine, twice or >> more is fine, just don't have swap less than twice ram". > > Are you sure of this? At least on current kernels (I'm currently on 4.19.72- > gentoo) the overcommit_accounting kernel mechanism using a heuristic over- > commit memory handling is set at 0, which refuses wilder over commits, but > allows more measured over commits to use swap space. > > > From the above and without further experimentation I assume having a swap > slightly larger than my RAM is more than adequate for a desktop, including > hibernating on swap. > As per Richard Brown, the current recommendation seems to be even less than that ... "2GB is plenty". That's the SUSE default. > >> My personal rule is to take the motherboard's max ram, double it, and >> create a swap partition that size on every disk. So my current desktop >> system has 80GB of ram/swap - 4x4GB slots times 2 disk drives. And my >> new system has 4x8GB so that'll be 160GB!!! HOWEVER - Richard Brown of >> SUSE said that's dangerous - if somebody fork-bombs you it'll take a >> long time to fill that much swap and regaining control of your system >> could well be a big red switch job. >> > > Each to their own, but I tend to think this huge amount of swap is probably > excessive, unless you're running some scientific applications which require > big over commits for their calculations. > Well, I do all my emerges on tmpfs, so if things like LO, firefox et al need maybe 10GB, I need at least that available ... (that said, 16GB ram could probably do it without needing swap :-) But seeing as I try to fill up my mobo ram, my disks are mirrored, and I still try and stick to the "twice ram" rule, this setup means any upgrades/changes to the computer means I don't break that rule. If it's overkill, well disk is cheap (and I can always nick a swap partition and repurpose it temporarily if needs be :-) Cheers, Wol
Re: [gentoo-user] Memory manager
On Sunday, 20 October 2019 00:35:56 BST Wol's lists wrote: > The original swap algorithm NEEDED twice ram as swap. And when Linus > ripped out all the "optimisation", the vanilla kernels only needed to > touch swap, and if they didn't have twice ram they would crash. Was this also the time when the default swappiness was set at 60? > At that point, the recommendation changed to "no swap is fine, twice or > more is fine, just don't have swap less than twice ram". Are you sure of this? At least on current kernels (I'm currently on 4.19.72- gentoo) the overcommit_accounting kernel mechanism using a heuristic over- commit memory handling is set at 0, which refuses wilder over commits, but allows more measured over commits to use swap space. $ cat /proc/sys/vm/overcommit_memory 0 When set at 2, where no over-commit of memory is allowed, the mechanism will be informed by the overcommit_ratio by default set at 50%, to calculate how much RAM will be used by malloc. In this case, all swap will be used but only 50% of RAM. $ cat /proc/sys/vm/overcommit_ratio 50 Unless someone has purposefully set their overcommit_memory to 1, where potentially wild over commits are allowed and the risk of OOM increases, the kernel will not allow over-committing of RAM without checking how much RAM + swap is available. On my system with default overcommit_memory settings I have: $ free --kilo total usedfree shared buff/cache available Mem: 15975043 13598841384940722528 76575114126170 Swap: 17179865 017179865 but only 24577524 is reported as the commit limit: $ grep CommitLimit /proc/meminfo CommitLimit:24577524 kB This is all swap and a percentage of RAM: 24577524 - 17179865 = 7397659 of RAM, or 46.3%. >From the above and without further experimentation I assume having a swap slightly larger than my RAM is more than adequate for a desktop, including hibernating on swap. > My personal rule is to take the motherboard's max ram, double it, and > create a swap partition that size on every disk. So my current desktop > system has 80GB of ram/swap - 4x4GB slots times 2 disk drives. And my > new system has 4x8GB so that'll be 160GB!!! HOWEVER - Richard Brown of > SUSE said that's dangerous - if somebody fork-bombs you it'll take a > long time to fill that much swap and regaining control of your system > could well be a big red switch job. > > Cheers, > Wol Each to their own, but I tend to think this huge amount of swap is probably excessive, unless you're running some scientific applications which require big over commits for their calculations. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
[gentoo-user] how did i get ~/26H1MJ8.txt?
today, i did `ls -a ~` and found the file in title. its content is here: https://gist.github.com/2eb82a8e31a3e560abd28a2c8102865f any idea what is this? and how did i get it? (i started to worry that it might be related to the sks key server poisoning. specially that i get no gpg warnings. could it be that i am poisoning real good?) rgrds, cm. Sent with [ProtonMail](https://protonmail.com) Secure Email.
