Re: Out of memory killer misconfigured?
On Wed 20 Apr 2022 at 18:45:47 (+0100), piorunz wrote:
> On 20/04/2022 17:11, Jonathan Dowland wrote:
> > On Wed, Apr 20, 2022 at 04:23:38PM +0100, piorunz wrote:
> > > Sorry but this happened to me a few times, each tome with the same Wine
> > > program. it just loves to eat all available memory when its doing heavy
> > > computations. When I am not careful and I start too many threads, is
> > > eats all memory. My system gets killed every single time. I am not
> > > extrapolating anything. Misbehaving app should get killed, but instead,
> > > Linux kills itself.
> >
> > Ok, not one experience, but one scenario. And there are plausible
> > reasons why this might be happening as outlined in my other mail to the
> > thread (OOM adjustments to make the killer skip over the process).
> > Occam's razor suggests something about your particular setup versus
> > the OOM killer simply being as bad as you think it is. FWIW, I invoke
> > the OOM killer a lot recently (due to some scientific experiments) and
> > the sensible processes were killed in my case, every time, leaving my
> > desktop functional.
> >
> > More data about your setup is needed to fathom out what's going on.
> >
> I didn't configured OOM. I use default Debian Testing with KDE. I
> changed nothing apart from user desktop things. MY situation is not "a
> scenario". It's what everyone can reproduce, just install Wine and
> MetaTester5 program, I can guide you. I cannot guarantee this will
> happen with non-Wine programs.
With respect, the combination of testing + KDE + Wine + MetaTester5
is difficult to categorise as more than one scenario.
> So I just reproduced this again, system crashed totally because I forgot
> to apply choom --adjust 1000 to all metatester processes. Reboot. After
> reboot, I started MT again and applied the following:
>
> renice 19 `pidof metatester64.ex`
> choom -p `pidof metatester64.ex | awk {'print $1'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $2'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $3'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $4'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $5'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $6'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $7'}` --adjust 1000
>
> I spawned more than 8 processes to saturate RAM quickly. Meaning, some
> processes were not adjusted to 1000 score. After 20 seconds, OOM event
> happened.
When I start my window manager, I use a construction like this:
if [ -x /usr/bin/fvwm ]; then
mv -f "$HOME/.xsession-fvwm-$Displaynumber-log"
"$HOME/.xsession-fvwm-$Displaynumber-log~"
exec /usr/bin/fvwm > "$HOME/.xsession-fvwm-$Displaynumber-log" 2>&1 &
Wmpid=$!
elif [ -x …
…
…
else
printf '%s\n' "Error - no window manager found"
fi
so that .xsession can avoid terminating with:
wait $Wmpid
Would it be sensible for you to use a similar construction so that you
systematically catch /all/ of these processes with their choom commands.
> And what Linux does? It has at least 8 processes with 1000 score, but
> let's see what is does:
>
> $ sudo dmesg | grep "Out of memory"
> [ 704.018238] Out of memory: Killed process 39674 (metatester64.ex)
> total-vm:2939256kB, anon-rss:250524kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:1344kB oom_score_adj:1000
> [ 708.528663] Out of memory: Killed process 25882 (QtWebEngineProc)
> total-vm:5451920kB, anon-rss:20588kB, file-rss:0kB, shmem-rss:116kB,
> UID:1000 pgtables:772kB oom_score_adj:300
> [ 709.811945] Out of memory: Killed process 23411 (krunner-keepass)
> total-vm:52680kB, anon-rss:12860kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:140kB oom_score_adj:200
> [ 710.788999] Out of memory: Killed process 23413 (pipewire-media-)
> total-vm:170052kB, anon-rss:9492kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:248kB oom_score_adj:200
> [ 712.748691] Out of memory: Killed process 23618 (kactivitymanage)
> total-vm:550696kB, anon-rss:8848kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:244kB oom_score_adj:200
> [ 714.856773] Out of memory: Killed process 23414 (pulseaudio)
> total-vm:1384772kB, anon-rss:8508kB, file-rss:0kB, shmem-rss:260kB,
> UID:1000 pgtables:320kB oom_score_adj:200
> [ 716.051274] Out of memory: Killed process 23560 (kglobalaccel5)
> total-vm:282368kB, anon-rss:6728kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:256kB oom_score_adj:200
> [ 716.082495] Out of memory: Killed process 23412 (pipewire)
> total-vm:53124kB, anon-rss:4484kB, file-rss:0kB, shmem-rss:52kB,
> UID:1000 pgtables:104kB oom_score_adj:200
> [ 717.965930] Out of memory: Killed process 23867 (ksystemstats)
> total-vm:166540kB, anon-rss:3844kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:176kB oom_score_adj:200
> [ 718.775772] Out of memory: Killed process 23703 (kscreen_backend)
> total-vm:227996kB, anon-rss:3060kB, file-rss:0kB, shmem-r
Re: Out of memory killer misconfigured?
On 20/04/2022 17:11, Jonathan Dowland wrote:
On Wed, Apr 20, 2022 at 04:23:38PM +0100, piorunz wrote:
Sorry but this happened to me a few times, each tome with the same Wine
program. it just loves to eat all available memory when its doing heavy
computations. When I am not careful and I start too many threads, is
eats all memory. My system gets killed every single time. I am not
extrapolating anything. Misbehaving app should get killed, but instead,
Linux kills itself.
Ok, not one experience, but one scenario. And there are plausible
reasons why this might be happening as outlined in my other mail to the
thread (OOM adjustments to make the killer skip over the process).
Occam's razor suggests something about your particular setup versus
the OOM killer simply being as bad as you think it is. FWIW, I invoke
the OOM killer a lot recently (due to some scientific experiments) and
the sensible processes were killed in my case, every time, leaving my
desktop functional.
More data about your setup is needed to fathom out what's going on.
I didn't configured OOM. I use default Debian Testing with KDE. I
changed nothing apart from user desktop things. MY situation is not "a
scenario". It's what everyone can reproduce, just install Wine and
MetaTester5 program, I can guide you. I cannot guarantee this will
happen with non-Wine programs.
So I just reproduced this again, system crashed totally because I forgot
to apply choom --adjust 1000 to all metatester processes. Reboot. After
reboot, I started MT again and applied the following:
renice 19 `pidof metatester64.ex`
choom -p `pidof metatester64.ex | awk {'print $1'}` --adjust 1000
choom -p `pidof metatester64.ex | awk {'print $2'}` --adjust 1000
choom -p `pidof metatester64.ex | awk {'print $3'}` --adjust 1000
choom -p `pidof metatester64.ex | awk {'print $4'}` --adjust 1000
choom -p `pidof metatester64.ex | awk {'print $5'}` --adjust 1000
choom -p `pidof metatester64.ex | awk {'print $6'}` --adjust 1000
choom -p `pidof metatester64.ex | awk {'print $7'}` --adjust 1000
I spawned more than 8 processes to saturate RAM quickly. Meaning, some
processes were not adjusted to 1000 score. After 20 seconds, OOM event
happened.
And what Linux does? It has at least 8 processes with 1000 score, but
let's see what is does:
$ sudo dmesg | grep "Out of memory"
[ 704.018238] Out of memory: Killed process 39674 (metatester64.ex)
total-vm:2939256kB, anon-rss:250524kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:1344kB oom_score_adj:1000
[ 708.528663] Out of memory: Killed process 25882 (QtWebEngineProc)
total-vm:5451920kB, anon-rss:20588kB, file-rss:0kB, shmem-rss:116kB,
UID:1000 pgtables:772kB oom_score_adj:300
[ 709.811945] Out of memory: Killed process 23411 (krunner-keepass)
total-vm:52680kB, anon-rss:12860kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:140kB oom_score_adj:200
[ 710.788999] Out of memory: Killed process 23413 (pipewire-media-)
total-vm:170052kB, anon-rss:9492kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:248kB oom_score_adj:200
[ 712.748691] Out of memory: Killed process 23618 (kactivitymanage)
total-vm:550696kB, anon-rss:8848kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:244kB oom_score_adj:200
[ 714.856773] Out of memory: Killed process 23414 (pulseaudio)
total-vm:1384772kB, anon-rss:8508kB, file-rss:0kB, shmem-rss:260kB,
UID:1000 pgtables:320kB oom_score_adj:200
[ 716.051274] Out of memory: Killed process 23560 (kglobalaccel5)
total-vm:282368kB, anon-rss:6728kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:256kB oom_score_adj:200
[ 716.082495] Out of memory: Killed process 23412 (pipewire)
total-vm:53124kB, anon-rss:4484kB, file-rss:0kB, shmem-rss:52kB,
UID:1000 pgtables:104kB oom_score_adj:200
[ 717.965930] Out of memory: Killed process 23867 (ksystemstats)
total-vm:166540kB, anon-rss:3844kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:176kB oom_score_adj:200
[ 718.775772] Out of memory: Killed process 23703 (kscreen_backend)
total-vm:227996kB, anon-rss:3060kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:164kB oom_score_adj:200
[ 718.792339] Out of memory: Killed process 23417 (dbus-daemon)
total-vm:10552kB, anon-rss:1332kB, file-rss:0kB, shmem-rss:0kB, UID:1000
pgtables:60kB oom_score_adj:200
[ 719.422552] Out of memory: Killed process 23714 (obexd)
total-vm:45836kB, anon-rss:732kB, file-rss:0kB, shmem-rss:0kB, UID:1000
pgtables:76kB oom_score_adj:200
[ 719.423354] Out of memory: Killed process 23555 (dconf-service)
total-vm:157600kB, anon-rss:544kB, file-rss:0kB, shmem-rss:0kB, UID:1000
pgtables:64kB oom_score_adj:200
[ 719.424525] Out of memory: Killed process 23397 ((sd-pam))
total-vm:169688kB, anon-rss:3724kB, file-rss:0kB, shmem-rss:0kB,
UID:1000 pgtables:92kB oom_score_adj:100
[ 719.425356] Out of memory: Killed process 23396 (systemd)
total-vm:18600kB, anon-rss:1912kB, file-rss:0kB, shmem-rss:0kB, UID:1000
pgtables:76kB oom_score_adj:100
[ 719.426070] Out of memory: Killed process 55808 (metatester64.ex)
Re: Out of memory killer misconfigured?
On Wed, Apr 20, 2022 at 04:23:38PM +0100, piorunz wrote: Sorry but this happened to me a few times, each tome with the same Wine program. it just loves to eat all available memory when its doing heavy computations. When I am not careful and I start too many threads, is eats all memory. My system gets killed every single time. I am not extrapolating anything. Misbehaving app should get killed, but instead, Linux kills itself. Ok, not one experience, but one scenario. And there are plausible reasons why this might be happening as outlined in my other mail to the thread (OOM adjustments to make the killer skip over the process). Occam's razor suggests something about your particular setup versus the OOM killer simply being as bad as you think it is. FWIW, I invoke the OOM killer a lot recently (due to some scientific experiments) and the sensible processes were killed in my case, every time, leaving my desktop functional. More data about your setup is needed to fathom out what's going on. -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Out of memory killer misconfigured?
On 20/04/2022 11:28, Jonathan Dowland wrote: On Thu, Mar 31, 2022 at 05:53:20PM +0100, piorunz wrote: Well, nowadays oom killer is not so picky. It just kills (almost) EVERYTHING and then offending memory hungry process as a last, destroying entire work session. You are extrapolating from your single experience to make a generalisation that might not be true. Sorry but this happened to me a few times, each tome with the same Wine program. it just loves to eat all available memory when its doing heavy computations. When I am not careful and I start too many threads, is eats all memory. My system gets killed every single time. I am not extrapolating anything. Misbehaving app should get killed, but instead, Linux kills itself. -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On Thu, Mar 31, 2022 at 05:53:20PM +0100, piorunz wrote: Well, nowadays oom killer is not so picky. It just kills (almost) EVERYTHING and then offending memory hungry process as a last, destroying entire work session. You are extrapolating from your single experience to make a generalisation that might not be true. -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Out of memory killer misconfigured?
On Tue, Mar 29, 2022 at 10:34:19AM +0100, piorunz wrote:
Instead of killing ONE 7.5 million-worth pagetable process, Linux is
killing everything else! KDE activity manager killed. Then it goes on to
kill EVERYTHING in the system:
Perhaps your wine processes had their oom_adj (etc) values tweaked to
make the algorithm not consider them (/proc//oom_{adj,score,score_adj})?
I can imagine some ill-advised launcher script for windows software on
linux doing this.
--
Please do not CC me for listmail.
👱🏻 Jonathan Dowland
✎j...@debian.org
🔗 https://jmtd.net
Re: Out of memory killer misconfigured?
On Tue, Apr 19, 2022, 11:08 AM wrote: > On Tue, Apr 19, 2022 at 04:44:36PM +0100, Tim Woodall wrote: > > On Mon, 18 Apr 2022, piorunz wrote: > > > > > > > > I look from desktop perspective. OS (Linux) runs my desktop and manage > > > all programs [...] > > > Because not every machine that has the linux kernel installed runs a > > desktop [...] > > As I already said: I think the OOM killer is the wrong tool for this > job. Once that fires, all bets are up. Its job is to give the sys > admin/system a chance to shut down cleanly, not much more. > I had not heard this before but... cgroup awareness of OOM killer Linux Kernel 4.19 (October 2018) introduced cgroup awareness of OOM killer implementation which adds an ability to kill a cgroup as a single unit and so guarantee the integrity of the workload. That's from https://en.m.wikipedia.org/wiki/Cgroups Of course you still need to identify and configure your cgroups effectively. It seems overdue to make the two tools cooperate but as others pointed out, they have different origins. And it's another one of those things handled differently in the data center from the desktop. Some resource manager (ulimits, control groups [1], what have you) > seems more appropriate. It's up to the desktop environment folks > or to the sysadmin to set them up properly, of course. > > As for why the OOM killer is not triggering the was piorunz expects, > no idea. The scores and the knobs to regulate them are well-known... > > @piorunz: you could start the browser with a worse score so it > gets killed earlier if that suits better your use case. > > Cheers > -- > t >
Re: Out of memory killer misconfigured?
On Tue, Apr 19, 2022 at 04:44:36PM +0100, Tim Woodall wrote: > On Mon, 18 Apr 2022, piorunz wrote: > > > > > I look from desktop perspective. OS (Linux) runs my desktop and manage > > all programs [...] > Because not every machine that has the linux kernel installed runs a > desktop [...] As I already said: I think the OOM killer is the wrong tool for this job. Once that fires, all bets are up. Its job is to give the sys admin/system a chance to shut down cleanly, not much more. Some resource manager (ulimits, control groups [1], what have you) seems more appropriate. It's up to the desktop environment folks or to the sysadmin to set them up properly, of course. As for why the OOM killer is not triggering the was piorunz expects, no idea. The scores and the knobs to regulate them are well-known... @piorunz: you could start the browser with a worse score so it gets killed earlier if that suits better your use case. Cheers -- t signature.asc Description: PGP signature
Re: Out of memory killer misconfigured?
On Mon, 18 Apr 2022, piorunz wrote: I look from desktop perspective. OS (Linux) runs my desktop and manage all programs. When one programs eats too much memory, program gets killed. That is default behaviour or any mature operating system, even in Windows 2000 era we had this. I don't understand why Linux, using all its tools available to itself, is not handling this. I highlight tools available to it, not to me - I should not need to know which internal Linux tool kills misbehaving programs and how it works to use my desktop. Because not every machine that has the linux kernel installed runs a desktop - or even if it does, not everybody who kicks off a long running process in the background wants that process to be killed if the system starts getting memory starved. There is no one right answer to this. If the "right way" is obvious to you in your use case then you can patch the kernel or configure the oom killer. https://www.oracle.com/in/technical-resources/articles/it-infrastructure/dev-oom-killer.html I've not read that page in detail but to summarize "if the kernel is going to kill our one critical process then we might as well reboot anyway" and explains how to make the critical process less likely to be killed. You could do the same with your desktop environment. Tim.
Re: Out of memory killer misconfigured?
On 15/04/2022 11:14, to...@tuxteam.de wrote: Thanks for your reply, I appreciate your patience. - At the "low level": you are talking about "process trees". Perhaps that's the problem. Perhaps, towards the OOM score, all those processes count as individual processes. Each 8GB individual process should be killed absolutely first, not entire KDE desktop and Xorg. That's obvious. Note that the OOM killer is just the last straw the system grabs to try to avoid sinking. Perhaps it's the wrong instrument for your case and you need to look into limits, or cgroup memory policies. I look from desktop perspective. OS (Linux) runs my desktop and manage all programs. When one programs eats too much memory, program gets killed. That is default behaviour or any mature operating system, even in Windows 2000 era we had this. I don't understand why Linux, using all its tools available to itself, is not handling this. I highlight tools available to it, not to me - I should not need to know which internal Linux tool kills misbehaving programs and how it works to use my desktop. - At the "higher level": "you see how things are broken" for you. You are still (in my opinion) committing the error of generalising your problem /before/ you have spent enough time investigating it. That might make it more difficult finding a satisfying solution :-) Yes, I look absolutely from higher, desktop level. Current defaults are broken. OOM is not working, and it's killing entire system instead of one offending application. That's a bug. Plus, don't assume random kernel programmers aren't smart. They usually are. And they usually spend quite a bit of effort until a patch of this kind goes in. Of course they are very smart. However, in this case, and in hundreds of similar cases which are being reported as bugs every year, defaults, or other mechanisms, needs to be adjusted so Denial of Service is not happening when application takes all memory. Right now, that is exactly what's happening. System-wide DoS in one second, entire system nuked and halted when one Wine application claims too much memory. Cheers -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On Fri, Apr 15, 2022 at 11:03:07AM +0100, piorunz wrote:
> On 15/04/2022 06:53, to...@tuxteam.de wrote:
> > If you want to learn more about that, the Linux MM ("memory
> > management") people have set up a wiki for that:
> >
> >https://linux-mm.org/OOM_Killer
> >
> > Enjoy:)
> >
> > (and yes, on my box, more memory-hungry processes have a higher
> > /proc//oom_score, so they seem to incur a higher risk of
> > being killed. The browser lies somewhat because it spawns quite
> > a few processes, but as a product of the propaganda industry,
> > lying is its second nature;-)
>
> Yes I understand that from my point of view everything is bad and I am
> not objective.
> However, I see how things are broken and that is indeed obvious.
> I seen before my own eyes how Linux killed KDE session including Xorg
> but left 10x wine's exe processes with 8GB RAM each as a last. Entire
> tree with parent process was using all of the memory that would be 50+
> GB. How this process tree is not having HIGHEST POSSIBLE OOM score and
> is not being killed in first microsecond of OOM situation is beyond my
> understanding.
- At the "low level": you are talking about "process trees".
Perhaps that's the problem. Perhaps, towards the OOM score,
all those processes count as individual processes. Note that
the OOM killer is just the last straw the system grabs to
try to avoid sinking. Perhaps it's the wrong instrument for
your case and you need to look into limits, or cgroup memory
policies.
- At the "higher level": "you see how things are broken" for
you. You are still (in my opinion) committing the error of
generalising your problem /before/ you have spent enough
time investigating it. That might make it more difficult
finding a satisfying solution :-) Plus, don't assume random
kernel programmers aren't smart. They usually are. And they
usually spend quite a bit of effort until a patch of this
kind goes in.
Cheers
--
t
signature.asc
Description: PGP signature
Re: Out of memory killer misconfigured?
On 15/04/2022 06:53, to...@tuxteam.de wrote:
If you want to learn more about that, the Linux MM ("memory
management") people have set up a wiki for that:
https://linux-mm.org/OOM_Killer
Enjoy:)
(and yes, on my box, more memory-hungry processes have a higher
/proc//oom_score, so they seem to incur a higher risk of
being killed. The browser lies somewhat because it spawns quite
a few processes, but as a product of the propaganda industry,
lying is its second nature;-)
Yes I understand that from my point of view everything is bad and I am
not objective.
However, I see how things are broken and that is indeed obvious.
I seen before my own eyes how Linux killed KDE session including Xorg
but left 10x wine's exe processes with 8GB RAM each as a last. Entire
tree with parent process was using all of the memory that would be 50+
GB. How this process tree is not having HIGHEST POSSIBLE OOM score and
is not being killed in first microsecond of OOM situation is beyond my
understanding.
--
With kindest regards, Piotr.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄
Re: Out of memory killer misconfigured?
On Fri, Apr 15, 2022 at 02:09:04AM +0100, piorunz wrote:
> On 01/04/2022 07:08, to...@tuxteam.de wrote:
[...]
> > See man (1) choom and search for oom in man (5) proc.
> Thanks! That's exactly what I needed. Amazing.
You're welcome :)
> Now, after I start the process, I run:
>
> Adjust to maximum setting (kills first):
> choom -p `pidof terminal64.exe` --adjust 1000
>
> Lower priority of a processes:
> renice 19 `pidof terminal64.exe`
>
> That way offending process will never hang or destabilize my system.
>
> I just wish that Linux kernel would give maximum oom score to process
> with most memory, that's so obvious!
To a certain extent, it does (see below).
> But instead, defaults are so bad,
> it kills everything but one offending process.
> Can this be reported as a bug against linux kernel in Debian bug track?
You can, of course, try. Your chances of success would be better
if you tried to understand what kind thoughts have gone into it,
and why it's not working for you. The way you are putting it, it
comes across (to me, at least!) as "what kind of stupid design
is this" (cf. your words: "obvious", "defaults so bad"). I know
you don't intend that, but, as you can imagine, it won't fly if
that's how others perceive it :-)
If you want to learn more about that, the Linux MM ("memory
management") people have set up a wiki for that:
https://linux-mm.org/OOM_Killer
Enjoy :)
(and yes, on my box, more memory-hungry processes have a higher
/proc//oom_score, so they seem to incur a higher risk of
being killed. The browser lies somewhat because it spawns quite
a few processes, but as a product of the propaganda industry,
lying is its second nature ;-)
Cheers
--
t
signature.asc
Description: PGP signature
Re: Out of memory killer misconfigured?
On 01/04/2022 07:08, to...@tuxteam.de wrote: On Tue, Mar 29, 2022 at 07:58:38PM +0100, piorunz wrote: On 29/03/2022 10:56, Sven Hoexter wrote: E.g. we now have PSI as an information source https://lwn.net/Articles/759781/ which can be used with the Facebook oomd or systemd-oomd to have userland control over which process to kill. Thanks, I've read this article. Unfortunately, this is just information tool which can be used by engineers and developers so design their own oomd. I am not a developer. Is there any config file I can edit so just simply ask oomd to kill most memory hugging process instead of entire system? See man (1) choom and search for oom in man (5) proc. Cheers Thanks! That's exactly what I needed. Amazing. Now, after I start the process, I run: Adjust to maximum setting (kills first): choom -p `pidof terminal64.exe` --adjust 1000 Lower priority of a processes: renice 19 `pidof terminal64.exe` That way offending process will never hang or destabilize my system. I just wish that Linux kernel would give maximum oom score to process with most memory, that's so obvious! But instead, defaults are so bad, it kills everything but one offending process. Can this be reported as a bug against linux kernel in Debian bug track? -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On Tue, Mar 29, 2022 at 07:58:38PM +0100, piorunz wrote: > On 29/03/2022 10:56, Sven Hoexter wrote: > > E.g. we now have PSI as an information source > > https://lwn.net/Articles/759781/ > > which can be used with the Facebook oomd or systemd-oomd to > > have userland control over which process to kill. > > Thanks, I've read this article. Unfortunately, this is just information > tool which can be used by engineers and developers so design their own > oomd. I am not a developer. > > Is there any config file I can edit so just simply ask oomd to kill most > memory hugging process instead of entire system? See man (1) choom and search for oom in man (5) proc. Cheers -- t signature.asc Description: PGP signature
Re: Out of memory killer misconfigured?
On 30/03/2022 09:18, Tixy wrote: I may be wrong here, but I seem to remember that something like that used to happen a long time ago, and it had a habit of picking the X server as the first thing to kill, not very friendly for GUI users. Well, nowadays oom killer is not so picky. It just kills (almost) EVERYTHING and then offending memory hungry process as a last, destroying entire work session. And actually I found out X session still running with everything else killed, even more laughs. -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On Tue, 2022-03-29 at 19:39 +0100, piorunz wrote: > On 29/03/2022 10:56, Sven Hoexter wrote: > > > The in kernel oom killing is a constant issue. If you look through the > > lwn.net articles of the past years there is work done to improve the > > situation, but I believe that's not in a default setup yet. > > Yes it's terrible, how this can be broken so badly? Logic is very simple > here, even back in Windows XP times it was already solved by Microsoft? > Just one simple thing, one line of logic: > > 1. If RAM memory usage > 95% AND no swap available AND I/O cache already > dropped THEN kill most memory hungry process > 2. Repeat > > Job done! I may be wrong here, but I seem to remember that something like that used to happen a long time ago, and it had a habit of picking the X server as the first thing to kill, not very friendly for GUI users. -- Tixy
Re: Out of memory killer misconfigured?
On Tue, Mar 29, 2022 at 09:03:14PM +0100, piorunz wrote: > On 29/03/2022 20:16, Greg Wooledge wrote: > > Resource limits are a thing. man setrlimit (you may have to install > > manpages-dev first). > > How do I use it? I've read manual but that would need to written down as > a C++ program or something? ;( Usually things are launched from either a shell, or from systemd. If the thing is launched from bash, then you can use bash's "ulimit" command to set the resource limits before launching the thing. If it's launched by systemd, look at systemd.exec(5) and search for "Limit" to see how to specify resource limits in a unit file. If it's launched from some shell that isn't bash, then you might be able to find a "ulimit" or "limit" command that works in the other shell. Otherwise, you can use wrapper tools like DJB's "softlimit" (from daemontools) to set resource limits and chain-load the desired thing. > I need, nothing else! I don't understand why memory > limit like that is not built in to Debian, opening possibility to DoS > every system by allocating too much memory by unprivileged userland > process. Resource limits are not set by default, because they will cause processes to die if they're set too low. Only the person who needs them will be able to determine which processes need a limit placed on them, and how low the limit should be set.
Re: Out of memory killer misconfigured?
On 29/03/2022 20:16, Greg Wooledge wrote: Resource limits are a thing. man setrlimit (you may have to install manpages-dev first). Are they crude as hell? Absolutely. But they allow you to set up a barrier that says "if you use more than x MB of memory, you die". They work well enough for some cases. That's exactly what I need, nothing else! I don't understand why memory limit like that is not built in to Debian, opening possibility to DoS every system by allocating too much memory by unprivileged userland process. Unless this is due to my configuration and everyone else is not experiencing that? How do I use it? I've read manual but that would need to written down as a C++ program or something? ;( -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On Tue, Mar 29, 2022 at 07:41:53PM +0100, piorunz wrote: > And I wanted to highlight that this is not some trivial problem, at the > moment any rogue non-privileged process running on user account, or > simple coding error can destroy entire Linux session, be is a important > server with many services running or just a normal PC, computer is > totally unusable until manual intervention (hard reset by hand)! Resource limits are a thing. man setrlimit (you may have to install manpages-dev first). Are they crude as hell? Absolutely. But they allow you to set up a barrier that says "if you use more than x MB of memory, you die". They work well enough for some cases.
Re: Out of memory killer misconfigured?
On Tue, Mar 29, 2022 at 1:59 PM piorunz wrote: > On 29/03/2022 10:56, Sven Hoexter wrote: > > E.g. we now have PSI as an information source > > https://lwn.net/Articles/759781/ > > which can be used with the Facebook oomd or systemd-oomd to > > have userland control over which process to kill. > > Thanks, I've read this article. Unfortunately, this is just information > tool which can be used by engineers and developers so design their own > oomd. I am not a developer. > > Is there any config file I can edit so just simply ask oomd to kill most > memory hugging process instead of entire system? > Yes. Looks like oomd came from facebook :-) https://github.com/facebookincubator/oomd/blob/main/docs/configuration.md That's the kind of tool that gets turned-off in backend servers. -- > With kindest regards, Piotr. > > ⢀⣴⠾⠻⢶⣦⠀ > ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system > ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ > ⠈⠳⣄ > >
Re: Out of memory killer misconfigured?
On 29/03/2022 10:56, Sven Hoexter wrote: E.g. we now have PSI as an information source https://lwn.net/Articles/759781/ which can be used with the Facebook oomd or systemd-oomd to have userland control over which process to kill. Thanks, I've read this article. Unfortunately, this is just information tool which can be used by engineers and developers so design their own oomd. I am not a developer. Is there any config file I can edit so just simply ask oomd to kill most memory hugging process instead of entire system? -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On 29/03/2022 10:56, Sven Hoexter wrote: And I wanted to highlight that this is not some trivial problem, at the moment any rogue non-privileged process running on user account, or simple coding error can destroy entire Linux session, be is a important server with many services running or just a normal PC, computer is totally unusable until manual intervention (hard reset by hand)! -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On 29/03/2022 10:56, Sven Hoexter wrote: The in kernel oom killing is a constant issue. If you look through the lwn.net articles of the past years there is work done to improve the situation, but I believe that's not in a default setup yet. Yes it's terrible, how this can be broken so badly? Logic is very simple here, even back in Windows XP times it was already solved by Microsoft? Just one simple thing, one line of logic: 1. If RAM memory usage > 95% AND no swap available AND I/O cache already dropped THEN kill most memory hungry process 2. Repeat Job done! But instead I ended up with 400 KB of logs showing how kernel oomd was busy destroying my working KDE session. E.g. we now have PSI as an information source https://lwn.net/Articles/759781/ which can be used with the Facebook oomd or systemd-oomd to have userland control over which process to kill. If you really want to fine tune your system this should give you a lead what to look for. Sven Thanks Sven, I will have a look. Facebook-oomd, systemd-oomd? And built-in Linux solution? How many are out there? :O Do you reckon I can report this as a bug against Linux kernel in Debian bug track? -- With kindest regards, Piotr. ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ ⠈⠳⣄
Re: Out of memory killer misconfigured?
On Tue, Mar 29, 2022 at 10:34:19AM +0100, piorunz wrote: > Hello, > > I use Debian Testing on AMD64, on a workstation with Ryzen 5800X - 16 > CPU cores and 64GB of ECC DDR4 RAM. > > Today, Windows application I run on Wine for work has decided to eat all > available memory, CPU and HDD I/O. I don't have swapfile, so Linux > kernel must kill something to remain online when all RAM is taken by > rogue application. > That's where problem I noticed comes in - Debian oom-kill has killed > EVERYTHING and actual offending memory hungry application at the end. > Why?! It destroyed working KDE session and I had to hard reset the PC. The in kernel oom killing is a constant issue. If you look through the lwn.net articles of the past years there is work done to improve the situation, but I believe that's not in a default setup yet. E.g. we now have PSI as an information source https://lwn.net/Articles/759781/ which can be used with the Facebook oomd or systemd-oomd to have userland control over which process to kill. If you really want to fine tune your system this should give you a lead what to look for. Sven
