Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

> memory is executable code, but it behaves like
> other process memory such as the heap or the stack

Seems like it is not a problem: with GNOME effect is the same as with other
environments.

пн, 14 сент. 2020 г. в 21:14, Benjamin Berg :

> On Mon, 2020-09-14 at 20:35 +0900, Alexey A. wrote:
> >
> > > may be different for GNOME with JIT'ed code
> > > potentially getting swapped out
> >
> > How to recognize JIT'ed code? Do you mean that we shouldn't lock it?
>
> I don't think you can protect it using your tool, as it lives in
> anonymous mappings. If it is reclaimed it will be written to swap.
>
> Said differently, that memory is executable code, but it behaves like
> other process memory such as the heap or the stack.
>
> Benjamin
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Mon, 2020-09-14 at 20:35 +0900, Alexey A. wrote:
> 
> > may be different for GNOME with JIT'ed code
> > potentially getting swapped out
> 
> How to recognize JIT'ed code? Do you mean that we shouldn't lock it?

I don't think you can protect it using your tool, as it lives in
anonymous mappings. If it is reclaimed it will be written to swap.

Said differently, that memory is executable code, but it behaves like
other process memory such as the heap or the stack.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

> may be different for GNOME with JIT'ed code
> potentially getting swapped out

How to recognize JIT'ed code? Do you mean that we shouldn't lock it?

пн, 14 сент. 2020 г. в 20:17, Benjamin Berg :

> On Sun, 2020-09-13 at 16:47 +, Alexey Avramov wrote:
> > > So with
> > > applications started, you might get higher.
> >
> > I think we should protect only basic GUI. On computers with 16G+ RAM
> > locking 1G memory with apps should not be a problem if it helps to
> > improve responsiveness.
>
> Yup, fully agree. We only need to make sure the user remains in
> control, we don't need to save random apps.
>
> > > Was that with or without swap?
> >
> > It was without swap, but with swap it has the same effect (faster
> > killer coming and system reclaiming) + responsiveness was improved
> > during heavy swapping.
>
> Yeah. I think things may be different for GNOME with JIT'ed code
> potentially getting swapped out.
>
> > > I feel that uresourced is just nicer conceptually.
> > > So uresourced is
> > > going to be ineffective for KDE for the time being.
> >
> > However, the advantage of prelockd is that may be used on old systems
> > with any DE and any file system and an init. On Debian 9 Mate locking
> > 150-200M takes very good effect. On Fedora with LXDE 200M is also
> > good value.
>
> Entirely true. I just personally prefer aiming for the long term here.
>
> Benjamin
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Sun, 2020-09-13 at 16:47 +, Alexey Avramov wrote:
> > So with
> > applications started, you might get higher.
> 
> I think we should protect only basic GUI. On computers with 16G+ RAM
> locking 1G memory with apps should not be a problem if it helps to
> improve responsiveness.

Yup, fully agree. We only need to make sure the user remains in
control, we don't need to save random apps.

> > Was that with or without swap?
> 
> It was without swap, but with swap it has the same effect (faster
> killer coming and system reclaiming) + responsiveness was improved
> during heavy swapping.

Yeah. I think things may be different for GNOME with JIT'ed code
potentially getting swapped out.

> > I feel that uresourced is just nicer conceptually.
> > So uresourced is
> > going to be ineffective for KDE for the time being.
> 
> However, the advantage of prelockd is that may be used on old systems
> with any DE and any file system and an init. On Debian 9 Mate locking
> 150-200M takes very good effect. On Fedora with LXDE 200M is also
> good value.

Entirely true. I just personally prefer aiming for the long term here.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

> If you actually tried to use the memory it says in MemAvailable, you
> may very well already get bad side effects as the kernel needs to
> reclaim memory used for other purposes (file caches, mmap'ed
> executables, heap, …). Depending on the workload, this may already
> cause the system to start thrashing.

memavaild[1] is yet another tool to improve responsiveness during heavy 
swapping. How it works: slices are swapped out when MemAvailable is low by 
reducing memory.max (values change dynamically). Effects: performance increases 
in tasks that requires heavy swapping, especially in io-bound tasks. At the 
same time, tasks are performed at less io and memory pressure. It may be used 
out of the box with any DE.

With combination of prelockd and memavaild GUI remains responsive with high 
memory and io pressure (pressure=85[2] with system and swap on HDD, I taked 
screenshot, GUI was not freezed). This proves that high memory pressure (by 
PSI) does not always correspond to system hang (psi-monitor uses mem 
pressure=10 as a thigger, for example).

[1] https://github.com/hakavlad/memavaild
[2] https://ibb.co/hKGy0bZ
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

> So with
> applications started, you might get higher.

I think we should protect only basic GUI. On computers with 16G+ RAM locking 1G 
memory with apps should not be a problem if it helps to improve responsiveness.

> Was that with or without swap?

It was without swap, but with swap it has the same effect (faster killer coming 
and system reclaiming) + responsiveness was improved during heavy swapping.

> I feel that uresourced is just nicer conceptually.
> So uresourced is
> going to be ineffective for KDE for the time being.

However, the advantage of prelockd is that may be used on old systems with any 
DE and any file system and an init. On Debian 9 Mate locking 150-200M takes 
very good effect. On Fedora with LXDE 200M is also good value.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Sat, 2020-09-12 at 20:53 +, Alexey Avramov wrote:
> > how much memory that amounts to in the usual scenarios
> 
> 700M on F32 without any apps started.

That seems like quite a lot. I mean, we get into a similar region of
memory that EarlyOOM would be protecting with no apps started. So with
applications started, you might get higher.

> Largest file: (207.9M) /usr/lib/locale/locale-archive
> 
> Files list with its sizes: https://pastebin.com/Hpr6D3Sv
> 
> Locking even 250M-300M takes good effect. For example, demo: 
> https://www.youtube.com/watch?v=vykUrP1UvcI - no freezes with
> prelockd and freeze without prelockd

Was that with or without swap? For GNOME some executable code is JITed
javascript and we really need to protect that somehow.

250-300MiB sounds fine to me, though I do wonder how much of that is
really not needed often (e.g. library code only used at startup).

In general, I really prefer the cgroups approach taken with uresourced.
prelockd is a cool experiment and it proves that executable code being
reclaimed and refaulted is a major cause of latency.
But I feel that uresourced is just nicer conceptually. i.e. we first
segment the system into various parts, which is really simple to do
with systemd (everyone is already moving to systemd). And then we
simply ask the kernel for memory and other guarantees for important
parts of the system.

Unfortunately, while KDE is very close to using systemd for login, I
don't think it is ready yet (at least not on F33). So uresourced is
going to be ineffective for KDE for the time being.

On my F32 with a fixed uresourced[1] and increasing memory protection
to 350MiB, my GNOME session only has really minor hangs and only the
first time I do the tail /dev/zero test. This is with swap on an SSD
and EarlyOOM stopped.

Benjamin

[1] Right now there are two small issues that I will fix once beta
freeze is lifted:
 - Memory protection is ineffective due to a systemd bug which prevents
   my workaround for another issue from working …
 - MemoryMin needs to be used for OOM scenarios (rather than MemoryLow)
I'll leave the protected area at 250MiB though as that matches the
initial change request.


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

> how much memory that amounts to in the usual scenarios

700M on F32 without any apps started.

Largest file: (207.9M) /usr/lib/locale/locale-archive

Files list with its sizes: https://pastebin.com/Hpr6D3Sv

Locking even 250M-300M takes good effect. For example, demo: 
https://www.youtube.com/watch?v=vykUrP1UvcI - no freezes with prelockd and 
freeze without prelockd.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Simon Farnsworth]

> On 20 Jul 2020, at 15:40, Stephen John Smoogen  wrote:
> 
> On Mon, 20 Jul 2020 at 05:04, Kevin Kofler  > wrote:
>> 
>> John M. Harris Jr wrote:
>>> Userspace isn't dead when a system is thrashing. Your software is still
>>> running. If it gets killed, you're most likely going to lose your data.
>> 
>> The thing is, there are various levels of thrashing. In some cases, the
>> system is so busy that you have no chance to bring it back to responsiveness
>> for many minutes, up to hours. (Other than hitting the Reset or Power
>> button, of course.) I have had cases where not even sshd would respond. (The
>> fact that login has been blocking on D-Bus since the introduction of
>> systemd-logind does not help either. Login timeouts are something that was
>> just never happening in the past, now they are common under heavy load.)
>> 
>> That said, I do not see how the EarlyOOM heuristic, which allows, depending
>> on the exact settings, something like 80-90% of swap to be used IN ADDITION
>> to 90+% RAM (and will only start doing anything if BOTH RAM and swap are
>> full) can prevent thrashing in any reliable way. My thrashing scenarios have
>> had much less swap than that used. (I have twice as much swap than RAM, so
>> when the EarlyOOM heuristics trigger, my programs are already trying to use
>> almost 3 times as much RAM as is actually available!)
>> 
> 
> I think the problem is that you are using way too much swap for modern
> systems. The reasons for swap being 2x to 4x real memory was a 1980's
> solution when big RAM systems had 64 MB of ram but a server might need
> 128MB for certain tasks. This was 'reasonable' because the processors
> were slow but could still walk through 128 MB of space 'pretty' fast.
> As RAM got larger this 2x became 'cargo-culted' in various
> documentation and was still reasonable while processor speed went up.
> You could still have a system with 512MB of ram and walk through 1024
> to 2048 GB of swap in similar times as the 128 MB.

Also affected by the way some UNIXes handled commit and fork (the things Linux 
heuristic overcommit was designed to avoid needing excess swap for). In the 
best case, you needed as much swap as RAM, so that a full size process (64 MB 
in your 1980s system) could allocate an extra 64 MB to fork; in the worst case, 
things were using that swap and you needed some multiple just for commit 
accounting.

IIRC, there were also some systems where the available space for committing was 
set to the size of swap - hence the need for 2x RAM. One for matching your RAM 
and allowing you to use it, one for the space consumed during a fork.

Linux's use of overcommit means that this isn't an issue for Fedora, though.
-- 
Simon___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Stephen John Smoogen]

On Mon, 20 Jul 2020 at 05:04, Kevin Kofler  wrote:
>
> John M. Harris Jr wrote:
> > Userspace isn't dead when a system is thrashing. Your software is still
> > running. If it gets killed, you're most likely going to lose your data.
>
> The thing is, there are various levels of thrashing. In some cases, the
> system is so busy that you have no chance to bring it back to responsiveness
> for many minutes, up to hours. (Other than hitting the Reset or Power
> button, of course.) I have had cases where not even sshd would respond. (The
> fact that login has been blocking on D-Bus since the introduction of
> systemd-logind does not help either. Login timeouts are something that was
> just never happening in the past, now they are common under heavy load.)
>
> That said, I do not see how the EarlyOOM heuristic, which allows, depending
> on the exact settings, something like 80-90% of swap to be used IN ADDITION
> to 90+% RAM (and will only start doing anything if BOTH RAM and swap are
> full) can prevent thrashing in any reliable way. My thrashing scenarios have
> had much less swap than that used. (I have twice as much swap than RAM, so
> when the EarlyOOM heuristics trigger, my programs are already trying to use
> almost 3 times as much RAM as is actually available!)
>

I think the problem is that you are using way too much swap for modern
systems. The reasons for swap being 2x to 4x real memory was a 1980's
solution when big RAM systems had 64 MB of ram but a server might need
128MB for certain tasks. This was 'reasonable' because the processors
were slow but could still walk through 128 MB of space 'pretty' fast.
As RAM got larger this 2x became 'cargo-culted' in various
documentation and was still reasonable while processor speed went up.
You could still have a system with 512MB of ram and walk through 1024
to 2048 GB of swap in similar times as the 128 MB.

As processor speeds increased while disk speeds did not, there was a
tipping point where going into larger swap would be catastrophic. By
the early 2000's systems with 4x swap would go catatonic in thrashing
by the time you got even 1/2 of it full. Things seemed to have gotten
worse with multicore processors and threaded programs. This means that
larger numbers of tools are all trying to get access to 'live' RAM all
the time and if you go off to disks, you are sunk. These days, I think
that if you are needing more than 0.5 RAM in swap.. you are probably
going to go into thrashing hell pretty regularly... the speed
difference between memory and disks is too large and there are too
many processes all wanting some CPU time all the time.



-- 
Stephen J Smoogen.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Michael Catanzaro]

On Mon, Jul 20, 2020 at 10:55 am, Kevin Kofler  
wrote:
That said, I do not see how the EarlyOOM heuristic, which allows, 
depending
on the exact settings, something like 80-90% of swap to be used IN 
ADDITION
to 90+% RAM (and will only start doing anything if BOTH RAM and swap 
are

full) can prevent thrashing in any reliable way.


Well you're right: on its own, earlyoom is unlikely to save you from 
thrashing if you have swap on disk. This is really designed to work 
well in combination with zram, and best with Benjamin's resource 
protection work as well. So please consider it alongside the other work.


Michael

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Mon, 2020-07-20 at 10:55 +0200, Kevin Kofler wrote:
> That said, I do not see how the EarlyOOM heuristic, which allows, depending 
> on the exact settings, something like 80-90% of swap to be used IN ADDITION 
> to 90+% RAM (and will only start doing anything if BOTH RAM and swap are 
> full) can prevent thrashing in any reliable way. My thrashing scenarios have 
> had much less swap than that used. (I have twice as much swap than RAM, so 
> when the EarlyOOM heuristics trigger, my programs are already trying to use 
> almost 3 times as much RAM as is actually available!)

Yeah, I think that EarlyOOM will mostly help in the scenario where you
end up not having enough file caches to run your processes. I do
believe that this is a common scenario where the machine continues to
thrash for long periods[1].

In most cases where plenty of swap is available, the symptoms should be
a lot less sever. i.e. I would expect it to become really sluggish, but
things *should* recover much more quickly. Though, e.g. a fork-bomb
could easily cause something like that.

Anyway, I think to resolve it, we really need to enable the CPU and IO
cgroup controllers. CPU is easy (you just need to set a few properties
in systemd), but there are some road blocks to get the IO controller
working.

Also, we can probably fix systemd-logind hanging simply by assigning a
MemoryLow= allocation of e.g. 60M to system.slice and also setting
either DisableControllers=memory (create a large pool of memory for all
system services) or DefaultMemoryLow=X (delegate memory further to each
service separately).

Benjamin

[1] To be honest, I am wondering if some process might have actually
filled up all the RAM even in your case. It can be hard to tell later
on what exactly happened.


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Kevin Kofler]

John M. Harris Jr wrote:
> Userspace isn't dead when a system is thrashing. Your software is still
> running. If it gets killed, you're most likely going to lose your data.

The thing is, there are various levels of thrashing. In some cases, the 
system is so busy that you have no chance to bring it back to responsiveness 
for many minutes, up to hours. (Other than hitting the Reset or Power 
button, of course.) I have had cases where not even sshd would respond. (The 
fact that login has been blocking on D-Bus since the introduction of 
systemd-logind does not help either. Login timeouts are something that was 
just never happening in the past, now they are common under heavy load.)

That said, I do not see how the EarlyOOM heuristic, which allows, depending 
on the exact settings, something like 80-90% of swap to be used IN ADDITION 
to 90+% RAM (and will only start doing anything if BOTH RAM and swap are 
full) can prevent thrashing in any reliable way. My thrashing scenarios have 
had much less swap than that used. (I have twice as much swap than RAM, so 
when the EarlyOOM heuristics trigger, my programs are already trying to use 
almost 3 times as much RAM as is actually available!)

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Ariadne Conill]

Hello,

On Sunday, July 19, 2020 2:53:32 AM MDT John M. Harris Jr wrote:
> On Sunday, July 19, 2020 1:47:23 AM MST Alexey A. wrote:
> 
> > >Killing users' programs needlessly is not welcome
> > 
> > 
> > Setting limits for cgroup (MemoryMax, MemorySwapMax) leads to "Killing
> > users' programs needlessly": system-wide available memory may be not
> > exhausted, but OOM killer will be invoked in this cgroup.
> 
> 
> I'm sure we can all agree that only killing off the software that people 
> complain causes these events is better than killing off everything else just
> because the system doesn't have a ton of RAM available.
> 
> 
> > >The goal is to ensure the kernel can keep doing its job, it's
> > >not going to try to figure out what you intend for userspace, as well it
> > >shouldn't.
> > 
> > 
> > The goal is to ensure the kernel can keep doing its job *and userspace*
> > doing its job. I don't need a system where the kernel is alive, but
> > userspace is dead.
> 
> 
> Userspace isn't dead when a system is thrashing. Your software is still 
> running. If it gets killed, you're most likely going to lose your data.

While this is true, most people will powercycle the machine at this point 
anyway out of frustration if nothing else.  I certainly have powercycled 
machines in this state before.

Powercycling a machine is going to potentially lose more data than just 
killing a runaway firefox or chrome process.

Ariadne

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

> Userspace isn't dead

If something looks like dead it is dead. A userspace that does not respond
to user actions is a dead userspace.

вс, 19 июл. 2020 г. в 17:54, John M. Harris Jr :

> On Sunday, July 19, 2020 1:47:23 AM MST Alexey A. wrote:
> > >Killing users' programs needlessly is not welcome
> >
> > Setting limits for cgroup (MemoryMax, MemorySwapMax) leads to "Killing
> > users' programs needlessly": system-wide available memory may be not
> > exhausted, but OOM killer will be invoked in this cgroup.
>
> I'm sure we can all agree that only killing off the software that people
> complain causes these events is better than killing off everything else
> just
> because the system doesn't have a ton of RAM available.
>
> > >The goal is to ensure the kernel can keep doing its job, it's
> > >not going to try to figure out what you intend for userspace, as well it
> > >shouldn't.
> >
> > The goal is to ensure the kernel can keep doing its job *and userspace*
> > doing its job. I don't need a system where the kernel is alive, but
> > userspace is dead.
>
> Userspace isn't dead when a system is thrashing. Your software is still
> running. If it gets killed, you're most likely going to lose your data.
>
> --
> John M. Harris, Jr.
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Sunday, July 19, 2020 1:47:23 AM MST Alexey A. wrote:
> >Killing users' programs needlessly is not welcome
> 
> Setting limits for cgroup (MemoryMax, MemorySwapMax) leads to "Killing
> users' programs needlessly": system-wide available memory may be not
> exhausted, but OOM killer will be invoked in this cgroup.

I'm sure we can all agree that only killing off the software that people 
complain causes these events is better than killing off everything else just 
because the system doesn't have a ton of RAM available.

> >The goal is to ensure the kernel can keep doing its job, it's
> >not going to try to figure out what you intend for userspace, as well it
> >shouldn't.
> 
> The goal is to ensure the kernel can keep doing its job *and userspace*
> doing its job. I don't need a system where the kernel is alive, but
> userspace is dead.

Userspace isn't dead when a system is thrashing. Your software is still 
running. If it gets killed, you're most likely going to lose your data.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

>Killing users' programs needlessly is not welcome

Setting limits for cgroup (MemoryMax, MemorySwapMax) leads to "Killing
users' programs needlessly": system-wide available memory may be not
exhausted, but OOM killer will be invoked in this cgroup.

>The goal is to ensure the kernel can keep doing its job, it's
>not going to try to figure out what you intend for userspace, as well it
>shouldn't.

The goal is to ensure the kernel can keep doing its job *and userspace*
doing its job. I don't need a system where the kernel is alive, but
userspace is dead.

вс, 19 июл. 2020 г. в 12:42, John M. Harris Jr :

> On Saturday, July 18, 2020 6:33:11 PM MST Brandon Nielsen wrote:
> > What about the case of the developer whose code accidentally does
> > something that blows through all available memory, leading to swap
> > thrashing. I've been there. The system is very much unusable, to the
> > point where a user without the knowledge of the magic sysrq key will
> > almost certainly be reaching for the power button.
>
> Well, sysrq is disabled by default in Fedora anyway. I get what you mean,
> however, as I also re-enable it.
>
> > Or when a compile takes more memory than you expect, leading to the
> > same? Again, I've been there. The system is absolutely unusable for any
> > regular user use-case I can think of.
>
> This is another example that came up, and cgroups can help with that as
> well.
>
> > Decompression "bomb" (malicious or otherwise) on a memory taxed system?
> > Again, definitely unusable.
> >
> > Web browsers are definitely not the only way thrashing can be
> > encountered. While I agree resource limits via cgroups or other method
> > are needed, an EarlyOOM emergency brake is definitely welcome as well.
> > The kernel OOM killer is certainly not adequate for an interactive user
> > session in my experience.
>
> Killing users' programs needlessly is not welcome, at least not by me, and
> I'd
> certainly hope others wouldn't stand for it either. The correct solution
> is to
> prevent the few programs that this is an issue with from eating all of our
> RAM, not killing everything but those. The kernel OOM killer does its job,
> and
> it does it well. The goal is to ensure the kernel can keep doing its job,
> it's
> not going to try to figure out what you intend for userspace, as well it
> shouldn't.
>
> --
> John M. Harris, Jr.
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Saturday, July 18, 2020 6:33:11 PM MST Brandon Nielsen wrote:
> What about the case of the developer whose code accidentally does 
> something that blows through all available memory, leading to swap 
> thrashing. I've been there. The system is very much unusable, to the 
> point where a user without the knowledge of the magic sysrq key will 
> almost certainly be reaching for the power button.

Well, sysrq is disabled by default in Fedora anyway. I get what you mean, 
however, as I also re-enable it.

> Or when a compile takes more memory than you expect, leading to the 
> same? Again, I've been there. The system is absolutely unusable for any 
> regular user use-case I can think of.

This is another example that came up, and cgroups can help with that as well. 

> Decompression "bomb" (malicious or otherwise) on a memory taxed system? 
> Again, definitely unusable.
> 
> Web browsers are definitely not the only way thrashing can be 
> encountered. While I agree resource limits via cgroups or other method 
> are needed, an EarlyOOM emergency brake is definitely welcome as well. 
> The kernel OOM killer is certainly not adequate for an interactive user 
> session in my experience.

Killing users' programs needlessly is not welcome, at least not by me, and I'd 
certainly hope others wouldn't stand for it either. The correct solution is to 
prevent the few programs that this is an issue with from eating all of our 
RAM, not killing everything but those. The kernel OOM killer does its job, and 
it does it well. The goal is to ensure the kernel can keep doing its job, it's 
not going to try to figure out what you intend for userspace, as well it 
shouldn't.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Brandon Nielsen]

On 7/18/20 5:09 PM, John M. Harris Jr wrote:

On Saturday, July 18, 2020 3:25:35 AM MST Benjamin Berg wrote:





But thrashing scenarios are exactly that, *technically* running but
*practically* dead.


Thrashing doesn't mean the system is unusable, or anything of the sort. It's
only an issue if you're thrashing for too long.


I think it only makes sense to continue a discussion if you acknowledge
the existence and really understand the scenario described here:
   https://lkml.org/lkml/2019/8/4/15


I've linked to that very thread several times in this thread. The bug here is
in the web browsers that cause that behavior. The solution is to either fix
the web browsers or limit the amount of memory they can run away with.



What about the case of the developer whose code accidentally does 
something that blows through all available memory, leading to swap 
thrashing. I've been there. The system is very much unusable, to the 
point where a user without the knowledge of the magic sysrq key will 
almost certainly be reaching for the power button.


Or when a compile takes more memory than you expect, leading to the 
same? Again, I've been there. The system is absolutely unusable for any 
regular user use-case I can think of.


Decompression "bomb" (malicious or otherwise) on a memory taxed system? 
Again, definitely unusable.


Web browsers are definitely not the only way thrashing can be 
encountered. While I agree resource limits via cgroups or other method 
are needed, an EarlyOOM emergency brake is definitely welcome as well. 
The kernel OOM killer is certainly not adequate for an interactive user 
session in my experience.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Saturday, July 18, 2020 3:25:35 AM MST Benjamin Berg wrote:
> On Fri, 2020-07-17 at 19:44 -0700, John M. Harris Jr wrote:
> > On Friday, July 17, 2020 10:06:53 AM MST Benjamin Berg wrote:
> > > What we achieve by killing a process is that we give the kernel more
> > > flexibility in how it manages the available memory. It really doesn't
> > > matter what you kill, all that matters is that some memory is free'ed
> > > up again.
> > 
> > It does matter what you kill, because you're wiping out users' data and
> > stopping software the user intended to have run. The kernel is already
> > more
> > than capable of freeing memory for itself, that's not what this Change is
> > for either. This Change is to abuse the OOM killer to run in non-OOM
> > scenarios using a userspace daemon.
> 
> No, an OOM scenario from a kernel point of view means, that it has no
> other choice than to kill a process.

Users' processes should NEVER be killed, unless there is no other choice to 
keep the system running.

> You *really* need to accept that the kernel OOM killer is insufficient
> in many scenarios. It is only the last line of defence, that is solely
> concerned about whether the system is *technically* capable of running.

It's more than sufficient for the goal you listed, which is to keep the kernel 
working.

> But thrashing scenarios are exactly that, *technically* running but
> *practically* dead.

Thrashing doesn't mean the system is unusable, or anything of the sort. It's 
only an issue if you're thrashing for too long.

> I think it only makes sense to continue a discussion if you acknowledge
> the existence and really understand the scenario described here:
>   https://lkml.org/lkml/2019/8/4/15

I've linked to that very thread several times in this thread. The bug here is 
in the web browsers that cause that behavior. The solution is to either fix 
the web browsers or limit the amount of memory they can run away with.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Fri, 2020-07-17 at 19:44 -0700, John M. Harris Jr wrote:
> On Friday, July 17, 2020 10:06:53 AM MST Benjamin Berg wrote:
> > What we achieve by killing a process is that we give the kernel more
> > flexibility in how it manages the available memory. It really doesn't
> > matter what you kill, all that matters is that some memory is free'ed
> > up again.
> 
> It does matter what you kill, because you're wiping out users' data and 
> stopping software the user intended to have run. The kernel is already more 
> than capable of freeing memory for itself, that's not what this Change is for 
> either. This Change is to abuse the OOM killer to run in non-OOM scenarios 
> using a userspace daemon.

No, an OOM scenario from a kernel point of view means, that it has no
other choice than to kill a process.

You *really* need to accept that the kernel OOM killer is insufficient
in many scenarios. It is only the last line of defence, that is solely
concerned about whether the system is *technically* capable of running.

But thrashing scenarios are exactly that, *technically* running but
*practically* dead.

I think it only makes sense to continue a discussion if you acknowledge
the existence and really understand the scenario described here:
  https://lkml.org/lkml/2019/8/4/15

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Sat, 2020-07-18 at 05:07 +0900, Alexey A. wrote:
> > And this will turn bad quickly, as it will eventually
> > include the executables/libraries that must be loaded as they are doing
> > work!
> 
> > So, we don't want to get the kernel into the situation where it must
> > remove executables/libraries from main memory. If that happens, you can
> > end up hitting the disk for *every* function call.
> 
> BTW, with prelockd[1] executables/libraries will not be removed from
> memory. This daemon can mlock executables/libraries in memory.
> 
> Effects:
> - OOM killer comes faster.
> - Fast system reclaiming after OOM.

Interesting, I had not seen that.

I wonder how much memory that amounts to in the usual scenarios. Could
be interesting to compare this to the point where EarlyOOM or similar
would kick in.

That said, I am sure that it is really effective at preventing many
really bad thrashing scenarios.

Benjamin

> This daemon was written recently, published today.
> 
> 1. https://github.com/hakavlad/prelockd
> 
> вт, 14 июл. 2020 г. в 17:19, Benjamin Berg :
> > On Mon, 2020-07-13 at 19:25 -0600, Chris Murphy wrote:
> > > On Mon, Jul 13, 2020 at 12:00 PM Benjamin Berg  wrote:
> > > > If MemAvailable drops below 250MiB (roughly 6GiB * 4%), then this means
> > > > that we have less than 500MiB left for file caches. If we can't swap
> > > > (remember, swap is already pretty full too), then a big chunk of these
> > > > caches need to be dropped to make the memory available to applications.
> > > 
> > > I regularly see impressively low MemAvailable before earlyoom does
> > > SIGTERM, it's almost always swap available (amount of total that's not
> > > used) that's the defining factor. Well below 100MiB. This doesn't tend
> > > to last very long. Once memory is under this kind of pressure, so is
> > > swap, and as swap on zram is so fast, it gets to threshold fast as
> > > well.
> > 
> > Yep, EarlyOOM doesn't apply the limit if there is swap available. That
> > makes a lot of sense. When swap page is available, the kernel has the
> > choice which memory to reclaim (anonymous pages, i.e. swap, or file
> > backed pages, i.e. drop caches). So MemAvailable may drop much lower
> > temporarily and depending on the workload.
> > 
> > You could say that when swap is available you assume the kernel has the
> > ability to keep the system running reasonably well. Once swap runs out,
> > the kernel stops having a choice. It can only make room by reclaiming
> > important caches. And this will turn bad quickly, as it will eventually
> > include the executables/libraries that must be loaded as they are doing
> > work!
> > 
> > So, we don't want to get the kernel into the situation where it must
> > remove executables/libraries from main memory. If that happens, you can
> > end up hitting the disk for *every* function call.
> > 
> > Benjamin
> > ___
> > devel mailing list -- devel@lists.fedoraproject.org
> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: 
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> 
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Friday, July 17, 2020 10:06:53 AM MST Benjamin Berg wrote:
> On Fri, 2020-07-17 at 09:12 -0700, John M. Harris Jr wrote:
> > On Tuesday, July 14, 2020 1:18:08 AM MST Benjamin Berg wrote:
> > > So, we don't want to get the kernel into the situation where it must
> > > remove executables/libraries from main memory. If that happens, you can
> > > end up hitting the disk for *every* function call.
> > 
> > If this is true, then we shouldn't enable EarlyOOM, as it will kill
> > processes from in main memory.
> 
> I am not sure what you are saying, and I fear there is some sort of
> misunderstanding here or at least a different way of thinking about the
> problem.

As described below, this Change is enabling a daemon that purposefully kills 
users processes. That's not a good thing.

> What we achieve by killing a process is that we give the kernel more
> flexibility in how it manages the available memory. It really doesn't
> matter what you kill, all that matters is that some memory is free'ed
> up again.

It does matter what you kill, because you're wiping out users' data and 
stopping software the user intended to have run. The kernel is already more 
than capable of freeing memory for itself, that's not what this Change is for 
either. This Change is to abuse the OOM killer to run in non-OOM scenarios 
using a userspace daemon.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

> And this will turn bad quickly, as it will eventually
> include the executables/libraries that must be loaded as they are doing
> work!

> So, we don't want to get the kernel into the situation where it must
> remove executables/libraries from main memory. If that happens, you can
> end up hitting the disk for *every* function call.

BTW, with prelockd[1] executables/libraries will not be removed from
memory. This daemon can mlock executables/libraries in memory.

Effects:
- OOM killer comes faster.
- Fast system reclaiming after OOM.

This daemon was written recently, published today.

1. https://github.com/hakavlad/prelockd

вт, 14 июл. 2020 г. в 17:19, Benjamin Berg :

> On Mon, 2020-07-13 at 19:25 -0600, Chris Murphy wrote:
> > On Mon, Jul 13, 2020 at 12:00 PM Benjamin Berg  wrote:
> > > If MemAvailable drops below 250MiB (roughly 6GiB * 4%), then this means
> > > that we have less than 500MiB left for file caches. If we can't swap
> > > (remember, swap is already pretty full too), then a big chunk of these
> > > caches need to be dropped to make the memory available to applications.
> >
> > I regularly see impressively low MemAvailable before earlyoom does
> > SIGTERM, it's almost always swap available (amount of total that's not
> > used) that's the defining factor. Well below 100MiB. This doesn't tend
> > to last very long. Once memory is under this kind of pressure, so is
> > swap, and as swap on zram is so fast, it gets to threshold fast as
> > well.
>
> Yep, EarlyOOM doesn't apply the limit if there is swap available. That
> makes a lot of sense. When swap page is available, the kernel has the
> choice which memory to reclaim (anonymous pages, i.e. swap, or file
> backed pages, i.e. drop caches). So MemAvailable may drop much lower
> temporarily and depending on the workload.
>
> You could say that when swap is available you assume the kernel has the
> ability to keep the system running reasonably well. Once swap runs out,
> the kernel stops having a choice. It can only make room by reclaiming
> important caches. And this will turn bad quickly, as it will eventually
> include the executables/libraries that must be loaded as they are doing
> work!
>
> So, we don't want to get the kernel into the situation where it must
> remove executables/libraries from main memory. If that happens, you can
> end up hitting the disk for *every* function call.
>
> Benjamin
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Fri, 2020-07-17 at 09:12 -0700, John M. Harris Jr wrote:
> On Tuesday, July 14, 2020 1:18:08 AM MST Benjamin Berg wrote:
> > So, we don't want to get the kernel into the situation where it must
> > remove executables/libraries from main memory. If that happens, you can
> > end up hitting the disk for *every* function call.
> 
> If this is true, then we shouldn't enable EarlyOOM, as it will kill processes 
> from in main memory.

I am not sure what you are saying, and I fear there is some sort of
misunderstanding here or at least a different way of thinking about the
problem.

What we achieve by killing a process is that we give the kernel more
flexibility in how it manages the available memory. It really doesn't
matter what you kill, all that matters is that some memory is free'ed
up again.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Tuesday, July 14, 2020 1:18:08 AM MST Benjamin Berg wrote:
> So, we don't want to get the kernel into the situation where it must
> remove executables/libraries from main memory. If that happens, you can
> end up hitting the disk for *every* function call.

If this is true, then we shouldn't enable EarlyOOM, as it will kill processes 
from in main memory.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Mon, 2020-07-13 at 19:25 -0600, Chris Murphy wrote:
> On Mon, Jul 13, 2020 at 12:00 PM Benjamin Berg  wrote:
> > If MemAvailable drops below 250MiB (roughly 6GiB * 4%), then this means
> > that we have less than 500MiB left for file caches. If we can't swap
> > (remember, swap is already pretty full too), then a big chunk of these
> > caches need to be dropped to make the memory available to applications.
> 
> I regularly see impressively low MemAvailable before earlyoom does
> SIGTERM, it's almost always swap available (amount of total that's not
> used) that's the defining factor. Well below 100MiB. This doesn't tend
> to last very long. Once memory is under this kind of pressure, so is
> swap, and as swap on zram is so fast, it gets to threshold fast as
> well.

Yep, EarlyOOM doesn't apply the limit if there is swap available. That
makes a lot of sense. When swap page is available, the kernel has the
choice which memory to reclaim (anonymous pages, i.e. swap, or file
backed pages, i.e. drop caches). So MemAvailable may drop much lower
temporarily and depending on the workload.

You could say that when swap is available you assume the kernel has the
ability to keep the system running reasonably well. Once swap runs out,
the kernel stops having a choice. It can only make room by reclaiming
important caches. And this will turn bad quickly, as it will eventually
include the executables/libraries that must be loaded as they are doing
work!

So, we don't want to get the kernel into the situation where it must
remove executables/libraries from main memory. If that happens, you can
end up hitting the disk for *every* function call.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Mon, Jul 13, 2020 at 12:00 PM Benjamin Berg  wrote:
>
> If MemAvailable drops below 250MiB (roughly 6GiB * 4%), then this means
> that we have less than 500MiB left for file caches. If we can't swap
> (remember, swap is already pretty full too), then a big chunk of these
> caches need to be dropped to make the memory available to applications.

I regularly see impressively low MemAvailable before earlyoom does
SIGTERM, it's almost always swap available (amount of total that's not
used) that's the defining factor. Well below 100MiB. This doesn't tend
to last very long. Once memory is under this kind of pressure, so is
swap, and as swap on zram is so fast, it gets to threshold fast as
well.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Mon, Jul 13, 2020 at 11:56 AM John M. Harris Jr  wrote:
>
> On Monday, July 13, 2020 10:48:03 AM MST Samuel Sieb wrote:
> > On 7/13/20 8:21 AM, John M. Harris Jr wrote:
> >
> > > On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:
> > >
> > >> But, I also think that the people proposing this have done quite a lot
> > >> of testing to find reasonable values for various scenarios. If they
> > >> have done their job correctly, then EarlyOOM will *not* prevent you
> > >> from fully utilizing your system memory in most scenarios.
> > >
> > >
> > > By the very nature of the configuration, that's not the case. For example,
> > > on my system, for example, it will start sending SIGTERM where I have
> > > over 600 MiB free, and will begin to simply kill software when I have
> > > over a quarter of a gigabyte left.
> >
> >
> > You keep making this claim as if you actually know.  Have you tested it?
> >   Have you even heard of someone else testing it that ran into that?
>
> Yes, I have tested it. With Swap on ZRAM, it's precisely as I described. With
> no swap it's like that, and with swap of equal size, it's at 300 MiB free and
> only an eight of a gigabyte left.
>
> That's what this software does. Its entire purpose is to kill software based
> on memory criteria.

It's based on both memory *and* swap criteria.

Could you provide a bug report of this behavior and include a complete
journal? earlyoom records the exact parameters, memory and swap free
remaining at the time of either SIGTERM or SIGKILL. And in effect the
defining parameter in most cases is not memory but swap, where memory
free is quite substantially low.

Thanks,

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Tue, 2020-07-14 at 00:55 +0900, Alexey A. wrote:
> > MemAvailable is a bad heuristic, it does *not* represent "free"
> > memory.
> 
> MemAvailable is a good  heuristic, it does represent available
> memory.

Maybe. But the point is that I expect a system to be in trouble long
before the value of MemAvailable approaches zero.

In the end, it will depend on the workload. MemAvailable basically
assumes that half of the file cache (plus some other reclaimable
memory, see si_mem_available in linux/mm/page_alloc.c) can be safely
used for other purposes.

A good chunk of what MemAvailable shows will be
 `MemFree + (Active(file) + Inactive(file) + SReclaimable) / 2`
(there is more added, but that just helps the argument)

If MemAvailable drops below 250MiB (roughly 6GiB * 4%), then this means
that we have less than 500MiB left for file caches. If we can't swap
(remember, swap is already pretty full too), then a big chunk of these
caches need to be dropped to make the memory available to applications.

But this is the problem. You need a good amount of file caches for a
system to function properly. The kernel can *run* with that little
memory, but your system will likely start to thrash eventually. Which
is exactly what we are trying to prevent here.


So, I think calling this "available" or "free" memory is really
misleading. Because, what we you are essentially doing with EarlyOOM is
ensuring that there is enough space available for (file) caches[1].

Basically, I think you could turn it around and translate the 4% to
mean "reserve ~8% of system memory to be used for essential caches".

Benjamin

[1] And yeah, you'll be able to construct workloads that only require a
few MiB of file backed data.


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Monday, July 13, 2020 10:48:03 AM MST Samuel Sieb wrote:
> On 7/13/20 8:21 AM, John M. Harris Jr wrote:
> 
> > On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:
> > 
> >> But, I also think that the people proposing this have done quite a lot
> >> of testing to find reasonable values for various scenarios. If they
> >> have done their job correctly, then EarlyOOM will *not* prevent you
> >> from fully utilizing your system memory in most scenarios.
> > 
> > 
> > By the very nature of the configuration, that's not the case. For example,
> > on my system, for example, it will start sending SIGTERM where I have
> > over 600 MiB free, and will begin to simply kill software when I have
> > over a quarter of a gigabyte left.
> 
> 
> You keep making this claim as if you actually know.  Have you tested it? 
>   Have you even heard of someone else testing it that ran into that?

Yes, I have tested it. With Swap on ZRAM, it's precisely as I described. With 
no swap it's like that, and with swap of equal size, it's at 300 MiB free and 
only an eight of a gigabyte left.

That's what this software does. Its entire purpose is to kill software based 
on memory criteria.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Samuel Sieb]

On 7/13/20 8:21 AM, John M. Harris Jr wrote:

On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:

But, I also think that the people proposing this have done quite a lot
of testing to find reasonable values for various scenarios. If they
have done their job correctly, then EarlyOOM will *not* prevent you
from fully utilizing your system memory in most scenarios.


By the very nature of the configuration, that's not the case. For example, on
my system, for example, it will start sending SIGTERM where I have over 600
MiB free, and will begin to simply kill software when I have over a quarter of
a gigabyte left.


You keep making this claim as if you actually know.  Have you tested it? 
 Have you even heard of someone else testing it that ran into that?

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

> MemAvailable is a bad heuristic, it does *not* represent "free"
> memory.

MemAvailable is a good  heuristic, it does represent available memory.

вт, 14 июл. 2020 г. в 00:49, Benjamin Berg :

> On Mon, 2020-07-13 at 08:21 -0700, John M. Harris Jr wrote:
> > On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:
> > > But, I also think that the people proposing this have done quite a lot
> > > of testing to find reasonable values for various scenarios. If they
> > > have done their job correctly, then EarlyOOM will *not* prevent you
> > > from fully utilizing your system memory in most scenarios.
> >
> > By the very nature of the configuration, that's not the case. For
> example, on
> > my system, for example, it will start sending SIGTERM where I have over
> 600
> > MiB free, and will begin to simply kill software when I have over a
> quarter of
> > a gigabyte left.
>
> You keep saying these numbers, yet I don't see how your claim adds up.
>
>  1. 600MiB assumes 10% of main memory.
> However, Fedora uses 4% (capped at 400MiB)
> (see /etc/default/earlyoom, which contains "-m 4 -M 409600")
>
>  2. So EarlyOOM uses this calculation on Fedora:
>   `MemAvailable < min(MemTotal * 4%, 400MiB)`
> (values come from /proc/meminfo)
>
>  3. MemAvailable is a bad heuristic, it does *not* represent "free"
> memory.
>
> Benjamin
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Mon, 2020-07-13 at 08:21 -0700, John M. Harris Jr wrote:
> On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:
> > But, I also think that the people proposing this have done quite a lot
> > of testing to find reasonable values for various scenarios. If they
> > have done their job correctly, then EarlyOOM will *not* prevent you
> > from fully utilizing your system memory in most scenarios.
> 
> By the very nature of the configuration, that's not the case. For example, on 
> my system, for example, it will start sending SIGTERM where I have over 600 
> MiB free, and will begin to simply kill software when I have over a quarter 
> of 
> a gigabyte left.

You keep saying these numbers, yet I don't see how your claim adds up.

 1. 600MiB assumes 10% of main memory.
However, Fedora uses 4% (capped at 400MiB)
(see /etc/default/earlyoom, which contains "-m 4 -M 409600")

 2. So EarlyOOM uses this calculation on Fedora:
  `MemAvailable < min(MemTotal * 4%, 400MiB)`
(values come from /proc/meminfo)

 3. MemAvailable is a bad heuristic, it does *not* represent "free"
memory.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Mon, Jul 13, 2020 at 9:22 AM John M. Harris Jr  wrote:
>
> On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:
> > But, I also think that the people proposing this have done quite a lot
> > of testing to find reasonable values for various scenarios. If they
> > have done their job correctly, then EarlyOOM will *not* prevent you
> > from fully utilizing your system memory in most scenarios.
>
> By the very nature of the configuration, that's not the case. For example, on
> my system, for example, it will start sending SIGTERM where I have over 600
> MiB free, and will begin to simply kill software when I have over a quarter of
> a gigabyte left.

The low water mark for memory *and* swap must be reached to trigger,
not memory alone as you keep suggesting.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Monday, July 13, 2020 1:58:30 AM MST Benjamin Berg wrote:
> But, I also think that the people proposing this have done quite a lot
> of testing to find reasonable values for various scenarios. If they
> have done their job correctly, then EarlyOOM will *not* prevent you
> from fully utilizing your system memory in most scenarios.

By the very nature of the configuration, that's not the case. For example, on 
my system, for example, it will start sending SIGTERM where I have over 600 
MiB free, and will begin to simply kill software when I have over a quarter of 
a gigabyte left.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Sun, 2020-07-12 at 14:46 -0700, John M. Harris Jr wrote:
> That sounds like an excellent idea, but I'm not convinced that killing users' 
> processes, while there's still tons of memory free available, is actually an 
> improvement.

I think you should rather think of "MemAvailable" as a measure of how
flexible the kernel believes it is in its effort to do memory
management.


As I said elsewhere already. MemAvailable (i.e. the kernel measure that
is read from /proc), does *not* reliably tell you how much memory you
can use. There often is no such thing as free memory, it'll be in use
for some purpose.

And, even worse, the kernel cannot even know. Unless you are already at
the point of swapping/thrashing. This is because the kernel does not
know know which memory pages the applications are actively using. These
measure really are just a guess, rather than a reliably and hard
number.

If you actually tried to use the memory it says in MemAvailable, you
may very well already get bad side effects as the kernel needs to
reclaim memory used for other purposes (file caches, mmap'ed
executables, heap, …). Depending on the workload, this may already
cause the system to start thrashing.


I am not trying to claim that EarlyOOM is perfect. These are all just
rough heuristics and how well they work will heavily depend on the
workload. I am sure you can find scenarios where EarlyOOM will be too
aggressive.
But, I also think that the people proposing this have done quite a lot
of testing to find reasonable values for various scenarios. If they
have done their job correctly, then EarlyOOM will *not* prevent you
from fully utilizing your system memory in most scenarios.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[David Schwörer]

On 7/10/20 11:12 PM, John M. Harris Jr wrote:
> On Thursday, July 9, 2020 2:19:27 PM MST Przemek Klosowski via devel wrote:
>> On 7/9/20 8:44 AM, Kevin Kofler wrote:
>>
>>> Przemek Klosowski via devel wrote:
>>>
* disk access is literally O(1) slower than RAM access
>>>
>>> This notation is meaningless. By the definition of the O notation,
>>> O(1)=O(1)=O(k) for any constant k.
>>
>>
>> Yes, you are right of course, but I just hope that everyone understood 
>> that was just a shortcut for saying that the speed ratio is somewhere 
>> between 1e3 and 1e5
> 
> That's not accurate, as demonstrated earlier. At least, it's not accurate for 
> my hardware. I haven't tested on anything more recent.
> 

You keep repeating this claim.

Please show how you measured, and what the results are, or stop making
these claims.

Cheers,
David
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Saturday, July 11, 2020 2:36:04 PM MST Neal Gompa wrote:
> There's a difference between half-baked and a roadmap of incremental
> improvement. This Change is an example of the latter. If anything,
> your fanciful and speculative conjecture has done only harm for your
> credibility. You continue to attack and lambast every change proposed
> for the past few cycles,

On the contrary, I've only been against Changes that I've considered to be 
harmful to Fedora. There have been a surprisingly high number of those this 
release cycle, but that doesn't mean that that I'm against all Changes.

> and you provide no useful alternatives.

The alternative to this would be.. not enabling EarlyOOM, which will kill our 
users' processes needlessly.

> Additionally, you continue to insult our intelligence by assuming we
> haven't done our homework and attack the Workstation working group for
> doing a lot of legwork to try to solve very real problems that users
> see.

I'm sorry if it came off that way, but that wasn't the intention. 

> With respect to EarlyOOM, I *wanted* to introduce this to all desktop
> variants at the same time last cycle, but the amount of research and
> effort we spent to figure out what we should even *do* meant that
> there was no time to do any coordination work last cycle to bring this
> everywhere. The evidence from user reports and reviews has been clear:
> Fedora 32 Workstation Edition had a phenomenal user experience
> improvement in terms of desktop responsiveness over its predecessors
> as well as many of its counterparts that released at the same time by
> our fellow Linux distribution communities. Much of the work we're
> doing for Fedora 33 and beyond is a continuation of that effort.

The real question is whether or not EarlyOOM has anything to do with this 
"phenomenal user experience", which seems unlikely. Generally, users' 
processes getting killed isn't a good thing.

> My personal mission is to bring all the improvements we've done to the
> Linux desktop experience in Fedora Workstation to at least Fedora KDE,
> if not all Fedora desktop variants. And where it makes sense, I will
> attempt to bring these improvements to *all* Fedora variants.

That sounds like an excellent idea, but I'm not convinced that killing users' 
processes, while there's still tons of memory free available, is actually an 
improvement.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Thu, 2020-07-09 at 07:51 -0700, John M. Harris Jr wrote:
> What's the KDE SIG's rationale behind supporting this? This actively limits 
> the amount of RAM that end users are able to make use of. The more RAM the 
> end 
> user has, the more RAM is not available for use, because EarlyOOM will kill 
> software long before it's able to be used. For example, on my system, with 6 
> GiB of RAM, this will send SIGTERM while I still have over half of a gigabyte 
> of RAM, and SIGKILL while I still have over a quarter of a gigabyte of RAM. 
> There's absolutely no justification for this, as I see it.

Sounds like you are misunderstanding the meaning of MemAvailable:

See https://www.kernel.org/doc/html/latest/filesystems/proc.html
"""
An estimate of how much memory is available for starting new
applications, without swapping. Calculated from MemFree, SReclaimable,
the size of the file LRU lists, and the low watermarks in each zone.
The estimate takes into account that the system needs some page cache
to function well, and that not all reclaimable slab will be
reclaimable, due to items being in use. The impact of those factors
will vary from system to system.
"""

It is more complicated than that. I believe that MemAvailable will
basically never reach zero. IIRC, it considers half of the memory
currently used for mmap'ed files to be reclaimable. So, for
MemAvailable to become as low as you are suggesting, you have most
likely already reclaimed major parts of the system. MemAvailable does
*not* imply that you can simply allocate that amount of memory and the
system will be fine.

So, you shouldn't be taking this number from the kernel at face value.
The kernel is also just second guessing here and simply does not really
know which memory is reclaimable and which is in active use.

> > 2.  It's clear you don't like this feature, but characterizing it as random
> >  and ruining things is going a bit far (to put it mildly).
> 
> Not at all, see the above example. Most users have more RAM than I do, not 
> less, and even then, it's hurting the UX.

I don't think your above opinion is based on facts.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Sun, 2020-07-12 at 12:46 +0200, Kevin Kofler wrote:
> Neal Gompa wrote:
> > There's a difference between half-baked and a roadmap of incremental
> > improvement. This Change is an example of the latter.
> 
> No, it is actually an example of deliberately implementing a simplistic 
> "solution" that lags behind the state of the art on the grounds that it is 
> "simple", i.e., dumb.

While I really don't like Neal's line of argument here, this isn't true
either.

EarlyOOM is not a perfect solution. If I am honest, I don't really like
the solution either. But, it is a solution that
 1. exists and
 2. has been well tested
so that one can be reasonably confident that it works well in many
scenarios.

> Chris Murphy literally stated "you'll see there are many more knobs in 
> nohang, much more jargon" as the reason not to use a smarter solution, i.e., 
> this is just the usual GNOME attitude of not letting the user configure 
> anything, which is again getting in the way of a useful solution.

You need to be careful with that argument. I do agree that GNOME
sometimes ends up ignoring certain use-cases or feature requests from
users on the basis that they are not really useful to "most" people.
Where "most" is hard to define, because there is no real evidence
available.

However, this is different. As far as I know, EarlyOOM's simplicity has
been validated to work reasonably well. I expect that everyone would be
happy to ship "nohang" or another solution, as long as there is
evidence that you have found a configuration that actually behaves
better.

But, that is the problem. There are a lot of options to play with, and
few good tests to help find the optimal settings. In the end, the added
complexity and configurability will not help you, because you will not
be able to show that it actually behaves considerably better overall.

> (That said, I also have doubts that ANY userspace solution can reliably 
> solve the problems at hand. But what is sure is that EarlyOOM's approach is 
> too simple to be adequate. I already pointed out the ways in which it can 
> end up doing the wrong thing.)

The optimal solution requires BOTH user space and the kernel to
collaborate. And the tool to do this are cgroups.

> So it is not true that EarlyOOM is the best solution that exists out there. 
> It is just the only one that fits the GNOME no-options design.

It is true that EarlyOOM is not the best solution. But that isn't the
goal here, the goal is to solve a problem.

i.e. you have a number of choices:
   1. Start from scratch and try to find an optimal "nohang"
  configuration. I doubt you even have a methodology to figure this
  out, and all your efforts will be worthless in a couple of years.
   2. Just ignore he issue for now, and stick with the data loss scenario
  of a system freezing in certain situations.
   3. Use EarlyOOM, which has already been extensively tested. i.e. you
  trade one data loss scenario (system freezes completely, user needs
  to turn off) with the possibility of another data loss scenario
  (EarlyOOM kills something unintentional). Considering I haven't see
  reports of EarlyOOM killing too often, this seems pretty safe
  (please, prove me wrong on this one).

I would think that 1. is simply out of the question due to resource
constraints. Which only leaves 2. and 3. as valid options. And, I do
consider both valid. It is a trade-off, and I don't think that either
of the two risks (system freezing vs. unintentional OOM kill) can be
estimated to an extend that makes this decision trivial.

Said differently, there is no fully objective way to know which choice
is better. There are possible indicators (e.g. people having tested
this quite a lot, bug reports should have come in by now if EarlyOOM
was problematic in F32). But, if you are looking for definite prove
that the risk of a regression (i.e. unintended kill) outweighs the
benefit of not freezing the system in some scenarios, then you'll have
a hard time.

> [SNIP]

Benjamin


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Kevin Kofler]

Neal Gompa wrote:
> There's a difference between half-baked and a roadmap of incremental
> improvement. This Change is an example of the latter.

No, it is actually an example of deliberately implementing a simplistic 
"solution" that lags behind the state of the art on the grounds that it is 
"simple", i.e., dumb.

Chris Murphy literally stated "you'll see there are many more knobs in 
nohang, much more jargon" as the reason not to use a smarter solution, i.e., 
this is just the usual GNOME attitude of not letting the user configure 
anything, which is again getting in the way of a useful solution.

(That said, I also have doubts that ANY userspace solution can reliably 
solve the problems at hand. But what is sure is that EarlyOOM's approach is 
too simple to be adequate. I already pointed out the ways in which it can 
end up doing the wrong thing.)

So it is not true that EarlyOOM is the best solution that exists out there. 
It is just the only one that fits the GNOME no-options design.

> If anything, your fanciful and speculative conjecture has done only harm
> for your credibility. You continue to attack and lambast every change
> proposed for the past few cycles, and you provide no useful alternatives.

One important point that you have to realize is that a change is not 
necessarily an improvement. A change can either be an improvement, or have 
no effect, or actually make things worse. Throughout the history of Fedora, 
there have been many change proposals that would just have made things 
worse. Some of those were thankfully shot down, but others were 
unfortunately implemented over our objections, with that same fallacious 
argument that you are floating here, i.e., that the opponents are supposedly 
just naysayers opposed to any change out of principle. And I think that the 
more Fedora matures, the lower the probability that a change will actually 
be an improvement will get.

As a result, there has been an accumulation lately of change proposals that 
I personally think are (or would be) detrimental, for various reasons. So I 
have also been opposing them all, and I fully understand why John is against 
so many changes. We both always state the reasons for why we disagree with 
the changes. I can only urge you to read those reasons and take them into 
consideration rather than just filing them off as "naysayer complaints". The 
fact that a person opposes many changes does not necessarily mean there is 
something wrong with that person, you have to also consider the possibility 
that there might be something wrong with all those changes after all.

We need to be careful not to run into the fallacy of allowing change just 
for the sake of change. We always need to evaluate whether the new state 
will actually be better than the old one, and learn to just say "no" 
otherwise. If it ain't broke, don't fix it.

> My personal mission is to bring all the improvements we've done to the
> Linux desktop experience in Fedora Workstation to at least Fedora KDE,
> if not all Fedora desktop variants. And where it makes sense, I will
> attempt to bring these improvements to *all* Fedora variants.

There are a lot of those "improvements [Workstation has] done to the 
[GNU/]Linux desktop experience" that I definitely do not want to see on the 
KDE Spin: increased reliance on Flatpaks instead of packages, easier 
installation of proprietary drivers and other proprietary software, etc. I 
think those run counter to the principles of Fedora and of GNU/Linux as a 
whole. (And Linux is just the kernel of GNU/Linux.)

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Neal Gompa]

On Sat, Jul 11, 2020 at 5:15 PM John M. Harris Jr  wrote:
>
> On Saturday, July 11, 2020 11:43:29 AM MST Chris Murphy wrote:
> > It's known from the outset that earlyoom is temporary. And as the
> > resource control picture more fully develops on the desktop, both
> > GNOME and KDE, that we have an eye on a PSI based approach. It's a
> > long arc with multiple components and incremental changes. The pro and
> > con to earlyoom is the same: it's simple.
>
> How about, instead of a half-baked solution that you're *plan* to get rid of
> soon, you keep the playtesting to GNOME Spin, and don't hurt the KDE Spin in
> the process, as this Change will do?
>

There's a difference between half-baked and a roadmap of incremental
improvement. This Change is an example of the latter. If anything,
your fanciful and speculative conjecture has done only harm for your
credibility. You continue to attack and lambast every change proposed
for the past few cycles, and you provide no useful alternatives.

Additionally, you continue to insult our intelligence by assuming we
haven't done our homework and attack the Workstation working group for
doing a lot of legwork to try to solve very real problems that users
see.

Finally, as a member of the KDE SIG (and many other teams, including
the Workstation and Server working groups), I have been witness to the
literal months of research and deliberations on how to solve problems
that affect the Linux desktop regardless of desktop environment. When
I replaced Rex as the Workstation WG KDE representative due to his
lack of time to maintain the position, I brought my perspective to the
team to help come up with solutions that can be implemented
Fedora-wide. Moreover, as a "plumbing layer" guy, I bring to the table
a unique perspective understanding most (if not all) layers of the
Linux system stack, which has helped us evolve our strategy of
improving the desktop experience.

With respect to EarlyOOM, I *wanted* to introduce this to all desktop
variants at the same time last cycle, but the amount of research and
effort we spent to figure out what we should even *do* meant that
there was no time to do any coordination work last cycle to bring this
everywhere. The evidence from user reports and reviews has been clear:
Fedora 32 Workstation Edition had a phenomenal user experience
improvement in terms of desktop responsiveness over its predecessors
as well as many of its counterparts that released at the same time by
our fellow Linux distribution communities. Much of the work we're
doing for Fedora 33 and beyond is a continuation of that effort.

My personal mission is to bring all the improvements we've done to the
Linux desktop experience in Fedora Workstation to at least Fedora KDE,
if not all Fedora desktop variants. And where it makes sense, I will
attempt to bring these improvements to *all* Fedora variants.





-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Sat, Jul 11, 2020 at 3:15 PM John M. Harris Jr  wrote:
>
> On Saturday, July 11, 2020 11:43:29 AM MST Chris Murphy wrote:
> > It's known from the outset that earlyoom is temporary. And as the
> > resource control picture more fully develops on the desktop, both
> > GNOME and KDE, that we have an eye on a PSI based approach. It's a
> > long arc with multiple components and incremental changes. The pro and
> > con to earlyoom is the same: it's simple.
>
> How about, instead of a half-baked solution that you're *plan* to get rid of
> soon, you keep the playtesting to GNOME Spin, and don't hurt the KDE Spin in
> the process, as this Change will do?

Why wait to fix people's real problems? You don't like it, disable it.
How about stop being such a whiner all the time? It's obnoxious and
old. If you could complain about just one thing per week, it would be
an improvement.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Saturday, July 11, 2020 11:43:29 AM MST Chris Murphy wrote:
> It's known from the outset that earlyoom is temporary. And as the
> resource control picture more fully develops on the desktop, both
> GNOME and KDE, that we have an eye on a PSI based approach. It's a
> long arc with multiple components and incremental changes. The pro and
> con to earlyoom is the same: it's simple.

How about, instead of a half-baked solution that you're *plan* to get rid of 
soon, you keep the playtesting to GNOME Spin, and don't hurt the KDE Spin in 
the process, as this Change will do?

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Sergio Belkin]

El sáb., 11 jul. 2020 a las 16:39, Benjamin Berg ()
escribió:

> On Sat, 2020-07-11 at 14:31 -0300, Sergio Belkin wrote:
> > My only question is if measuring memory pressure is a better indication.
> > If nohang-desktop uses PSI, isn't it a more proper solution?
>
> The basic problem is that PSI can only be reactive. You need to measure
> it over a period of time and then react if matters have turned bad for
> long enough. Generally, I believe that you will be looking at reacting
> after 10-30s, which is already a really bad situation.
>
> Also, to do this properly, you may want to have different rules
> depending on which parts of the system (i.e. systemd unit or cgroup) is
> under memory pressure. This requires a few things:
>
>  1. A daemon to monitor cgroups
> (possibly nohang, but realistically you really want systemd-oomd)
>  2. A desktop that properly places everything into separate cgroups
>
> Both of these items are work-in-progress areas. It is going to happen,
> but we are not quite there yet (KDE is also working it).
>
>
> Back to the original claim. If PSI requires measuring the pressure over
> a period of time, then the reaction will only happen *after* the
> situation has turned bad. In general this is a *good* thing, you don't
> want to go into panic mode just because the system is sluggish for a
> bit. However, it is also *bad*, because the graphical user interface
> really should *not* freeze for 10-30s.
>
> This is where the uresourced proposal ("Reserve resources for active
> users") that I submitted earlier ties in. It approaches the problem
> from the other side, by protecting the core session processes from the
> effects of memory pressure[1]. It does so by guaranteeing a memory
> allocation of 250MiB to those important processes[2].
>
> By doing this, the UI should remain reasonable responsive even if the
> rest of the system is under huge memory pressure and is heavily
> thrashing.
>
>
> So, from my point of view the right thing here is to:
>
>  1. Go with a simple approach for now, because things are changing.
> EarlyOOM probably fits the bill here.
>  2. Also consider using uresourced, it is simple and should improve
> things a bit further.
> This only makes sense if KDE is already starting using systemd.
>  3. Move to a purely PSI based approach once systemd-oomd comes along.
>
> Benjamin
>
> [1] The KDE systemd startup work is fully compatible. Not sure if that
> is merged and usable in Fedora.
> [2] From a memory management point of view the kernel basically behaves
> as if those processes are using 250MiB less. Which means considerably
> less swapping and such for those processes.
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>

Thanks Chris and Benjamin, I love this kind of constructive discussions :)
-- 
--
Sergio Belkin
LPIC-2 Certified - http://www.lpi.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Benjamin Berg]

On Sat, 2020-07-11 at 14:31 -0300, Sergio Belkin wrote:
> My only question is if measuring memory pressure is a better indication.
> If nohang-desktop uses PSI, isn't it a more proper solution?

The basic problem is that PSI can only be reactive. You need to measure
it over a period of time and then react if matters have turned bad for
long enough. Generally, I believe that you will be looking at reacting
after 10-30s, which is already a really bad situation.

Also, to do this properly, you may want to have different rules
depending on which parts of the system (i.e. systemd unit or cgroup) is
under memory pressure. This requires a few things:

 1. A daemon to monitor cgroups
(possibly nohang, but realistically you really want systemd-oomd)
 2. A desktop that properly places everything into separate cgroups

Both of these items are work-in-progress areas. It is going to happen,
but we are not quite there yet (KDE is also working it).


Back to the original claim. If PSI requires measuring the pressure over
a period of time, then the reaction will only happen *after* the
situation has turned bad. In general this is a *good* thing, you don't
want to go into panic mode just because the system is sluggish for a
bit. However, it is also *bad*, because the graphical user interface
really should *not* freeze for 10-30s.

This is where the uresourced proposal ("Reserve resources for active
users") that I submitted earlier ties in. It approaches the problem
from the other side, by protecting the core session processes from the
effects of memory pressure[1]. It does so by guaranteeing a memory
allocation of 250MiB to those important processes[2].

By doing this, the UI should remain reasonable responsive even if the
rest of the system is under huge memory pressure and is heavily
thrashing.


So, from my point of view the right thing here is to:

 1. Go with a simple approach for now, because things are changing.
EarlyOOM probably fits the bill here.
 2. Also consider using uresourced, it is simple and should improve
things a bit further.
This only makes sense if KDE is already starting using systemd.
 3. Move to a purely PSI based approach once systemd-oomd comes along.

Benjamin

[1] The KDE systemd startup work is fully compatible. Not sure if that
is merged and usable in Fedora.
[2] From a memory management point of view the kernel basically behaves
as if those processes are using 250MiB less. Which means considerably
less swapping and such for those processes.


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Sat, Jul 11, 2020 at 11:32 AM Sergio Belkin  wrote:
>>
>> My suggestion is to stop the 'complaining for the sake of complaining'
>> phase of the feature. And move to the "when I do X Y Z, this other app
>> is killed off - how to tweak this?" And then does the tweak represent
>> covering an edge case? Or is it good enough to be the new default?
>>
>
>  Nice, I'm all in favour of a proactive approach that makes the Linux desktop 
> more responsive.
> AFAIK earlyoom uses free physical memory to send TERM/KILL signals. My only 
> question is if measuring memory pressure is a better indication.
> If nohang-desktop uses PSI, isn't it a more proper solution?

In theory yes. In practice, it's harder. But this is an active area of
research and development.

Alexey can better speak about the details of the difference between
earlyoom and nohang. My suggestion is if you're more than casually
interested in this subject, you can "graduate" from earlyoom to nohang
right now. The gotcha is that if you have to tweak for your workload,
you'll see there are many more knobs in nohang, much more jargon. But
if you go through that process, it'll probably be a more proper
solution. And also, Alexey was instrumental in encouraging the use of
earlyoom for the reason that a PSI approach is more complicated.
Should edge cases emerge, it's easy to let folks know what settings to
change in earlyoom, and push updated settings.

It's known from the outset that earlyoom is temporary. And as the
resource control picture more fully develops on the desktop, both
GNOME and KDE, that we have an eye on a PSI based approach. It's a
long arc with multiple components and incremental changes. The pro and
con to earlyoom is the same: it's simple.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Sergio Belkin]

>
> My suggestion is to stop the 'complaining for the sake of complaining'
> phase of the feature. And move to the "when I do X Y Z, this other app
> is killed off - how to tweak this?" And then does the tweak represent
> covering an edge case? Or is it good enough to be the new default?
>
>
 Nice, I'm all in favour of a proactive approach that makes the Linux
desktop more responsive.
AFAIK earlyoom uses free physical memory to send TERM/KILL signals. My only
question is if measuring memory pressure is a better indication.
If nohang-desktop uses PSI, isn't it a more proper solution?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 2:19:27 PM MST Przemek Klosowski via devel wrote:
> On 7/9/20 8:44 AM, Kevin Kofler wrote:
> 
> > Przemek Klosowski via devel wrote:
> > 
> >>* disk access is literally O(1) slower than RAM access
> > 
> > This notation is meaningless. By the definition of the O notation,
> > O(1)=O(1)=O(k) for any constant k.
> 
> 
> Yes, you are right of course, but I just hope that everyone understood 
> that was just a shortcut for saying that the speed ratio is somewhere 
> between 1e3 and 1e5

That's not accurate, as demonstrated earlier. At least, it's not accurate for 
my hardware. I haven't tested on anything more recent.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Friday, July 10, 2020 2:27:02 AM MST Kevin Kofler wrote:
> Matthew Miller wrote:
> 
> > I also don't see any reason to be required to keep unsolicited negative
> > "private" emails secret. There's no agreement that they should be.
> > However, I would request that you ignore them rather than bringing them
> > back to the list. We can unsubscribe and moderate, but we can't do
> > anything about reading and off-list replies, and I'd prefer for them to
> > not get dragged back here. (My suggestion is to just ignore.)
> 
> 
> Well, as I understand it, the main reason he is sending those private 
> replies is that he was banned from the mailing list, or put on moderation or
> something.
> 
> If this mailing list were actually a place where people are allowed to voice
> their technical criticisms without censorship, we would not have this
> situation to begin with.
> 
> Judging from the forwarded mail, I disagree with Harald on this particular 
> issue, but that does not mean I think we should not listen to his points. 

Agreed, it's unfortunate that he was banned from this list. My only criticism 
was being contacted in private for what I felt was more for the list. At the 
time, I wasn't aware that the individual was banned from the list.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Johannes Lips]

> Matthew Miller wrote:
> 
> Well, as I understand it, the main reason he is sending those private 
> replies is that he was banned from the mailing list, or put on moderation or 
> something.
> 
> If this mailing list were actually a place where people are allowed to voice 
> their technical criticisms without censorship, we would not have this 
> situation to begin with.
I think one of the major problems was, which is also evident in the above 
"private" mail, that it never was only technical and most of the time also 
included personal insults. I think since he was banned/moderated the overall 
climate on this mailing list improved considerably.

Johannes
> 
> Judging from the forwarded mail, I disagree with Harald on this particular 
> issue, but that does not mean I think we should not listen to his points.
> 
> Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Kevin Kofler]

Matthew Miller wrote:
> I also don't see any reason to be required to keep unsolicited negative
> "private" emails secret. There's no agreement that they should be.
> However, I would request that you ignore them rather than bringing them
> back to the list. We can unsubscribe and moderate, but we can't do
> anything about reading and off-list replies, and I'd prefer for them to
> not get dragged back here. (My suggestion is to just ignore.)

Well, as I understand it, the main reason he is sending those private 
replies is that he was banned from the mailing list, or put on moderation or 
something.

If this mailing list were actually a place where people are allowed to voice 
their technical criticisms without censorship, we would not have this 
situation to begin with.

Judging from the forwarded mail, I disagree with Harald on this particular 
issue, but that does not mean I think we should not listen to his points.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Przemek Klosowski via devel]

On 7/9/20 8:44 AM, Kevin Kofler wrote:

Przemek Klosowski via devel wrote:

   * disk access is literally O(1) slower than RAM access

This notation is meaningless. By the definition of the O notation,
O(1)=O(1)=O(k) for any constant k.


Yes, you are right of course, but I just hope that everyone understood 
that was just a shortcut for saying that the speed ratio is somewhere 
between 1e3 and 1e5

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Thu, Jul 9, 2020 at 12:53 PM Brandon Nielsen  wrote:
>
> On 7/9/20 12:24 PM, Alexey A. wrote:
> > I agree. I think that 4GB cap is too small and 4 GB may be used quickly.
> >
> 
>
> Since the values being used seem to have been determined somewhat
> heuristically for both EarlyOOM and SwapOnZRAM, I was wondering if there
> was any prediction for how the values might change if one, the other, or
> both get switched on for Server builds?
>
> SwapOnZRAM in particular is of interest to me. I have one system with a
> workload that's bursty on memory use, and when it starts using the swap
> the system becomes unresponsive, even using active SSH sessions is not
> guaranteed until the contention clears. The last time it happened I
> found myself wondering if SwapOnZRAM would make things more pleasant. It
> certainly does on my workstation machines, especially memory limited ones.

Yeah we need to learn a little more about these cases. Spikes are very
well suited by memory base swap (zram) or cache (zswap), because they
can absorb the hit way faster than disk based swap. But those pools
can become exhausted. So if the workload is spikey but not long lived,
that's good for memory based swap. If it's enduring page out, or the
anonymous pages become stale, that's a case where we might need a way
to dump those to disk based swap still.

Eventually I'd like to see either zram's writeback cache option (to
disk, something exposed by sysfs but not by zramctl or
zram-generator), or zswap have more clear benefit in these cases. So
for those who like to experiment, I've got ideas.

Also, we can detect reclaim via cgroups. Reclaim can look like swap
thrashing in terms of IO. But it's with file pages, rather than
anonymous pages. If reclaim pressure is increasing, it means we don't
have enough swap. Or something needs to be killed off. Which is it, is
the question. So what we don't want to do is just start adding more
swap, because then we just end up back in the situation we're trying
to solve.

There is a need to choreograph a strategy between having enough of and
the right kind of swap, a user space oom killer that responds to
pressure, and resource control that ensures the apps we care about get
minimum cpu, memory, and IO.

It is possible to reincorporate disk based swap, possibly dynamically
by creating swapfiles, but use resource control to restrict IO so that
we don't get runaway swap thrashing. So even though swap partitions
are going away by default in F33, does not necessarily mean we're done
with disk based swap.

These things are super esoteric and highly technical. But it's really
been cool being part of this work happening in Fedora. The Workstation
WG has been the central launching point of the motivating factors: the
UX here is terrible, we have to do better than this. And then we get
the kernel cgroup2, mm, zram, zswap folks building really interesting
technologies to have the ability to get information about system state
and control it. The systemd folks doing the work to make it  possible
to bridge that kernel work, making it accessible to the desktop
developers - where this resource control isolation work is heavily
developed in GNOME and KDE. And then it feeds back quite quickly into
Fedora. There's a long list of developers who have done 8 bajillion
percent more work than my emails and change proposals on these
subjects.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 9:22:01 AM MST Alexey A. wrote:
> >You can limit the process with cgroups
> 
> In this case the process will be killed by OOM killer. Note that OOM killer
> exists by default. Limiting cgroup means the OOM killer will be invoked in
> the cgroup. With earlyoom you database server will get SIGTERM and maybe
> will gracefully shutdown.

I'm suggesting the use of cgroups on software like web browsers, that users 
complain run away with RAM, certainly not databases.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Brandon Nielsen]

On 7/9/20 12:24 PM, Alexey A. wrote:

I agree. I think that 4GB cap is too small and 4 GB may be used quickly.




Since the values being used seem to have been determined somewhat 
heuristically for both EarlyOOM and SwapOnZRAM, I was wondering if there 
was any prediction for how the values might change if one, the other, or 
both get switched on for Server builds?


SwapOnZRAM in particular is of interest to me. I have one system with a 
workload that's bursty on memory use, and when it starts using the swap 
the system becomes unresponsive, even using active SSH sessions is not 
guaranteed until the contention clears. The last time it happened I 
found myself wondering if SwapOnZRAM would make things more pleasant. It 
certainly does on my workstation machines, especially memory limited ones.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Matthew Miller]

On Thu, Jul 09, 2020 at 10:40:39AM -0700, John M. Harris Jr wrote:
> I briefly discussed the situation with bcotton, and that's how it'll handle 
> that in the future. I wasn't aware of the particular situation.

Thanks John!

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 10:32:06 AM MST Matthew Miller wrote:
> On Thu, Jul 09, 2020 at 08:46:57AM -0700, John M. Harris Jr wrote:
> 
> > On Thursday, July 9, 2020 8:17:50 AM MST you wrote:
> > 
> > > keep private mail private idiot
> > 
> > 
> > You're responding to my post on a mailing list. I don't see any reason to
> > keep  this private. I'm not the only one on this list that you've
> > attempted to contact directly with this sort of comment either, and I
> > believe it's important that others are aware.
> 
> 
> I also don't see any reason to be required to keep unsolicited negative
> "private" emails secret. There's no agreement that they should be. However,
> I would request that you ignore them rather than bringing them back to the
> list. We can unsubscribe and moderate, but we can't do anything about
> reading and off-list replies, and I'd prefer for them to not get dragged
> back here. (My suggestion is to just ignore.)
> 
> -- 
> Matthew Miller
> 
> Fedora Project Leader

I briefly discussed the situation with bcotton, and that's how it'll handle 
that in the future. I wasn't aware of the particular situation.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Matthew Miller]

On Thu, Jul 09, 2020 at 08:46:57AM -0700, John M. Harris Jr wrote:
> On Thursday, July 9, 2020 8:17:50 AM MST you wrote:
> > keep private mail private idiot
> 
> You're responding to my post on a mailing list. I don't see any reason to 
> keep 
> this private. I'm not the only one on this list that you've attempted to 
> contact directly with this sort of comment either, and I believe it's 
> important that others are aware.

I also don't see any reason to be required to keep unsolicited negative
"private" emails secret. There's no agreement that they should be. However,
I would request that you ignore them rather than bringing them back to the
list. We can unsubscribe and moderate, but we can't do anything about
reading and off-list replies, and I'd prefer for them to not get dragged
back here. (My suggestion is to just ignore.)

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

>75% and 6G
>for the cap might be better.

I agree. I think that 4GB cap is too small and 4 GB may be used quickly.

>I regularly see 3 to 1 and even 4 to 1.

It's true. It's easy to get 4:1 with browser. In fact, the compression
ratio is highly dependent on the workload. I get 1.4:1 with blender, and
maybe 100:1 with compressing zeroes in tmpfs: https://imgur.com/a/XfNRedA

пт, 10 июл. 2020 г. в 01:46, Chris Murphy :

> On Thu, Jul 9, 2020 at 8:52 AM John M. Harris Jr 
> wrote:
>
> > What's the KDE SIG's rationale behind supporting this? This actively
> limits
> > the amount of RAM that end users are able to make use of. The more RAM
> the end
> > user has, the more RAM is not available for use, because EarlyOOM will
> kill
> > software long before it's able to be used. For example, on my system,
> with 6
> > GiB of RAM, this will send SIGTERM while I still have over half of a
> gigabyte
> > of RAM, and SIGKILL while I still have over a quarter of a gigabyte of
> RAM.
>
> 6G RAM means a 3G /dev/zram0 device that will use ~ 1.5G RAM *if* the
> swap device is full and we're only getting 2 to 1 compression. 2 to 1
> is conservative I regularly see 3 to 1 and even 4 to 1. But let's go
> with 2:1. This means your system has the effective benefit of running
> 9G RAM at a cost of 1.5G. i.e a net of 7.5G RAM. Without having to use
> disk based swap.
>
> Is it possible a case can be made for using zswap instead (this is not
> related to zram at all - this is compressed memory cache that acts as
> a writeback cache to an existing disk based swap)? Yes. I've made this
> argument myself, so has Alexey. And as we learn more about those use
> case and workloads, it is possible that it may be a future feature.
> It's even possible it gets rolled into zram-generator so that users
> don't have to fuss with these things. But in the meantime? We are
> learning and innovating. And by all means try to break it. No one
> wants users to have a suboptimal experience out of the box.
>
> My suggestion is to stop the 'complaining for the sake of complaining'
> phase of the feature. And move to the "when I do X Y Z, this other app
> is killed off - how to tweak this?" And then does the tweak represent
> covering an edge case? Or is it good enough to be the new default?
>
> We can certainly change the defaults in the F33 cycle for both
> swaponzram and earlyoom. In fact we can change the defaults with a
> regular update if we have clear data that we should do that. But we've
> entered the real world phase of testing. And the hypotheticals from
> just complaining isn't very useful, even if those complaints later
> turn out to be valid. Data is what will persuade.
>
> There's lots of data from Fedora IoT and other use cases out there
> that 50% RAM for the /dev/zram size is a  pretty good start. It's
> likely it could go to 75% even in the Fedora 33 time frame. So folks
> should really try to do too much with the defaults and see if they can
> get some failures or unexpected behaviors. And then see if 75%
> consistently solves it. Not that some folks might need to bump the cap
> above 4GB to see an increase if they change the fraction of memory
> used for zram.
>
> For sure this only seems like magic. The efficacy of disk based swap
> is 100%. When a page is evicted, 100% of it goes to disk and 0%
> remains in memory. For swaponzram, the efficacy depends on the
> compression ratio. It's definitely not 0% (unless it's random or
> encrypted data) and it's definitely not 100% even if it's all zeros in
> the page. But we're going to get at least 50% efficacy. The overall
> efficiency of memory utilization is still better. But yeah, in tight
> situations it could be a problem. We need to learn more. 75% and 6G
> for the cap might be better. But that also has a risk for upgrades,
> which is that now on a 6G RAM system, /dev/zram is 4.5G which might
> consume 2.3G RAM. So it's going to be a particularly special workload
> that really wants to live fully in 4+ GB of memory. If it has to do a
> bunch of paging in and out through? It's certainly going to be faster
> still than doing that same paging to/from disk based swap.
>
> And for sure it is better than the noswap case, which means 0%
> eviction efficacy. Those anonymous pages can't go anywhere, they're
> pinned in RAM. At least with a zram based swap, they take up 1/2 or
> less the memory.
>
> So it's a bit more like magic than not. It's pretty cool. But yeah we
> should try to break it.
>
>
>
> --
> Chris Murphy
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Thu, Jul 9, 2020 at 8:52 AM John M. Harris Jr  wrote:

> What's the KDE SIG's rationale behind supporting this? This actively limits
> the amount of RAM that end users are able to make use of. The more RAM the end
> user has, the more RAM is not available for use, because EarlyOOM will kill
> software long before it's able to be used. For example, on my system, with 6
> GiB of RAM, this will send SIGTERM while I still have over half of a gigabyte
> of RAM, and SIGKILL while I still have over a quarter of a gigabyte of RAM.

6G RAM means a 3G /dev/zram0 device that will use ~ 1.5G RAM *if* the
swap device is full and we're only getting 2 to 1 compression. 2 to 1
is conservative I regularly see 3 to 1 and even 4 to 1. But let's go
with 2:1. This means your system has the effective benefit of running
9G RAM at a cost of 1.5G. i.e a net of 7.5G RAM. Without having to use
disk based swap.

Is it possible a case can be made for using zswap instead (this is not
related to zram at all - this is compressed memory cache that acts as
a writeback cache to an existing disk based swap)? Yes. I've made this
argument myself, so has Alexey. And as we learn more about those use
case and workloads, it is possible that it may be a future feature.
It's even possible it gets rolled into zram-generator so that users
don't have to fuss with these things. But in the meantime? We are
learning and innovating. And by all means try to break it. No one
wants users to have a suboptimal experience out of the box.

My suggestion is to stop the 'complaining for the sake of complaining'
phase of the feature. And move to the "when I do X Y Z, this other app
is killed off - how to tweak this?" And then does the tweak represent
covering an edge case? Or is it good enough to be the new default?

We can certainly change the defaults in the F33 cycle for both
swaponzram and earlyoom. In fact we can change the defaults with a
regular update if we have clear data that we should do that. But we've
entered the real world phase of testing. And the hypotheticals from
just complaining isn't very useful, even if those complaints later
turn out to be valid. Data is what will persuade.

There's lots of data from Fedora IoT and other use cases out there
that 50% RAM for the /dev/zram size is a  pretty good start. It's
likely it could go to 75% even in the Fedora 33 time frame. So folks
should really try to do too much with the defaults and see if they can
get some failures or unexpected behaviors. And then see if 75%
consistently solves it. Not that some folks might need to bump the cap
above 4GB to see an increase if they change the fraction of memory
used for zram.

For sure this only seems like magic. The efficacy of disk based swap
is 100%. When a page is evicted, 100% of it goes to disk and 0%
remains in memory. For swaponzram, the efficacy depends on the
compression ratio. It's definitely not 0% (unless it's random or
encrypted data) and it's definitely not 100% even if it's all zeros in
the page. But we're going to get at least 50% efficacy. The overall
efficiency of memory utilization is still better. But yeah, in tight
situations it could be a problem. We need to learn more. 75% and 6G
for the cap might be better. But that also has a risk for upgrades,
which is that now on a 6G RAM system, /dev/zram is 4.5G which might
consume 2.3G RAM. So it's going to be a particularly special workload
that really wants to live fully in 4+ GB of memory. If it has to do a
bunch of paging in and out through? It's certainly going to be faster
still than doing that same paging to/from disk based swap.

And for sure it is better than the noswap case, which means 0%
eviction efficacy. Those anonymous pages can't go anywhere, they're
pinned in RAM. At least with a zram based swap, they take up 1/2 or
less the memory.

So it's a bit more like magic than not. It's pretty cool. But yeah we
should try to break it.



-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

>You can limit the process with cgroups

In this case the process will be killed by OOM killer. Note that OOM killer
exists by default. Limiting cgroup means the OOM killer will be invoked in
the cgroup. With earlyoom you database server will get SIGTERM and maybe
will gracefully shutdown.

пт, 10 июл. 2020 г. в 00:10, John M. Harris Jr :

> On Thursday, July 9, 2020 7:57:25 AM MST Reindl Harald (privat) wrote:
> > Am 09.07.20 um 16:53 schrieb John M. Harris Jr:
> > > On Thursday, July 9, 2020 6:24:41 AM MST Sergio Belkin wrote:
> > >> +1 This would be a genuine improvement for end users!
> > >
> > > In what way do you believe this will be an improvement for end users?
> By
> > > killing their software, while it's legitimately using RAM, as expected?
> > > How
> > > exactly is that beneficial? If anything, this is actively working
> against
> > > the best interests of the end user.
> >
> > could you just shutup about topics you don't understand?
>
> I understand exactly what EarlyOOM does, which is precisely why I take
> issue
> with enabling this by default.
>
> > my co-developer had a recursion in his code and after a few seconds the
> > machine was completly unresponsible due debugging the root cause
>
> That doesn't sound like anything to do with recursion, but either a memory
> leak, or unchecked allocation. It may help to limit the amount of memory
> that
> specific program can use, in order to help debug the issue.
>
> > even power-key took *15 minutes* for a clean shutdown while he was
> > tempted to hard power off the machine which is not much fun witch 32 GB
> > RAM and all sort of database services
>
> Yeah, OOM situations are never fun. However, with all sorts of database
> services, you wouldn't want something killing processes all willy nilly.
>
> > guess what was the solution: kill the fucking httpd worker before it
> > kills the system
>
> You can limit the process with cgroups, so that only the software you
> actually
> intend to limit is affected, and the rest works as you'd expect.
>
> --
> John M. Harris, Jr.
> Splentity
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

>By killing their software, while it's legitimately using RAM, as expected?

Memory usage causes system hang is not legitimately using RAM. Expected
behavior is killing memory hog.

чт, 9 июл. 2020 г. в 23:53, John M. Harris Jr :

> On Thursday, July 9, 2020 6:24:41 AM MST Sergio Belkin wrote:
> > +1 This would be a genuine improvement for end users!
>
> In what way do you believe this will be an improvement for end users? By
> killing their software, while it's legitimately using RAM, as expected?
> How
> exactly is that beneficial? If anything, this is actively working against
> the
> best interests of the end user.
>
> --
> John M. Harris, Jr.
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 8:17:50 AM MST you wrote:
> keep private mail private idiot

You're responding to my post on a mailing list. I don't see any reason to keep 
this private. I'm not the only one on this list that you've attempted to 
contact directly with this sort of comment either, and I believe it's 
important that others are aware.

> Am 09.07.20 um 17:09 schrieb John M. Harris Jr:
> > On Thursday, July 9, 2020 7:57:25 AM MST Reindl Harald (privat) wrote:
> >> my co-developer had a recursion in his code and after a few seconds the
> >> machine was completly unresponsible due debugging the root cause
> > 
> > That doesn't sound like anything to do with recursion
> 
> stop telling people what things sound like when they know the truth

It's true that I don't know your code, was just a guess.

> > but either a memory
> > leak, or unchecked allocation. It may help to limit the amount of memory
> > that specific program can use, in order to help debug the issue.
> 
> endless recursion of program code allocates endless memory, it's that
> easy and it does it pretty quick

Endless recursion will normally cause a stack overflow fairly quickly, and it 
doesn't necessarily allocate a lot more memory. I don't know what language 
you're using, but the only way that'd actually be "infinite" would be if it 
were fork()ing or exec*()ing another process. See below for my suggestions on 
protecting that, if you're interested.

> >> even power-key took *15 minutes* for a clean shutdown while he was
> >> tempted to hard power off the machine which is not much fun witch 32 GB
> >> RAM and all sort of database services
> > 
> > Yeah, OOM situations are never fun. However, with all sorts of database
> > services, you wouldn't want something killing processes all willy nilly.
> 
> that's why database services have OOMScoreAdjust=-1000 kid - they are
> not killed willy nilly

That'd be the case only if they're running as systemd units, not if they're 
spawned as a user process. For example, MySQL is spawned by Akonadi as the 
backend of KMail (and other KDE software). This wouldn't have an oom_score_adj 
set.

> >> guess what was the solution: kill the fucking httpd worker before it
> >> kills the system
> > 
> > You can limit the process with cgroups, so that only the software you
> > actually intend to limit is affected, and the rest works as you'd expect
> 
> jesus christ you don't know *before* it happens what drives crazy and
> putting arbitary limits everywhere does more harm

If you're working on something that may "run out of control", you can apply 
cgroups yourself, to that process tree. You can always remove the limits when 
you're done working on it.

The exact argument you've made against arbitrary limits is my argument against 
EarlyOOM. I hope that helps you understand where I'm coming from.

-- 
John M. Harris, Jr.
Splentity

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[David Schwörer]

On 7/8/20 11:15 PM, John M. Harris Jr wrote:
>>   * disk access is literally O(1) slower than RAM access
> This is just false, and you can prove that on your own system using only 
> `dd`. 
> In fact, if your system is newer than my Lenovo ThinkPad X200 Tablet, you'll 
> probably have even faster reads/writes from/to disk.
> 

Can you explain how you do this?
How do you ensure dd is not using some cache?
And how do you measure ram i/o with dd?
As far as I understand dd is not a memory benchmarking tool, not even
parallised to fully saturate the memory throughput.

Just copying to/from ramdisk with dd gave me speeds of below 2 GB/s, but
a streaming test [1] gives me 15 GB/s for copy using a single thread and
18 GB/s if I use both cores.

Using dd to copy some file, calling time sync before and after, and
adding the time after to the estimate by dd gives me an estimate of 30
MB/s for disk speed.

So the ratio is only ~1000 but I am not sure how to do it with just dd ...

But back to the point - 1000 is still way to slow and I love earlyOOM. I
wrote a script myself before I learned about earlyOOM, and I think it is
great to have earlyOOM enabled by default.

[1] https://www.cs.virginia.edu/stream/FTP/Code/Versions/stream_mpi.c
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 7:57:25 AM MST Reindl Harald (privat) wrote:
> Am 09.07.20 um 16:53 schrieb John M. Harris Jr:
> > On Thursday, July 9, 2020 6:24:41 AM MST Sergio Belkin wrote:
> >> +1 This would be a genuine improvement for end users!
> > 
> > In what way do you believe this will be an improvement for end users? By
> > killing their software, while it's legitimately using RAM, as expected?
> > How
> > exactly is that beneficial? If anything, this is actively working against
> > the best interests of the end user.
> 
> could you just shutup about topics you don't understand?

I understand exactly what EarlyOOM does, which is precisely why I take issue 
with enabling this by default.

> my co-developer had a recursion in his code and after a few seconds the
> machine was completly unresponsible due debugging the root cause

That doesn't sound like anything to do with recursion, but either a memory 
leak, or unchecked allocation. It may help to limit the amount of memory that 
specific program can use, in order to help debug the issue.

> even power-key took *15 minutes* for a clean shutdown while he was
> tempted to hard power off the machine which is not much fun witch 32 GB
> RAM and all sort of database services

Yeah, OOM situations are never fun. However, with all sorts of database 
services, you wouldn't want something killing processes all willy nilly.

> guess what was the solution: kill the fucking httpd worker before it
> kills the system

You can limit the process with cgroups, so that only the software you actually 
intend to limit is affected, and the rest works as you'd expect.

-- 
John M. Harris, Jr.
Splentity

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Michael Catanzaro]

On Thu, Jul 9, 2020 at 7:51 am, John M. Harris Jr 
 wrote:

There's absolutely no justification for this, as I see it.


You're willfully ignoring what everybody else is telling you: we don't 
want out-of-control applications to bring down the desktop. GNOME 
doesn't want it, and KDE doesn't want it either. Is it so hard to 
believe...?


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Rex Dieter]

John M. Harris Jr wrote:

> On Thursday, July 9, 2020 5:25:36 AM MST Rex Dieter wrote:
>> John M. Harris Jr wrote:
>> 
>> 
>> > That's not what this discussion results from. This discussion results
>> > from
>> > somebody outside the KDE SIG deciding the KDE Spin needs EarlyOOM
>> > killing our applications at random, and ruining our desktop experience.
>> 
>> 
>> This is full of inaccurate statements
>> 
>> 1.  The KDE SIG (at least some) was made aware of this feature and were
>> ok
>> with it.  It wasn't someone outside deciding anything.
> 
> What's the KDE SIG's rationale behind supporting this? This actively
> limits the amount of RAM that end users are able to make use of. The more
> RAM the end user has, the more RAM is not available for use, because
> EarlyOOM will kill software long before it's able to be used. For example,
> on my system, with 6 GiB of RAM, this will send SIGTERM while I still have
> over half of a gigabyte of RAM, and SIGKILL while I still have over a
> quarter of a gigabyte of RAM. There's absolutely no justification for
> this, as I see it.
> 

You're welcome to your opinion, so far I do not share it.  One example: I've 
hit several times in the past where compiling something killed my box and 
made it unresponsive.  I would have loved it for something to have bailed me 
out of those situations.

I've been testing it on a laptop with only 4gb ram, and it's been working 
well for me so far.  Though I honestly think I've not yet hit any thresholds 
, primarily only doing local package builds and web browsing.  

Also, you might want to double-check your math and the configuration values 
in account here, my default earlyoom config is set with -m 4 value, and your 
comment does not take into account swap (default is threshold of 10% free).

-- Rex


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 6:24:41 AM MST Sergio Belkin wrote:
> +1 This would be a genuine improvement for end users!

In what way do you believe this will be an improvement for end users? By 
killing their software, while it's legitimately using RAM, as expected? How 
exactly is that beneficial? If anything, this is actively working against the 
best interests of the end user.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Thursday, July 9, 2020 5:25:36 AM MST Rex Dieter wrote:
> John M. Harris Jr wrote:
> 
> 
> > That's not what this discussion results from. This discussion results
> > from
> > somebody outside the KDE SIG deciding the KDE Spin needs EarlyOOM killing
> > our applications at random, and ruining our desktop experience.
> 
> 
> This is full of inaccurate statements
> 
> 1.  The KDE SIG (at least some) was made aware of this feature and were ok 
> with it.  It wasn't someone outside deciding anything.

What's the KDE SIG's rationale behind supporting this? This actively limits 
the amount of RAM that end users are able to make use of. The more RAM the end 
user has, the more RAM is not available for use, because EarlyOOM will kill 
software long before it's able to be used. For example, on my system, with 6 
GiB of RAM, this will send SIGTERM while I still have over half of a gigabyte 
of RAM, and SIGKILL while I still have over a quarter of a gigabyte of RAM. 
There's absolutely no justification for this, as I see it.

> 2.  It's clear you don't like this feature, but characterizing it as random
>  and ruining things is going a bit far (to put it mildly).

Not at all, see the above example. Most users have more RAM than I do, not 
less, and even then, it's hurting the UX.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Sergio Belkin]

El mar., 30 jun. 2020 a las 10:26, Ben Cotton ()
escribió:

> https://fedoraproject.org/wiki/Changes/KDEEarlyOOM
>
> == Summary ==
> As [[Changes/EnableEarlyoom|Fedora Workstation did in F32]], install
> earlyoom package, and enable it by default. If both RAM and swap go below
> 10% free, earlyoom issues SIGTERM to the process with the largest
> oom_score. If both RAM and swap go below 5% free, earlyoom issues SIGKILL
> to the process with the largest oom_score. The idea is to recover from out
> of memory situations sooner, rather than the typical complete system hang
> in which the user has no other choice but to force power off.
>
> == Owner ==
> * Name: [[User:bcotton|Ben Cotton]]
> * Email: bcot...@redhat.com
>
> == Detailed Description ==
> Shamelessly copied from Workstation, which did it in the last release:
>
> Certain workloads have heavy memory demands, quickly consume all of RAM,
> and start to heavily page out to swap. (Heavy paging, is often called "swap
> thrashing" for added descriptive effect, probably because it's noticeable
> and annoying). Incidental swap usage is a good thing, it frees up memory
> for active pages used by a process. Heavy swap usage quickly leads to a
> very negative UX, because it's slow, even on modern SSDs. Due to installer
> defaults, the swap partition is made the same size as available memory (at
> install time), which can be huge. This just extends swap thrashing time.
>
> On the one hand, we want this resource hungry job to complete. On the
> other hand, we want our system to be responsive while that other work is
> going on. But once the GUI stutters or even comes to an apparent stand
> still (hang), we're really wishing the kernel oom-killer would kick in and
> free up memory, so we can start over (maybe using memory or thread limiting
> options - which arguably should be more intelligently figured out, and that
> too is a work in progress but beyond the scope of this feature).
>
> However, once in a heavy swap scenario, it's relatively common the system
> gets stuck in it, where GUI interactivity is terrible to non-existent, and
> also the kernel oom-killer doesn't trigger. From a certain point of view,
> this is working as intended. The kernel oom-killer is concerned about
> keeping the kernel running. It's not at all concerned about user space
> responsiveness.
>
> Instead of the system becoming completely unresponsive for tens of
> minutes, hours or days, this feature expects that an offending process
> (determined by oom_score, same as the kernel oom-killer) will be killed off
> within seconds or a few minutes.
>
> == Benefit to Fedora ==
>
> KDE users will be able to take advantage of the benefits Workstation users
> got from enabling earlyOOM in Fedora 32:
>
> * improved user experience by more quickly regaining control over one's
> system, rather than having to force power off in low-memory situations
> where there's aggressive swapping. Once a system becomes unresponsive, it's
> completely reasonable for the user to assume the system is lost, but that
> includes high potential for data loss.
> * reducing forced poweroff as the main work around will increase data
> collection, improving understanding of low memory situations and how to
> handle them better
> * earlyoom first sends SIGTERM to the chosen process, so it has a chance
> of a proper shutdown, unlike the kernel's oom-killer
>
> == Scope ==
> * Proposal owners:
> ** Modify {{code|
> https://pagure.io/fedora-comps/blob/master/f/comps-f33.xml.in}} to
> include earlyoom package for in {{code|kde-desktop}} section.
> ** Add {{code|
> https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/80-kde.preset}}
> to include:
> 
> # enable earlyoom by default on KDE
> enable earlyoom.service
> 
>
> * Other developers: None, unless KDE-based Spins/Labs want to opt out
>
> * Release engineering: N/A
> * Policies and guidelines: N/A
> * Trademark approval: N/A
>
> == Upgrade/compatibility impact ==
> earlyoom.service will be enabled on upgrade. An upgraded system should
> exhibit the same behaviors as a newly-installed system.
>
> == How To Test ==
> * Fedora 31/32 KDE users can test today:
> ** {{code|sudo dnf install earlyoom}}
> ** {{code|sudo systemctl enable --now earlyoom}}
>
> And then attempt to cause an out of memory situation. Examples:
> ** {{code|tail /dev/zero}}
> ** https://lkml.org/lkml/2019/8/4/15
>
> == User Experience ==
> earlyoom sends SIGTERM to processes based on oom_score when both memory
> and swap have less than 10% free and SIGKILL when below 5%.
>
> == Dependencies ==
> None
>
> == Contingency Plan ==
>
> * Contingency mechanism: (What to do?  Who will do it?) Owner reverts
> changes
> * Contingency deadline: Final freeze
> * Blocks release? No
>
> == Documentation ==
> * {{code|man earlyoom}}
> * https://github.com/rfjakob/earlyoom
> * https://www.kernel.org/doc/gorman/html/understand/understand016.html
>
> == Release Notes ==
> The earlyoom service is now enabled by 

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Kevin Kofler]

Przemek Klosowski via devel wrote:
>   * disk access is literally O(1) slower than RAM access

This notation is meaningless. By the definition of the O notation, 
O(1)=O(1)=O(k) for any constant k.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Rex Dieter]

John M. Harris Jr wrote:

> That's not what this discussion results from. This discussion results from
> somebody outside the KDE SIG deciding the KDE Spin needs EarlyOOM killing
> our applications at random, and ruining our desktop experience.

This is full of inaccurate statements

1.  The KDE SIG (at least some) was made aware of this feature and were ok 
with it.  It wasn't someone outside deciding anything.

2.  It's clear you don't like this feature, but characterizing it as random 
and ruining things is going a bit far (to put it mildly).

-- Rex
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Wednesday, July 8, 2020 12:53:05 PM MST Przemek Klosowski via devel wrote:
> On 7/8/20 12:15 PM, John M. Harris Jr wrote:
> > Really, this is starting to sound like it's more of an issue with web
> > browsers, and less of a problem with our current configurations, without
> > EarlyOOM needlessly killing things.
> > [...]
> > Currently, pages that haven't been used in a while are the ones that would
> > get swapped out first, which I'm sure we can all agree is the most sane
> > option. Your GIMP example is accurate, but that'll take a fraction of a
> > second.
> Argumentative, Your Honor! It's not just an issue with web
> browsers---you say that yourself few lines further down, it happens with
> every program that uses big data---GIMP with lots of images, FreeCAD
> with a complex geometry, rmaxima with a combinatiorally exploding
> symbolic expression, even your editor where you read in the entire
> /var/log/httpd/access_log against your better judgement. Literally all
> those examples happened to me fairly recently---the system went
> unresponsive, essentially requiring hard reset, whereas the preferred
> outcome would have been to abort those runaway tasks.

That other software's data would get swapped out doesn't mean "it's not just 
an issue with web browsers". It's not an issue with anything else. It's okay 
to swap out a few pages, and it doesn't hurt GIMP, FreeCAD, etc. If it does 
with browsers, that's a bug.

> >> One way to think about it is that disk is tens of thousands times slower
> >> than RAM. If you need to use it, your system is commensurably slower.
> >> That's why zram is such a good idea. Swap was always a tradeoff: you
> >> saved $'s not spent on RAM, and paid with your time sitting idle waiting
> >> for the computer.
> > 
> > Well, no. It's not "tens of thousands times slower than RAM". If you need
> > to use it, you're swapping in a few pages at a time, not the entire
> > contents of swap. Swap isn't a replacement for RAM. It's an optimization
> > that doesn't waste RAM needlessly.
> 
> I think we both understand what the other person is trying to say, to
> the point where no further explanations are needed. Having said that,
> I'd prefer if you would qualify and augment instead of denying my
> statements. I stand by both of them:
> 
>   * disk access is literally O(1) slower than RAM access

This is just false, and you can prove that on your own system using only `dd`. 
In fact, if your system is newer than my Lenovo ThinkPad X200 Tablet, you'll 
probably have even faster reads/writes from/to disk.

>   * swap is a cheap substitute for RAM, with the right swap/RAM mix
> determined by cost-benefit considerations

It can be used as such, but really shouldn't be. It's more of an optimization 
such that you don't waste RAM than anything else.

> You're right that there's a sweet spot where swap just provides a buffer
> for occasional peak demand---but this entire discussion results from
> complaints about system behavior under heavy swap use, when swap is
> being an inadequate replacement for the needed RAM.

That's not what this discussion results from. This discussion results from 
somebody outside the KDE SIG deciding the KDE Spin needs EarlyOOM killing our 
applications at random, and ruining our desktop experience.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Przemek Klosowski via devel]

On 7/8/20 12:15 PM, John M. Harris Jr wrote:

I'd rather crash and restart where I left off than have
the computer drag me along trying to save my application.

Sorry, what? Why would your data not be on your system? What about "the modern
way of computing" would move your data from your system to something else? I'd
rather not see software crash, and risk data loss, or have my system "drag me
along".


I am talking about every process that has enough safeguards to be 
effectively idempotent, either because it doesn't use local data or 
saves it often enough to have a reliable, resumeable checkpoints. Here's 
a couple of examples:


 * browsing, because it both displays remote data and because it saves
   the state (tabs and whatnot)
 * make -j 30
 * even emacs editing, because emacs saves the buffers when it's killed

If the computer gets in trouble doing those things, I don't want it to 
do heroics trying to recoverit's OK to abort and retry. I think the 
'modern cloud computing' is, for many reasons, having to be like 
that---resilient to failures and idempotent.



Really, this is starting to sound like it's more of an issue with web
browsers, and less of a problem with our current configurations, without
EarlyOOM needlessly killing things.
[...]
Currently, pages that haven't been used in a while are the ones that would get
swapped out first, which I'm sure we can all agree is the most sane option.
Your GIMP example is accurate, but that'll take a fraction of a second.
Argumentative, Your Honor! It's not just an issue with web 
browsers---you say that yourself few lines further down, it happens with 
every program that uses big data---GIMP with lots of images, FreeCAD 
with a complex geometry, rmaxima with a combinatiorally exploding 
symbolic expression, even your editor where you read in the entire 
/var/log/httpd/access_log against your better judgement. Literally all 
those examples happened to me fairly recently---the system went 
unresponsive, essentially requiring hard reset, whereas the preferred 
outcome would have been to abort those runaway tasks.

One way to think about it is that disk is tens of thousands times slower
than RAM. If you need to use it, your system is commensurably slower.
That's why zram is such a good idea. Swap was always a tradeoff: you
saved $'s not spent on RAM, and paid with your time sitting idle waiting
for the computer.

Well, no. It's not "tens of thousands times slower than RAM". If you need to
use it, you're swapping in a few pages at a time, not the entire contents of
swap. Swap isn't a replacement for RAM. It's an optimization that doesn't
waste RAM needlessly.


I think we both understand what the other person is trying to say, to 
the point where no further explanations are needed. Having said that, 
I'd prefer if you would qualify and augment instead of denying my 
statements. I stand by both of them:


 * disk access is literally O(1) slower than RAM access
 * swap is a cheap substitute for RAM, with the right swap/RAM mix
   determined by cost-benefit considerations

You're right that there's a sweet spot where swap just provides a buffer 
for occasional peak demand---but this entire discussion results from 
complaints about system behavior under heavy swap use, when swap is 
being an inadequate replacement for the needed RAM.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Tuesday, July 7, 2020 8:54:40 AM MST Przemek Klosowski via devel wrote:
> On 7/6/20 6:49 PM, John M. Harris Jr wrote:
> > Unless you're actively using all of those tabs (I don't know how you would
> > be, but it's certainly possible), swap sounds like the perfect solution.
> > Unless Firefox keeps JS running in there, and it's updating the DOM,
> > these would likely be able to get swapped out.
> > 
> > Firefox will actually unload tabs that you haven't done anything with in a
> > while under specific circumstances, but I don't know what those are. You
> > may notice, for example, that the page "reloads" without network traffic,
> > when going to a tab you haven't had open in a while. I've seen this on my
> > system recently.
> 
> Take a look at the Task Manager. You will see tabs running even though
> you're not touching that: the pages have elements (ads, animations, etc)
> that run even though the tabs are not visible. True, the browser tries
> to pacify them (turns off sound/video, and whatnot) but they still
> run---and if the JS engine has memory leaks their memory footprint
> increases. You can see the culprits---sort them by "Energy Impact" or
> "Memory" by clicking on the column headers.

I don't really use web browsers on "intensive" pages, so I've never noticed 
that sort of issue, but I alluded to that above, JS running or updating the 
DOM. Most of the pages I use, with the exception of those included in Fedora's 
repos (doh), don't require JavaScript in order to function, for example, LWN.

Really, this is starting to sound like it's more of an issue with web 
browsers, and less of a problem with our current configurations, without 
EarlyOOM needlessly killing things.

> >> More swap doesn't necessarily solve this problem: remember that 1GB/min
> >> is a ballpark HD speed so if you have 10GB swap  that your system is
> >> actually trying to use, you will just sit there for 10 minutes.
> > 
> > I don't really understand how that'd be the case. For that to happen,
> > you'd
> > have to load all of those into memory, have them swap out, then try to
> > swap
> > them all back in at the same time.
> 
> That's my point---you don't have control over it. Swap algorithm decides
> which pages are evicted from RAM or brought back---if the browser starts
> allocating memory, my FreeCAD might get pushed out, and if I click on
> GIMP window after not using it for an hour it tries to bring it back in.

You do have control over it, with the swappiness value, for example. 
Currently, pages that haven't been used in a while are the ones that would get 
swapped out first, which I'm sure we can all agree is the most sane option. 
Your GIMP example is accurate, but that'll take a fraction of a second.

> One way to think about it is that disk is tens of thousands times slower
> than RAM. If you need to use it, your system is commensurably slower.
> That's why zram is such a good idea. Swap was always a tradeoff: you
> saved $'s not spent on RAM, and paid with your time sitting idle waiting
> for the computer.

Well, no. It's not "tens of thousands times slower than RAM". If you need to 
use it, you're swapping in a few pages at a time, not the entire contents of 
swap. Swap isn't a replacement for RAM. It's an optimization that doesn't 
waste RAM needlessly.

> With the modern way of computing, where your data is mostly NOT on your
> system---so you don't lose it if your application shuts down---I am
> beginning to think that application crashes aren't such a big deal as
> they used to be. I'd rather crash and restart where I left off than have
> the computer drag me along trying to save my application.

Sorry, what? Why would your data not be on your system? What about "the modern 
way of computing" would move your data from your system to something else? I'd 
rather not see software crash, and risk data loss, or have my system "drag me 
along".

> Having said that, of course lots of applications ARE local and will lose
> data if crashed, so maybe the cgroup-based approach is the definitive
> solution: hard-limit the memory for cloud apps, to protect the local
> apps from resource exhaustion.

How would you differentiate between "cloud apps" and "local apps"? Are you 
defining "cloud apps" as things that run in web browsers, or use a web browser 
engine, or just anything running from a remote system? If the web browser 
approach, I'd support that handling.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Przemek Klosowski via devel]

On 7/6/20 6:49 PM, John M. Harris Jr wrote:

Unless you're actively using all of those tabs (I don't know how you would be,
but it's certainly possible), swap sounds like the perfect solution. Unless
Firefox keeps JS running in there, and it's updating the DOM, these would
likely be able to get swapped out.

Firefox will actually unload tabs that you haven't done anything with in a
while under specific circumstances, but I don't know what those are. You may
notice, for example, that the page "reloads" without network traffic, when
going to a tab you haven't had open in a while. I've seen this on my system
recently.
Take a look at the Task Manager. You will see tabs running even though 
you're not touching that: the pages have elements (ads, animations, etc) 
that run even though the tabs are not visible. True, the browser tries 
to pacify them (turns off sound/video, and whatnot) but they still 
run---and if the JS engine has memory leaks their memory footprint 
increases. You can see the culprits---sort them by "Energy Impact" or 
"Memory" by clicking on the column headers.



More swap doesn't necessarily solve this problem: remember that 1GB/min
is a ballpark HD speed so if you have 10GB swap  that your system is
actually trying to use, you will just sit there for 10 minutes.

I don't really understand how that'd be the case. For that to happen, you'd
have to load all of those into memory, have them swap out, then try to swap
them all back in at the same time.


That's my point---you don't have control over it. Swap algorithm decides 
which pages are evicted from RAM or brought back---if the browser starts 
allocating memory, my FreeCAD might get pushed out, and if I click on 
GIMP window after not using it for an hour it tries to bring it back in.


One way to think about it is that disk is tens of thousands times slower 
than RAM. If you need to use it, your system is commensurably slower. 
That's why zram is such a good idea. Swap was always a tradeoff: you 
saved $'s not spent on RAM, and paid with your time sitting idle waiting 
for the computer.


With the modern way of computing, where your data is mostly NOT on your 
system---so you don't lose it if your application shuts down---I am 
beginning to think that application crashes aren't such a big deal as 
they used to be. I'd rather crash and restart where I left off than have 
the computer drag me along trying to save my application.


Having said that, of course lots of applications ARE local and will lose 
data if crashed, so maybe the cgroup-based approach is the definitive 
solution: hard-limit the memory for cloud apps, to protect the local 
apps from resource exhaustion.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Monday, July 6, 2020 8:47:24 AM MST Przemek Klosowski via devel wrote:
> On 7/4/20 8:18 PM, John M. Harris Jr wrote:
> 
> > I've never managed to get one of my own Fedora machines to the point of
> > OOMing, and, when I have seen others do it, it's a problem that would
> > have
> > been solved by having more swap space.
> 
> 
> I am a tab hoarder so I used to wedge the browser due to memory leaks 
> (some live ad pages would blow up into gigabytes of RAM). I think recent 
> browser versions are more resistant to that---I haven't locked up in a 
> while, although it may be because I also tend to have a Task Manager 
> open, sorted by Memory size, and I kill pages that keep growing.

Unless you're actively using all of those tabs (I don't know how you would be, 
but it's certainly possible), swap sounds like the perfect solution. Unless 
Firefox keeps JS running in there, and it's updating the DOM, these would 
likely be able to get swapped out.

Firefox will actually unload tabs that you haven't done anything with in a 
while under specific circumstances, but I don't know what those are. You may 
notice, for example, that the page "reloads" without network traffic, when 
going to a tab you haven't had open in a while. I've seen this on my system 
recently.

> More swap doesn't necessarily solve this problem: remember that 1GB/min 
> is a ballpark HD speed so if you have 10GB swap  that your system is 
> actually trying to use, you will just sit there for 10 minutes.

I don't really understand how that'd be the case. For that to happen, you'd 
have to load all of those into memory, have them swap out, then try to swap 
them all back in at the same time.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Przemek Klosowski via devel]

On 7/4/20 8:18 PM, John M. Harris Jr wrote:

I've never managed to get one of my own Fedora machines to the point of
OOMing, and, when I have seen others do it, it's a problem that would have
been solved by having more swap space.


I am a tab hoarder so I used to wedge the browser due to memory leaks 
(some live ad pages would blow up into gigabytes of RAM). I think recent 
browser versions are more resistant to that---I haven't locked up in a 
while, although it may be because I also tend to have a Task Manager 
open, sorted by Memory size, and I kill pages that keep growing.


More swap doesn't necessarily solve this problem: remember that 1GB/min 
is a ballpark HD speed so if you have 10GB swap  that your system is 
actually trying to use, you will just sit there for 10 minutes.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Sun, Jul 5, 2020 at 12:46 AM John M. Harris Jr  wrote:
>
> On Saturday, July 4, 2020 11:27:47 PM MST Alexey Avramov wrote:
> > >Linux handles low memory situations just fine, but it's much better if you
> > >
>  have an appropriately sized swap partition and let the kernel do its job
> >
> > No, by default Linux can hang at low memory condition. Huge swap space will
> > not help you if a leak occurs. With a large swap space, the hang can happen
> > later, but it can still happen. Another point is that the swap space is
> > slow. With fast leaks and slow swap space, freezing is possible throughout
> > the entire swap filling time. A typical problem: "once all the ram is used
> > up the whole system freezes as swap starts getting full, but really the
> > whole system is completely locked-up. I thought my swapfile was too small
> > so I made it match my ram (12 GB) but the system still gets frozen" [1].
>
> Sure, swap is slower than RAM. That's fine. We don't need it to be as fast as
> RAM, that's not what it's for. It's for things that can be swapped out,
> because it's loaded into memory, but not actively used. RAM doesn't "fill"
> until long after unused things have been swapped out.

Eviction of inactive anonymous pages isn't what causes the problem
under discussion. That's fairly efficient and exactly what swap is
well suited for. The problem is when the workload has greater demand
for memory, than there is RAM. That can cause two events to happen:
reclaim (file page faults), and eviction of active anonymous pages.
Both lead to "thrashing" and can completely clobber all responsibility
of a system, for more than a few reasons, but one of those is by
directly causing significant IO pressure.

In these cases, you simply need more RAM right now. The idea of
resource control suggests providing a minimum guarantee of IO for the
processes required to maintain system responsiveness, thereby
restricting IO demand by other processes. The same is applied to
memory and CPU.



>
> > >If you didn't mean for the program to use as much memory as it tried to,
> > >the correct solution would be to use cgroups.
> >
> >
> > 1. This is not configured by default.
> > 2. This can be inconvenient even for advanced users.
> > 3. Quick leaks can happen unexpectedly.
>
> What software in the default image leads to low memory issues? Web browsers?
> If so, there's a simple solution to this. We can put a default cgroup on web
> browsers so they don't take over the OS.

Pretty much. It'll actually work in reverse where a specific stack is
"protected" or given minimum resource guarantees via cgroups - and
that will automagically translate into everything else having
resources clamped down.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>What software in the default image leads to low memory issues? Web browsers? 

For example, browsers, electron-based apps, blender, compilation, VM, openings 
files, opening file manager (once I came across this: when I opened the file 
manager, an uncontrolled leak occurred somewhere in the thumbnail.so, and tand 
the system froze). It can leak anywhere, and it can happen unexpectedly.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Saturday, July 4, 2020 11:27:47 PM MST Alexey Avramov wrote:
> >Linux handles low memory situations just fine, but it's much better if you
> >
 have an appropriately sized swap partition and let the kernel do its job
> 
> No, by default Linux can hang at low memory condition. Huge swap space will
> not help you if a leak occurs. With a large swap space, the hang can happen
> later, but it can still happen. Another point is that the swap space is
> slow. With fast leaks and slow swap space, freezing is possible throughout
> the entire swap filling time. A typical problem: "once all the ram is used
> up the whole system freezes as swap starts getting full, but really the
> whole system is completely locked-up. I thought my swapfile was too small
> so I made it match my ram (12 GB) but the system still gets frozen" [1].

Sure, swap is slower than RAM. That's fine. We don't need it to be as fast as 
RAM, that's not what it's for. It's for things that can be swapped out, 
because it's loaded into memory, but not actively used. RAM doesn't "fill" 
until long after unused things have been swapped out.

> >If you didn't mean for the program to use as much memory as it tried to,
> >the correct solution would be to use cgroups.
> 
> 
> 1. This is not configured by default. 
> 2. This can be inconvenient even for advanced users. 
> 3. Quick leaks can happen unexpectedly.

What software in the default image leads to low memory issues? Web browsers? 
If so, there's a simple solution to this. We can put a default cgroup on web 
browsers so they don't take over the OS.


-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>Linux handles low memory situations just fine, but it's much better if you 
>have an appropriately sized swap partition and let the kernel do its job 

No, by default Linux can hang at low memory condition. Huge swap space will not 
help you if a leak occurs. With a large swap space, the hang can happen later, 
but it can still happen. Another point is that the swap space is slow. With 
fast leaks and slow swap space, freezing is possible throughout the entire swap 
filling time. A typical problem: "once all the ram is used up the whole system 
freezes as swap starts getting full, but really the whole system is completely 
locked-up. I thought my swapfile was too small so I made it match my ram (12 
GB) but the system still gets frozen" [1].

>If you didn't mean for the program to use as much memory as it tried to, the 
>correct solution would be to use cgroups.

1. This is not configured by default. 
2. This can be inconvenient even for advanced users. 
3. Quick leaks can happen unexpectedly.

[1] https://github.com/hakavlad/nohang/issues/85#issue-564348496
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Friday, July 3, 2020 10:11:53 PM MST Alexey Avramov wrote:
> >it  should be disabled so it doesn't kill our software
> 
> 
> What should people who suffer from the fact that the kernel's OOM killer
> does not work, and they are forced to hard reboot (and lose unsaved data)
> the computer when it freezes? This is a serious and very common problem
> that exists for a long time and has not been fixed out of the box.

I've never managed to get one of my own Fedora machines to the point of 
OOMing, and, when I have seen others do it, it's a problem that would have 
been solved by having more swap space.

 
> What do you suggest? Should we do nothing?

That's precisely what I suggest.

> 
> Of course, we can do nothing and wait for the inclusion of the system-oomd
> in the systemd [7]. Just wait.

How is that disabled?
 
> Also look at these discussions:
> - Why are low memory conditions handled so badly? [1]

They're not. The system lets you do precisely what you're trying to do.

> - Memory management "more effective" on Windows than Linux? (in preventing
> total system lockup) [2]

Windows memory management used to mean that you couldn't map more than 
something like 2 GiB per program. Has that been fixed, or is it still 
completely broken? I'd certainly not call artificial limitations like that 
"more effective".

> - Let's talk about the elephant in the room - the
> Linux kernel's inability to gracefully handle low memory pressure [3] [4]

Linux handles low memory situations just fine, but it's much better if you 
have an appropriately sized swap partition and let the kernel do its job 
(don't turn down swappiness)

> [5] - How do I prevent Linux from freezing when out of memory? Today I
> (accidentally) ran some program on my Linux box that quickly used a lot of
> memory. My system froze, became unresponsive and thus I was unable to kill
> the offender. How can I prevent this in the future? Can't it at least keep
> a responsive core or something running? [6] 

If you didn't mean for the program to use as much memory as it tried to, the 
correct solution would be to use cgroups.

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>It is also unclear that it can prevent full OOM (both 
>RAM and swap completely full) in all cases

Kernel's killer is also cannot prevent full OOM in all cases. earlyoom prevent 
full OOM (and situations close to OOM) much more effective - faster (selects 
victim in 5-50ms, monitoring interval is 0.1-1s) and softer (sends SIGTERM 
first).

>fail 
>to kill processes when it would be necessary to prevent swap thrashing 

OK, earlyoom does not prevent thrashing, but kernel also does not prevent it.
Thrashing may be reduced with killers that supports response to PSI: oomd [1], 
nohang [2], psi-monitor [3] (psi-monitor is used by default in Endless OS).

>I strongly believe that this kernel problem can only be solved within the 
>kernel, by a synchronous (hence race-free) hook on all allocations.

It's already solved in the kernel: just set `vm.overcommit_memory=2` and 
`vm.overcommit_ratio=50`:
"When this flag is 2, the kernel uses a "never overcommit" policy that attempts 
to prevent any overcommit of memory." [4]

1. https://github.com/facebookincubator/oomd
2. https://github.com/hakavlad/nohang
3. https://github.com/endlessm/eos-boot-helper/tree/master/psi-monitor
4. https://www.kernel.org/doc/Documentation/sysctl/vm.txt
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>it  should be disabled so it doesn't kill our software

What should people who suffer from the fact that the kernel's OOM killer does 
not work, and they are forced to hard reboot (and lose unsaved data) the 
computer when it freezes? This is a serious and very common problem that exists 
for a long time and has not been fixed out of the box.

What do you suggest? Should we do nothing?

Of course, we can do nothing and wait for the inclusion of the system-oomd in 
the systemd [7]. Just wait.

Also look at these discussions:
- Why are low memory conditions handled so badly? [1]
- Memory management "more effective" on Windows than Linux? (in preventing 
total system lockup) [2]
- Let's talk about the elephant in the room - the Linux kernel's inability to 
gracefully handle low memory pressure [3] [4] [5]
- How do I prevent Linux from freezing when out of memory? Today I 
(accidentally) ran some program on my Linux box that quickly used a lot of 
memory. My system froze, became unresponsive and thus I was unable to kill the 
offender. How can I prevent this in the future? Can't it at least keep a 
responsive core or something running? [6]

1. 
https://www.reddit.com/r/linux/comments/56r4xj/why_are_low_memory_conditions_handled_so_badly/
2. 
https://www.reddit.com/r/linux/comments/aqd9mh/memory_management_more_effective_on_windows_than/
3. https://lkml.org/lkml/2019/8/4/15
4. 
https://www.reddit.com/r/linux/comments/cmg48b/lets_talk_about_the_elephant_in_the_room_the/
5. https://news.ycombinator.com/item?id=20620545
6. 
https://serverfault.com/questions/390623/how-do-i-prevent-linux-from-freezing-when-out-of-memory
7. https://github.com/systemd/systemd/pull/15206


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Wednesday, July 1, 2020 10:44:03 AM MST Alexey Avramov wrote:
> >10% and 5% to 1% and 0%
> 
> 
> Default values is already changed to 4% (but not more than 400 MiB) and 2%
> (but not more than 200 MiB). A nonzero threshold helps maintain disk cache
> and speeds up system recovery after correction.

Sorry, that wasn't meant to be taken seriously, and I should have clarified. 
My actual suggestion is, "If it's going to be forced to be installed, it 
should be disabled so it doesn't kill our software."

-- 
John M. Harris, Jr.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>What about /usr/lib64/qt5/libexec/QtWebEngineProcess processes?

These processes get oom_score_adj=300 out of the box, see 
https://pastebin.com/AFVU9U8X
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Kevin Kofler]

Alexey A. wrote:
> 2. Firefox's children processes "Web Content" gets +300 to its oom_score.
> It means that earlyoom will prefer to kill firefox tabs rather than entire
> browser. Similar behavior is already practiced in chromium and
> electron-based apps by default.

What about /usr/lib64/qt5/libexec/QtWebEngineProcess processes?

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>EarlyOOM being a 
>userspace process that races with the memory-consuming processes and that 
>may end up not getting scheduled due to the very impending OOM condition 
>that it is trying to prevent.

earlyoom consumes 1 MiB VmRSS and all memory is locked by mlockall(). earlyoom 
works pretty fast and selects victim in 5-50 ms even with intense swapping.

>It is also unclear that it can prevent full OOM (both 
>RAM and swap completely full) 

It's clear: earlyoom forks pretty fast and well.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey Avramov]

>10% and 5% to 1% and 0%

Default values is already changed to 4% (but not more than 400 MiB) and 2% (but 
not more than 200 MiB). A nonzero threshold helps maintain disk cache and 
speeds up system recovery after correction.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Alexey A.]

>If both RAM and swap go below 10% free, earlyoom issues SIGTERM to the
process with the largest oom_score. If both RAM and swap go below 5% free,
earlyoom issues SIGKILL

Fedora's earlyoom package is provided with the changed default settings:

```
EARLYOOM_ARGS="-r 0 -m 4 -M 409600 --prefer '^Web Content$' --avoid
'^(dnf|packagekitd|gnome-shell|gnome-session-c|gnome-session-b|lightdm|sddm|sddm-helper|gdm|gdm-wayland-ses|gdm-session-wor|gdm-x-session|Xorg|Xwayland|systemd|systemd-logind|dbus-daemon|dbus-broker|cinnamon|cinnamon-sessio|kwin_x11|kwin_wayland|plasmashell|ksmserver|plasma_session|startplasma-way|xfce4-session|mate-session|marco|lxqt-session|openbox)$'"
```

It means that:

1. SIGTERM threshold for MemAvailable is 4% (but not more than 400 MiB) and
SIGKILL threshold for MemAvailable is 2% (but not more than 200 MiB) by
default. The change was due to the fact that earlyoom tree was criticized
for too aggressive thresholds by default, and this was taken into account.
Please update description in the proposal.

2. Firefox's children processes "Web Content" gets +300 to its oom_score.
It means that earlyoom will prefer to kill firefox tabs rather than entire
browser. Similar behavior is already practiced in chromium and
electron-based apps by default.

3. Processes, the killing of which can lead to the killing of the entire
session (kwin_x11|kwin_wayland|plasmashell|ksmserver|plasma_session etc),
receive reduced priority in choosing a victim. dnf also gets low prio. This
is yet another advantage that you can mention in the proposal.

see https://pagure.io/fedora-workstation/issue/119#comment-638366

вт, 30 июн. 2020 г. в 22:26, Ben Cotton :

> https://fedoraproject.org/wiki/Changes/KDEEarlyOOM
>
> == Summary ==
> As [[Changes/EnableEarlyoom|Fedora Workstation did in F32]], install
> earlyoom package, and enable it by default. If both RAM and swap go below
> 10% free, earlyoom issues SIGTERM to the process with the largest
> oom_score. If both RAM and swap go below 5% free, earlyoom issues SIGKILL
> to the process with the largest oom_score. The idea is to recover from out
> of memory situations sooner, rather than the typical complete system hang
> in which the user has no other choice but to force power off.
>
> == Owner ==
> * Name: [[User:bcotton|Ben Cotton]]
> * Email: bcot...@redhat.com
>
> == Detailed Description ==
> Shamelessly copied from Workstation, which did it in the last release:
>
> Certain workloads have heavy memory demands, quickly consume all of RAM,
> and start to heavily page out to swap. (Heavy paging, is often called "swap
> thrashing" for added descriptive effect, probably because it's noticeable
> and annoying). Incidental swap usage is a good thing, it frees up memory
> for active pages used by a process. Heavy swap usage quickly leads to a
> very negative UX, because it's slow, even on modern SSDs. Due to installer
> defaults, the swap partition is made the same size as available memory (at
> install time), which can be huge. This just extends swap thrashing time.
>
> On the one hand, we want this resource hungry job to complete. On the
> other hand, we want our system to be responsive while that other work is
> going on. But once the GUI stutters or even comes to an apparent stand
> still (hang), we're really wishing the kernel oom-killer would kick in and
> free up memory, so we can start over (maybe using memory or thread limiting
> options - which arguably should be more intelligently figured out, and that
> too is a work in progress but beyond the scope of this feature).
>
> However, once in a heavy swap scenario, it's relatively common the system
> gets stuck in it, where GUI interactivity is terrible to non-existent, and
> also the kernel oom-killer doesn't trigger. From a certain point of view,
> this is working as intended. The kernel oom-killer is concerned about
> keeping the kernel running. It's not at all concerned about user space
> responsiveness.
>
> Instead of the system becoming completely unresponsive for tens of
> minutes, hours or days, this feature expects that an offending process
> (determined by oom_score, same as the kernel oom-killer) will be killed off
> within seconds or a few minutes.
>
> == Benefit to Fedora ==
>
> KDE users will be able to take advantage of the benefits Workstation users
> got from enabling earlyOOM in Fedora 32:
>
> * improved user experience by more quickly regaining control over one's
> system, rather than having to force power off in low-memory situations
> where there's aggressive swapping. Once a system becomes unresponsive, it's
> completely reasonable for the user to assume the system is lost, but that
> includes high potential for data loss.
> * reducing forced poweroff as the main work around will increase data
> collection, improving understanding of low memory situations and how to
> handle them better
> * earlyoom first sends SIGTERM to the chosen process, so it has a chance
> of a proper shutdown, unlike the 

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Ben Cotton]

On Tue, Jun 30, 2020 at 6:30 PM Kevin Kofler  wrote:
>
> I strongly believe that this kernel problem can only be solved within the
> kernel, by a synchronous (hence race-free) hook on all allocations.
>
I don't disagree, but if that's not available, this is the best we
have. And it's tunable (or easy to disable) for people who don't want
it.

> > == Owner ==
> > * Name: [[User:bcotton|Ben Cotton]]
> > * Email: bcot...@redhat.com
>
> Why is this not owned by Rex Dieter and/or some other KDE SIG member?
>
Because I was the one who said "hey, let's do this." :-) It was
discussed on the KDE mailing list prior to submission.

-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Chris Murphy]

On Tue, Jun 30, 2020 at 4:42 PM Neal Gompa  wrote:
>
> On Tue, Jun 30, 2020 at 6:30 PM Kevin Kofler  wrote:
> >
> > I am opposed to this change, for the same reasons I was opposed to EarlyOOM
> > to begin with: this solution kills processes under arbitrary conditions that
> > are neither necessary nor sufficient to prevent a swap thrashing collapse
> > (i.e., it can both kill processes unnecessarily (false positives) and fail
> > to kill processes when it would be necessary to prevent swap thrashing
> > (false negatives)). It is also unclear that it can prevent full OOM (both
> > RAM and swap completely full) in all cases, due to EarlyOOM being a
> > userspace process that races with the memory-consuming processes and that
> > may end up not getting scheduled due to the very impending OOM condition
> > that it is trying to prevent.
> >
> > I strongly believe that this kernel problem can only be solved within the
> > kernel, by a synchronous (hence race-free) hook on all allocations.
> >
>
> I still believe that too, except *nobody cares in the Linux kernel
> community*. This problem has existed for a decade, and nobody in the
> Linux kernel community is interested in fixing it. At this point, I've
> given up. Most of us have given up. If they ever get around to doing
> the right thing, I'll happily punt all this stuff, but they won't, so
> here we are.

To put a fine nuance on this, the best oversimplification I've come up
with that squares with mm kernel developers in this area is: kernel's
built-in oomkiller cares about the kernel's survival. It's not
concerned about user space directly, insofar as it's being reasonably
fair (i.e. approximately equal misery for all processes even though
one in particular does deserve to be killed off the most). Once the
kernel's ability to manage the system itself comes under pressure?
That's when oomkiller will knock that shit off.

What the mm and cgroups2 folks have done is provide some knobs to make
it possible to (a) control processes consumption of resources with a
very granular level and (b) provide the necessary isolation for a user
space oom killer to do things more intelligently. That's oomd today,
and might be systemd-oomd down the road in a form that's simpler and
doesn't require any tuning - it can in effect come just from proper
resource control being applied.

So... like. You're both correct as weird as that might seem at first
and second gances. But yeah, earlyoom is known and intended to be a
bit simplistic and it's not always going to do what you want but
really we're trying to knock off the worst instances of these
problems. Because frankly the resource control picture isn't yet
complete, so not all this cgroup2 stuff is done on the desktop. And
also it's one of the top 3 reasons for btrfs by default because it's
the only file system right now the cgroup2 guys tell us they know does
IO isolation properly. It is not a hard requirement to use btrfs to
get improved resource control, but since memory and io pressures are
tightly coupled, to have a complete picture we need IO isolation. That
doesn't mean every time you're lost without btrfs. But some percent of
the time (whatever it is) the workload will be such that IO isolation
is needed and if we don't have it, the control won't be quite as good.

And yeah, we'll have to test it and try to break it because that's fun!

So I think it's a good idea to use earlyoom, and it's simple enough
for folks to tweak, if they want to tweak, and learn more about oom
stuff. And maybe they decide they want to know even more, and end up
looking at nohang, which is a more sophisticated user space oom
killer. And they can learn a ton more about how this is actually a
hard problem and people are learning all kinds of new things about
solving it. And then if they get really down into the rabbit hole
maybe they want to do some more cgroup2 and resource control work with
their own apps, or even contribute at the desktop or kernel level. Why
not?


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Neal Gompa]

On Tue, Jun 30, 2020 at 6:30 PM Kevin Kofler  wrote:
>
> Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/KDEEarlyOOM
> >
> > == Summary ==
> > As [[Changes/EnableEarlyoom|Fedora Workstation did in F32]], install
> > earlyoom package, and enable it by default. If both RAM and swap go below
> > 10% free, earlyoom issues SIGTERM to the process with the largest
> > oom_score. If both RAM and swap go below 5% free, earlyoom issues SIGKILL
> > to the process with the largest oom_score. The idea is to recover from out
> > of memory situations sooner, rather than the typical complete system hang
> > in which the user has no other choice but to force power off.
>
> I am opposed to this change, for the same reasons I was opposed to EarlyOOM
> to begin with: this solution kills processes under arbitrary conditions that
> are neither necessary nor sufficient to prevent a swap thrashing collapse
> (i.e., it can both kill processes unnecessarily (false positives) and fail
> to kill processes when it would be necessary to prevent swap thrashing
> (false negatives)). It is also unclear that it can prevent full OOM (both
> RAM and swap completely full) in all cases, due to EarlyOOM being a
> userspace process that races with the memory-consuming processes and that
> may end up not getting scheduled due to the very impending OOM condition
> that it is trying to prevent.
>
> I strongly believe that this kernel problem can only be solved within the
> kernel, by a synchronous (hence race-free) hook on all allocations.
>

I still believe that too, except *nobody cares in the Linux kernel
community*. This problem has existed for a decade, and nobody in the
Linux kernel community is interested in fixing it. At this point, I've
given up. Most of us have given up. If they ever get around to doing
the right thing, I'll happily punt all this stuff, but they won't, so
here we are.

You can believe it until you're blue in the face, it doesn't matter if
they don't care. Unless you are going to fix it? :)

> > == Owner ==
> > * Name: [[User:bcotton|Ben Cotton]]
> > * Email: bcot...@redhat.com
>
> Why is this not owned by Rex Dieter and/or some other KDE SIG member?
>

If you want, I'll happily attach my name to it if that's what it takes
to feel "blessed". I'm a KDE SIG member as well.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Kevin Kofler]

Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/KDEEarlyOOM
> 
> == Summary ==
> As [[Changes/EnableEarlyoom|Fedora Workstation did in F32]], install
> earlyoom package, and enable it by default. If both RAM and swap go below
> 10% free, earlyoom issues SIGTERM to the process with the largest
> oom_score. If both RAM and swap go below 5% free, earlyoom issues SIGKILL
> to the process with the largest oom_score. The idea is to recover from out
> of memory situations sooner, rather than the typical complete system hang
> in which the user has no other choice but to force power off.

I am opposed to this change, for the same reasons I was opposed to EarlyOOM 
to begin with: this solution kills processes under arbitrary conditions that 
are neither necessary nor sufficient to prevent a swap thrashing collapse 
(i.e., it can both kill processes unnecessarily (false positives) and fail 
to kill processes when it would be necessary to prevent swap thrashing 
(false negatives)). It is also unclear that it can prevent full OOM (both 
RAM and swap completely full) in all cases, due to EarlyOOM being a 
userspace process that races with the memory-consuming processes and that 
may end up not getting scheduled due to the very impending OOM condition 
that it is trying to prevent.

I strongly believe that this kernel problem can only be solved within the 
kernel, by a synchronous (hence race-free) hook on all allocations.

> == Owner ==
> * Name: [[User:bcotton|Ben Cotton]]
> * Email: bcot...@redhat.com

Why is this not owned by Rex Dieter and/or some other KDE SIG member?

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[John M. Harris Jr]

On Tuesday, June 30, 2020 6:25:02 AM MST Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/KDEEarlyOOM
> 
> == Summary ==
> As [[Changes/EnableEarlyoom|Fedora Workstation did in F32]], install
> earlyoom package, and enable it by default. If both RAM and swap go below
> 10% free, earlyoom issues SIGTERM to the process with the largest
> oom_score. If both RAM and swap go below 5% free, earlyoom issues SIGKILL
> to the process with the largest oom_score. The idea is to recover from out
> of memory situations sooner, rather than the typical complete system hang
> in which the user has no other choice but to force power off.
> 
> == Owner ==
> * Name: [[User:bcotton|Ben Cotton]]
> * Email: bcot...@redhat.com
> 
> == Detailed Description ==
> Shamelessly copied from Workstation, which did it in the last release:
> 
> Certain workloads have heavy memory demands, quickly consume all of RAM,
> and start to heavily page out to swap. (Heavy paging, is often called "swap
> thrashing" for added descriptive effect, probably because it's noticeable
> and annoying). Incidental swap usage is a good thing, it frees up memory
> for active pages used by a process. Heavy swap usage quickly leads to a
> very negative UX, because it's slow, even on modern SSDs. Due to installer
> defaults, the swap partition is made the same size as available memory (at
> install time), which can be huge. This just extends swap thrashing time.
> 
> On the one hand, we want this resource hungry job to complete. On the other
> hand, we want our system to be responsive while that other work is going
> on. But once the GUI stutters or even comes to an apparent stand still
> (hang), we're really wishing the kernel oom-killer would kick in and free
> up memory, so we can start over (maybe using memory or thread limiting
> options - which arguably should be more intelligently figured out, and that
> too is a work in progress but beyond the scope of this feature).
> 
> However, once in a heavy swap scenario, it's relatively common the system
> gets stuck in it, where GUI interactivity is terrible to non-existent, and
> also the kernel oom-killer doesn't trigger. From a certain point of view,
> this is working as intended. The kernel oom-killer is concerned about
> keeping the kernel running. It's not at all concerned about user space
> responsiveness.
> 
> Instead of the system becoming completely unresponsive for tens of minutes,
> hours or days, this feature expects that an offending process (determined
> by oom_score, same as the kernel oom-killer) will be killed off within
> seconds or a few minutes.
> 
> == Benefit to Fedora ==
> 
> KDE users will be able to take advantage of the benefits Workstation users
> got from enabling earlyOOM in Fedora 32:
> 
> * improved user experience by more quickly regaining control over one's
> system, rather than having to force power off in low-memory situations
> where there's aggressive swapping. Once a system becomes unresponsive, it's
> completely reasonable for the user to assume the system is lost, but that
> includes high potential for data loss.
> * reducing forced poweroff as the main work around will increase data
> collection, improving understanding of low memory situations and how to
> handle them better
> * earlyoom first sends SIGTERM to the chosen process, so it has a chance of
> a proper shutdown, unlike the kernel's oom-killer
> 
> == Scope ==
> * Proposal owners:
> ** Modify {{code|
> https://pagure.io/fedora-comps/blob/master/f/comps-f33.xml.in}} to include
> earlyoom package for in {{code|kde-desktop}} section.
> ** Add {{code|
> https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/80-kde.prese
> t}} to include:
> 
> # enable earlyoom by default on KDE
> enable earlyoom.service
> 
> 
> * Other developers: None, unless KDE-based Spins/Labs want to opt out
> 
> * Release engineering: N/A
> * Policies and guidelines: N/A
> * Trademark approval: N/A
> 
> == Upgrade/compatibility impact ==
> earlyoom.service will be enabled on upgrade. An upgraded system should
> exhibit the same behaviors as a newly-installed system.
> 
> == How To Test ==
> * Fedora 31/32 KDE users can test today:
> ** {{code|sudo dnf install earlyoom}}
> ** {{code|sudo systemctl enable --now earlyoom}}
> 
> And then attempt to cause an out of memory situation. Examples:
> ** {{code|tail /dev/zero}}
> ** https://lkml.org/lkml/2019/8/4/15
> 
> == User Experience ==
> earlyoom sends SIGTERM to processes based on oom_score when both memory and
> swap have less than 10% free and SIGKILL when below 5%.
> 
> == Dependencies ==
> None
> 
> == Contingency Plan ==
> 
> * Contingency mechanism: (What to do?  Who will do it?) Owner reverts
> changes
> * Contingency deadline: Final freeze
> * Blocks release? No
> 
> == Documentation ==
> * {{code|man earlyoom}}
> * https://github.com/rfjakob/earlyoom
> * https://www.kernel.org/doc/gorman/html/understand/understand016.html
> 
> == Release Notes ==
> The earlyoom service 

Re: Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Ben Cotton]

On Tue, Jun 30, 2020 at 9:32 AM Neal Gompa  wrote:
>
> * Has anyone reached out to the other spins to see if they'd like to
> include it? I think it actually makes sense to do it for all Fedora
> desktops.

I reached out to the other KDE-based deliverables. If other spins
owners want to sign on, I'd welcome them on board. I agree that it
makes sense to do it more broadly.

> * How will this get activated on upgrades? Same strategy we did for
> Workstation with Supplements or something else?

Yeah, I'm planning to copy/paste the implementation from Workstation,
the way I did the proposal text (thanks, Chris!)


-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Enable EarlyOOM on Fedora KDE - Fedora 33 Self-Contained Change proposal

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/KDEEarlyOOM

== Summary ==
As [[Changes/EnableEarlyoom|Fedora Workstation did in F32]], install
earlyoom package, and enable it by default. If both RAM and swap go below
10% free, earlyoom issues SIGTERM to the process with the largest
oom_score. If both RAM and swap go below 5% free, earlyoom issues SIGKILL
to the process with the largest oom_score. The idea is to recover from out
of memory situations sooner, rather than the typical complete system hang
in which the user has no other choice but to force power off.

== Owner ==
* Name: [[User:bcotton|Ben Cotton]]
* Email: bcot...@redhat.com

== Detailed Description ==
Shamelessly copied from Workstation, which did it in the last release:

Certain workloads have heavy memory demands, quickly consume all of RAM,
and start to heavily page out to swap. (Heavy paging, is often called "swap
thrashing" for added descriptive effect, probably because it's noticeable
and annoying). Incidental swap usage is a good thing, it frees up memory
for active pages used by a process. Heavy swap usage quickly leads to a
very negative UX, because it's slow, even on modern SSDs. Due to installer
defaults, the swap partition is made the same size as available memory (at
install time), which can be huge. This just extends swap thrashing time.

On the one hand, we want this resource hungry job to complete. On the other
hand, we want our system to be responsive while that other work is going
on. But once the GUI stutters or even comes to an apparent stand still
(hang), we're really wishing the kernel oom-killer would kick in and free
up memory, so we can start over (maybe using memory or thread limiting
options - which arguably should be more intelligently figured out, and that
too is a work in progress but beyond the scope of this feature).

However, once in a heavy swap scenario, it's relatively common the system
gets stuck in it, where GUI interactivity is terrible to non-existent, and
also the kernel oom-killer doesn't trigger. From a certain point of view,
this is working as intended. The kernel oom-killer is concerned about
keeping the kernel running. It's not at all concerned about user space
responsiveness.

Instead of the system becoming completely unresponsive for tens of minutes,
hours or days, this feature expects that an offending process (determined
by oom_score, same as the kernel oom-killer) will be killed off within
seconds or a few minutes.

== Benefit to Fedora ==

KDE users will be able to take advantage of the benefits Workstation users
got from enabling earlyOOM in Fedora 32:

* improved user experience by more quickly regaining control over one's
system, rather than having to force power off in low-memory situations
where there's aggressive swapping. Once a system becomes unresponsive, it's
completely reasonable for the user to assume the system is lost, but that
includes high potential for data loss.
* reducing forced poweroff as the main work around will increase data
collection, improving understanding of low memory situations and how to
handle them better
* earlyoom first sends SIGTERM to the chosen process, so it has a chance of
a proper shutdown, unlike the kernel's oom-killer

== Scope ==
* Proposal owners:
** Modify {{code|
https://pagure.io/fedora-comps/blob/master/f/comps-f33.xml.in}} to include
earlyoom package for in {{code|kde-desktop}} section.
** Add {{code|
https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/80-kde.preset}}
to include:

# enable earlyoom by default on KDE
enable earlyoom.service


* Other developers: None, unless KDE-based Spins/Labs want to opt out

* Release engineering: N/A
* Policies and guidelines: N/A
* Trademark approval: N/A

== Upgrade/compatibility impact ==
earlyoom.service will be enabled on upgrade. An upgraded system should
exhibit the same behaviors as a newly-installed system.

== How To Test ==
* Fedora 31/32 KDE users can test today:
** {{code|sudo dnf install earlyoom}}
** {{code|sudo systemctl enable --now earlyoom}}

And then attempt to cause an out of memory situation. Examples:
** {{code|tail /dev/zero}}
** https://lkml.org/lkml/2019/8/4/15

== User Experience ==
earlyoom sends SIGTERM to processes based on oom_score when both memory and
swap have less than 10% free and SIGKILL when below 5%.

== Dependencies ==
None

== Contingency Plan ==

* Contingency mechanism: (What to do?  Who will do it?) Owner reverts
changes
* Contingency deadline: Final freeze
* Blocks release? No

== Documentation ==
* {{code|man earlyoom}}
* https://github.com/rfjakob/earlyoom
* https://www.kernel.org/doc/gorman/html/understand/understand016.html

== Release Notes ==
The earlyoom service is now enabled by default in Fedora KDE.

The earlyoom service monitors system memory usage. If free memory falls
below a set limit, earlyoom terminates an appropriate process to free up
memory. As a result, the system does not become unresponsive for long
periods of time in low-memory