Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 11:28:45AM -0500, Ben Cotton wrote: > For memory pressure configuration, this will be > ManagedOOMMemoryPressure=kill and ManagedOOMMemoryPressureLimit=4% on > user@.service to have systemd-oomd send SIGKILLs to all processes > under a selected cgroup when total memory pressure on all tasks > exceeds 4% for 10 seconds. > > For swap based actions, systemd-oomd will monitor the system-wide swap > space and act when available swap falls below the configured > threshold, starting with the cgroups with the highest swap usage to > the least. Keeping some amount of swap (if enabled) available will > prevent the kernel OOM killer from killing processes unpredictably and > spending an unbounded amount of time afterwards. > > For swap configuration, this will be SwapUsedLimitPercent=90% in > oomd.conf and ManagedOOMSwap=kill on -.slice (root cgroup slice) to > have systemd-oomd send SIGKILLs to all processes under a cgroup when > swap used exceeds 90%. One implementation approach would be to put the relevant config in dropins (outside of the main config file), and to package them into a separate rpm, e.g. systemd-oomd-defaults, similarly to what was done with zram-generator-defaults. Dropins are easier to override, and it's easy to opt-in/opt-out of the config when that config is installable separately from the implementation. -- Replying to the discussion in other threads, I think it makes a lot of sense to deploy this in F34. Right now oomd works, but we don't have a lot of feedback and there aren't really any big outstanding issues. And without the feedback, it will not get improved. So postponing until F35 doesn't seem to serve any particular goal. -- Replying to the discussion in other threads, I think it's clear that many people will need/want to opt-out. (For example it would be great if sway would separate it's processes into .slices/.services, and I think it'd be fairly easy to implement since there's a common ipc entrypoint to launch things, but it's unlikely to happen for F34. Doing this will have other advantages too, it's slowly becoming a must-have feature.) I'm not sure how to make the opt-in/opt-out decision. Per-variant/spin/whatever is not granular enough, because different DE and different payloads that cut across variants need different handling. I think gnome-workstation and the kde spins would gain most from this, so maybe could pull in systemd-oomd-defaults from @gnome-desktop-environment and @kde-desktop-environment groups? Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 28, 2020 at 7:30 PM Davide Cavalca via devel wrote: > > On Mon, 2020-12-28 at 16:22 -0800, Kevin Fenzi wrote: > > On Mon, Dec 28, 2020 at 07:44:04PM -, Tom Seewald wrote: > > > > On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang > > > > > > > > > > > > > > > Another variation on this theme: enable by default in Fedora 34 > > > > Server > > > > edition. And more broadly rolled out for Fedora 35. > > > > > > > > If it's broadly ready for Fedora 34, great. Otherwise, it seems > > > > like a > > > > good fit for Fedora Server edition, given sd-oomd's server origin > > > > and > > > > oomd2 been used in production for a number of years. It'd be a > > > > significant headline feature for Server edition, especially > > > > fitting > > > > for the in-progress reboot of that project. Any thoughts from > > > > Server > > > > WG folks? > > > > > > I don't think enabling systemd-oomd for Fedora server by default > > > makes a lot of sense right now. Why would we want to automatically > > > enable systemd-oomd in cases where users either have to manually > > > manage cgroups or risk a worse experience than what currently > > > exists with earlyoom? If a user is already creating/managing > > > cgroups themselves, then manually enabling systemd-oomd would be a > > > minor extra step. But if the user isn't managing cgroups (which I > > > believe is the common case), then that user would be pretty > > > surprised if systemd-oomd wipes out a huge swath of running > > > programs that happen to be in the same cgroup. > > > > > > With Gnome and KDE Plasma most of the cgroup creation is done > > > automatically, so it makes more sense to enable systemd-oomd by > > > default as those systems are already set up for systemd-oomd to > > > work well. > > > > I'm a bit unclear on this and perhaps the change owners could > > comment: > > > > This operates on slices right? So in a server context each service is > > in > > it's own slice. So, while it might kill say a httpd slice, other > > services would be fine? That doesn't seem great if you are running > > a deadicated webserver, but perhaps not so bad if you are running a > > bunch of different services. > > This is correct. For server usecases, I can see services primarily > getting started via systemd or via a container manager. In both cases, > every service should get its own cgroup by default. Usecases I can see > as being problematic: > - stuff like runit would probably put all their charges under the same > cgroup > - all cronjobs will be under the cron cgroup (systemd timers would work > just fine though) > - if one starts a long running service in an unortodox way (say, using > screen), that's likely going to end up accounted as part of their user > sessions > > All in all, I think this should be a fairly safe change for servers, as > long as we document properly the potential pitfalls. > Indeed. In fact, I would suggest this probably would encourage people to move to the current best practice of separating contexts for each thing they are running, which is probably a good thing. > > Also, perhaps there's some way to teach the web server applications > > to > > put say different wsgi applications in different slices, but no idea > > if > > any work has gone into that. > > Depending on the web server, this shouldn't be too difficult to do > (either by leveraging template services in systemd, or by writing some > glue logic that leverages the run API to setup slices directly). > It's generally been good practice to have individual systemd service units for WSGI applications being run through harnesses like gunicorn or uwsgi. For example, when pagure is deployed using nginx, the main web app and the docs web app are both separate systemd services invoking gunicorn. Thus, they get their own resource controls through their own cgroups. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, 2020-12-28 at 16:22 -0800, Kevin Fenzi wrote: > On Mon, Dec 28, 2020 at 07:44:04PM -, Tom Seewald wrote: > > > On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang > > > > > > > > > > > Another variation on this theme: enable by default in Fedora 34 > > > Server > > > edition. And more broadly rolled out for Fedora 35. > > > > > > If it's broadly ready for Fedora 34, great. Otherwise, it seems > > > like a > > > good fit for Fedora Server edition, given sd-oomd's server origin > > > and > > > oomd2 been used in production for a number of years. It'd be a > > > significant headline feature for Server edition, especially > > > fitting > > > for the in-progress reboot of that project. Any thoughts from > > > Server > > > WG folks? > > > > I don't think enabling systemd-oomd for Fedora server by default > > makes a lot of sense right now. Why would we want to automatically > > enable systemd-oomd in cases where users either have to manually > > manage cgroups or risk a worse experience than what currently > > exists with earlyoom? If a user is already creating/managing > > cgroups themselves, then manually enabling systemd-oomd would be a > > minor extra step. But if the user isn't managing cgroups (which I > > believe is the common case), then that user would be pretty > > surprised if systemd-oomd wipes out a huge swath of running > > programs that happen to be in the same cgroup. > > > > With Gnome and KDE Plasma most of the cgroup creation is done > > automatically, so it makes more sense to enable systemd-oomd by > > default as those systems are already set up for systemd-oomd to > > work well. > > I'm a bit unclear on this and perhaps the change owners could > comment: > > This operates on slices right? So in a server context each service is > in > it's own slice. So, while it might kill say a httpd slice, other > services would be fine? That doesn't seem great if you are running > a deadicated webserver, but perhaps not so bad if you are running a > bunch of different services. This is correct. For server usecases, I can see services primarily getting started via systemd or via a container manager. In both cases, every service should get its own cgroup by default. Usecases I can see as being problematic: - stuff like runit would probably put all their charges under the same cgroup - all cronjobs will be under the cron cgroup (systemd timers would work just fine though) - if one starts a long running service in an unortodox way (say, using screen), that's likely going to end up accounted as part of their user sessions All in all, I think this should be a fairly safe change for servers, as long as we document properly the potential pitfalls. > Also, perhaps there's some way to teach the web server applications > to > put say different wsgi applications in different slices, but no idea > if > any work has gone into that. Depending on the web server, this shouldn't be too difficult to do (either by leveraging template services in systemd, or by writing some glue logic that leverages the run API to setup slices directly). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 28, 2020 at 07:44:04PM -, Tom Seewald wrote: > > On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang > > > > > > > Another variation on this theme: enable by default in Fedora 34 Server > > edition. And more broadly rolled out for Fedora 35. > > > > If it's broadly ready for Fedora 34, great. Otherwise, it seems like a > > good fit for Fedora Server edition, given sd-oomd's server origin and > > oomd2 been used in production for a number of years. It'd be a > > significant headline feature for Server edition, especially fitting > > for the in-progress reboot of that project. Any thoughts from Server > > WG folks? > > I don't think enabling systemd-oomd for Fedora server by default makes a lot > of sense right now. Why would we want to automatically enable systemd-oomd in > cases where users either have to manually manage cgroups or risk a worse > experience than what currently exists with earlyoom? If a user is already > creating/managing cgroups themselves, then manually enabling systemd-oomd > would be a minor extra step. But if the user isn't managing cgroups (which I > believe is the common case), then that user would be pretty surprised if > systemd-oomd wipes out a huge swath of running programs that happen to be in > the same cgroup. > > With Gnome and KDE Plasma most of the cgroup creation is done automatically, > so it makes more sense to enable systemd-oomd by default as those systems are > already set up for systemd-oomd to work well. I'm a bit unclear on this and perhaps the change owners could comment: This operates on slices right? So in a server context each service is in it's own slice. So, while it might kill say a httpd slice, other services would be fine? That doesn't seem great if you are running a deadicated webserver, but perhaps not so bad if you are running a bunch of different services. Also, perhaps there's some way to teach the web server applications to put say different wsgi applications in different slices, but no idea if any work has gone into that. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
> On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang wrote: > > > Another variation on this theme: enable by default in Fedora 34 Server > edition. And more broadly rolled out for Fedora 35. > > If it's broadly ready for Fedora 34, great. Otherwise, it seems like a > good fit for Fedora Server edition, given sd-oomd's server origin and > oomd2 been used in production for a number of years. It'd be a > significant headline feature for Server edition, especially fitting > for the in-progress reboot of that project. Any thoughts from Server > WG folks? I don't think enabling systemd-oomd for Fedora server by default makes a lot of sense right now. Why would we want to automatically enable systemd-oomd in cases where users either have to manually manage cgroups or risk a worse experience than what currently exists with earlyoom? If a user is already creating/managing cgroups themselves, then manually enabling systemd-oomd would be a minor extra step. But if the user isn't managing cgroups (which I believe is the common case), then that user would be pretty surprised if systemd-oomd wipes out a huge swath of running programs that happen to be in the same cgroup. With Gnome and KDE Plasma most of the cgroup creation is done automatically, so it makes more sense to enable systemd-oomd by default as those systems are already set up for systemd-oomd to work well. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants(System-Wide Change)
On Thu, Dec 24, 2020 at 06:01:51PM -0500, James Cassell wrote: > > On Tue, Dec 22, 2020, at 10:45 AM, Michael Catanzaro wrote: > > Each gnome-terminal tab runs in its own cgroup: > > > > │ │ │ ├─app-org.gnome.Terminal.slice > > │ │ │ │ > > ├─vte-spawn-1d1d5519-71d2-4035-929a-8a9ae5bc3822.scope > > │ │ │ │ │ ├─7848 bash > > │ │ │ │ │ └─7889 less /etc/hosts > > │ │ │ │ > > ├─vte-spawn-03fe4cc9-0400-4723-ac9b-e929b850ca02.scope > > │ │ │ │ │ ├─7892 bash > > │ │ │ │ │ ├─7927 systemd-cgls > > │ │ │ │ │ └─7928 less > > │ │ │ │ └─gnome-terminal-server.service > > │ │ │ │ └─7843 /usr/libexec/gnome-terminal-server > > > > Glad to hear gnome-terminal separates tabs. Any idea if tilix does the same? Yes, with Tilix there are also a number of vte-spawn-….scope units. -- Tomasz Torcz RIP is irrevelant. Spoofing is futile. to...@pipebreaker.pl Your routes will be aggreggated. -- Alex Yuriev ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
Am 21.12.20 um 17:45 schrieb Neal Gompa: To revert back to earlyoom, run: sudo systemctl disable --now systemd-oomd sudo systemctl enable --now earlyoom See man oomd.conf for configuration options. What really makes me happy, if oomds keep clamd running, instead of killing it, just because it has 750M+ memory and running for ages. A whitelist of some kind would be nice. best regards, Marius Schwarz ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants(System-Wide Change)
On Tue, Dec 22, 2020, at 10:45 AM, Michael Catanzaro wrote: > Each gnome-terminal tab runs in its own cgroup: > > │ │ │ ├─app-org.gnome.Terminal.slice > │ │ │ │ > ├─vte-spawn-1d1d5519-71d2-4035-929a-8a9ae5bc3822.scope > │ │ │ │ │ ├─7848 bash > │ │ │ │ │ └─7889 less /etc/hosts > │ │ │ │ > ├─vte-spawn-03fe4cc9-0400-4723-ac9b-e929b850ca02.scope > │ │ │ │ │ ├─7892 bash > │ │ │ │ │ ├─7927 systemd-cgls > │ │ │ │ │ └─7928 less > │ │ │ │ └─gnome-terminal-server.service > │ │ │ │ └─7843 /usr/libexec/gnome-terminal-server > > Firefox does this too. WebKit doesn't, but it won't be hard to fix. Not > sure about Chrome. Point is, apps have all the tools they need to use > cgroups to ensure out-of-control subprocesses don't cause the main > process to be killed, and our important default apps (Firefox and > gnome-terminal) actually do this. > It's there a way to make bash launch each command in a separate cgroup? I've usually got my various terminal apps in the background of a single bash session. It sounds like if any of them gets killed by oomd, they all would be killed. Glad to hear gnome-terminal separates tabs. Any idea if tilix does the same? V/r, James Cassell ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Wed, Dec 23, 2020 at 12:32 pm, Vít Ondruch wrote: You see that I have executed FF from terminal. But I think that if any other process running in the app-org.gnome.Terminal.slice starts to grow memory usage, the whole tree will be killed. systemd scopes get cgroups too, so only the scope corresponding to a particular vte widget should be killed (i.e. one terminal tab). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
Dne 22. 12. 20 v 16:45 Michael Catanzaro napsal(a): On Tue, Dec 22, 2020 at 10:45 am, Vít Ondruch wrote: And I wonder what will be the behavior for applications, which I start from my terminal? The most typical example for me is running GVim from gnome-terminal. Each gnome-terminal tab runs in its own cgroup: │ │ │ ├─app-org.gnome.Terminal.slice │ │ │ │ ├─vte-spawn-1d1d5519-71d2-4035-929a-8a9ae5bc3822.scope │ │ │ │ │ ├─7848 bash │ │ │ │ │ └─7889 less /etc/hosts │ │ │ │ ├─vte-spawn-03fe4cc9-0400-4723-ac9b-e929b850ca02.scope │ │ │ │ │ ├─7892 bash │ │ │ │ │ ├─7927 systemd-cgls │ │ │ │ │ └─7928 less │ │ │ │ └─gnome-terminal-server.service │ │ │ │ └─7843 /usr/libexec/gnome-terminal-server Firefox does this too. WebKit doesn't, but it won't be hard to fix. Not sure about Chrome. Point is, apps have all the tools they need to use cgroups to ensure out-of-control subprocesses don't cause the main process to be killed, and our important default apps (Firefox and gnome-terminal) actually do this. But I am worried about this scenario: │ │ │ ├─app-org.gnome.Terminal.slice │ │ │ │ ├─vte-spawn-10bea11a-51c3-4bf7-a3a1-8556d7bf35a6.scope │ │ │ │ │ ├─ 2444 bash │ │ │ │ │ ├─2863977 gvim /home/vondruch/scripts/ruby-devel-srpm.rb │ │ │ │ │ ├─3687301 gvim ruby.spec │ │ │ │ │ ├─3733657 /usr/bin/python3 -tt /usr/libexec/mock/mock -r ruby ruby-3.0.0-0.1.20201223git684649ea05.fc34.src.rpm -n │ │ │ │ │ └─3733903 /usr/bin/systemd-nspawn -q -M 31d5751c88234fe284db838faf24cba3 -D /var/lib/mock/ruby/root -a -u mockbuild --capability=cap_ipc_lock --bind=/tmp/mock-resolv.q2hfy7am:/etc/resolv.conf --bind> │ │ │ │ ├─vte-spawn-c37b3930-5585-4417-8948-eeaa497a595e.scope │ │ │ │ │ ├─3742577 bash │ │ │ │ │ ├─3759605 systemd-cgls │ │ │ │ │ └─3759606 less │ │ │ │ ├─vte-spawn-b02099b6-6f18-49c2-b8a8-47d484f18663.scope │ │ │ │ │ └─3706817 bash │ │ │ │ ├─vte-spawn-5c35e1e8-1227-4717-8c4f-a8698bf3918f.scope │ │ │ │ │ ├─3757639 bash │ │ │ │ │ ├─3759184 /usr/lib64/firefox/firefox │ │ │ │ │ ├─3759273 /usr/lib64/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 1 -prefMapSize 312094 -parentBuildID 20201106142436 -appdir /usr/lib64/firefox/browser 3759184 tab │ │ │ │ │ ├─3759355 /usr/lib64/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 6262 -prefMapSize 312094 -parentBuildID 20201106142436 -appdir /usr/lib64/firefox/browser 3759184 tab │ │ │ │ │ ├─3759427 /usr/bin/python3 /usr/bin/chrome-gnome-shell /usr/lib64/mozilla/native-messaging-hosts/org.gnome.chrome_gnome_shell.json chrome-gnome-sh...@gnome.org │ │ │ │ │ └─3759459 /usr/lib64/firefox/firefox -contentproc -childID 3 -isForBrowser -prefsLen 7194 -prefMapSize 312094 -parentBuildID 20201106142436 -appdir /usr/lib64/firefox/browser 3759184 tab You see that I have executed FF from terminal. But I think that if any other process running in the app-org.gnome.Terminal.slice starts to grow memory usage, the whole tree will be killed. Vít Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org OpenPGP_0x0CE09EE79917B87C.asc Description: application/pgp-keys OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang wrote: > > That's why I think this change need to be postponed to Fedora 35 (opt-in > > in F34 and default in F35). > > I'm actually not opposed to that. Another variation on this theme: enable by default in Fedora 34 Server edition. And more broadly rolled out for Fedora 35. If it's broadly ready for Fedora 34, great. Otherwise, it seems like a good fit for Fedora Server edition, given sd-oomd's server origin and oomd2 been used in production for a number of years. It'd be a significant headline feature for Server edition, especially fitting for the in-progress reboot of that project. Any thoughts from Server WG folks? -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
> Earlyoom maintainer here. I think it's too early to switch to > systemd-oomd, because it was just merged to the systemd codebase and is > still an experimental feature. Hi! I authored the PR for systemd-oomd. It was merged as a feature for "preview" rather than "release", but that was so the interface could be improved based on feedback. The feature itself was tested against a subset of Facebook's servers and it behaves the same as the stand alone oomd that we've been running for years (minus the knobs that were not implemented in systemd-oomd). > In earlyoom we have a list of processes that cannot be killed (eg. > dnf/packagekit/etc.), so it is absolutely safe to use as a default > userspace OOM solution. We currently don't know anything about the > systemd-oomd safety for regular use on end user desktops. We can't even > test it on the current stable Fedora release. PSI takes into account cgroup memory protections so you can bias away from cgroups with critical apps by setting MemoryLow= to the appropriate values to prevent reclaim. > That's why I think this change need to be postponed to Fedora 35 (opt-in > in F34 and default in F35). I'm actually not opposed to that. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 3:42 PM Tom Seewald wrote: > > > This is intended to be a generic approach to user space oom > > management, but it does tie into resource control too. And the > > resource control organization of what processes are considered > > critical are different between a desktop and a server. The idea of > > "user wants to take control or see what's going on" is a generally > > important goal for all of this work, regardless of the Fedora edition > > or spin. > > So are you confirming that users are now going to need to place things in > their own cgroup if they do not want systemd-oomd to potentially kill the > single cgroup containing all of their running applications? I'm not saying that. I don't mind user configuration for special cases, but that is not a general solution. Today earlyoom will SIGTERM a specific Firefox process, i.e. a tab can be clobbered and it just looks like it crashed, there's no other notification. It doesn't kill off the whole program with all of your tabs. But if it's true that all of my programs in the shell could be subject to being clobbered at once? Is that any better than just hitting the power button or pulling the power cable? The answer is, yes. But only incrementally. And it's not better than what we have today with earlyoom. Also, sd-oomd should be able to send SIGTERM so that there is a chance that process can save state before a SIGKILL is issued. Earlyoom also does that today. > I think this should at the very least be clearly documented in the change > proposal, as this user experience is in stark contrast to what Gnome and KDE > users will encounter. Yeah I'm not sure to what degree these knobs are even user facing. If we're talking about developers and packagers, then yes there should be documentation for them if there's things they need to consider. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 10:41 pm, Tom Seewald wrote: So are you confirming that users are now going to need to place things in their own cgroup if they do not want systemd-oomd to potentially kill the single cgroup containing all of their running applications? If this change is approved: yes! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020, at 2:45 PM, Tom Seewald wrote: > > If your desktop doesn't segregate apps and services into cgroups, > > systemd-oomd will kill the entire desktop whenever anything uses too > > much memory, because the desktop is going to be running in the same > > cgroup as the apps that it launches. So I think desktop spins (other > > than KDE) ought to opt out of this. It should be good for all Fedora > > editions, though (including Workstation, Server, Atomic, CoreOS), and > > also for KDE spin. > > How will this work on headless systems like Fedora Server, Atomic, and > CoreOS? Will it be expected that users manually create their own > cgroups? This should really be called out in the change. There's a lot of profound things going on here. First, just speaking for Fedora CoreOS I am skeptical of us just turning this on by default. It's tricky because I can see this potentially being beneficial for "self managed containers" (i.e. a few FCOS nodes with manually set up containers) but a major use case of FCOS is Kubernetes (e.g. OKD, Typhoon, upstream kubeadm etc.) and I don't see any mention about integration with kubelet. Today requests/limits are strongly emphasized: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits Now from experience it's definitely common for people to deploy pods without any limits, and that can benefit from better low-memory/OOM handling. Kubelet already has some of that code: https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-oom-behavior There's an even more obvious stumbling block that FCOS doesn't set up swap by default, and kubelet explicitly fails if it's enabled. Though it looks like there's recent movement in that area: https://github.com/kubernetes/kubernetes/issues/53533 But basically unless someone does groundwork on what swap looks like for FCOS (particularly in cloud instances) and has at least links to kubelet integration I'd expect this to be off on FCOS. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
> This is intended to be a generic approach to user space oom > management, but it does tie into resource control too. And the > resource control organization of what processes are considered > critical are different between a desktop and a server. The idea of > "user wants to take control or see what's going on" is a generally > important goal for all of this work, regardless of the Fedora edition > or spin. So are you confirming that users are now going to need to place things in their own cgroup if they do not want systemd-oomd to potentially kill the single cgroup containing all of their running applications? I think this should at the very least be clearly documented in the change proposal, as this user experience is in stark contrast to what Gnome and KDE users will encounter. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 12:45 PM Tom Seewald wrote: > > > If your desktop doesn't segregate apps and services into cgroups, > > systemd-oomd will kill the entire desktop whenever anything uses too > > much memory, because the desktop is going to be running in the same > > cgroup as the apps that it launches. So I think desktop spins (other > > than KDE) ought to opt out of this. It should be good for all Fedora > > editions, though (including Workstation, Server, Atomic, CoreOS), and > > also for KDE spin. > > How will this work on headless systems like Fedora Server, Atomic, and > CoreOS? Will it be expected that users manually create their own cgroups? This is intended to be a generic approach to user space oom management, but it does tie into resource control too. And the resource control organization of what processes are considered critical are different between a desktop and a server. The idea of "user wants to take control or see what's going on" is a generally important goal for all of this work, regardless of the Fedora edition or spin. On the desktop, this is mainly GUI responsiveness. On Server this is probably logging, the entire network stack, and sshd - at a minimum. So those critical components. Organizing such services so that they get the minimum resources they require, no matter what resource hog happens to come along, is something that'd work out of the box. Right now the program that has the highest "badness" score gets killed off. And the badness score doesn't take its relative hit to PSI into account. That is, the program that's actually causing the worse pressure may not be the one killed off today. And that's because badness score is pretty simplistic. All things being equal, the one with the highest badness is the most recently launched program. That may not be how you want it scored even by default, you'd probably want to kill the one that's contributing the most inefficiency overall due to its behavior with respect to memory, io, and cpu consumption. I posted this to the server list the other day, but it gives a general overview of how oomd2 works in server user cases and explains a lot of the basic concepts which I think are easier to grasp for the server use case than the more complex desktop case. https://www.youtube.com/watch?v=30i7SamZxRU -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 7:45 pm, Tom Seewald wrote: How will this work on headless systems like Fedora Server, Atomic, and CoreOS? Will it be expected that users manually create their own cgroups? Either that, or don't use too much memory, yes. I assume SIGKILL is probably at least somewhat better than being unable to reboot your server because your terminal is no longer responsive, right? Well, I think you exposed the flaw in my argument. I say that on servers, it's OK to kill everything rather than freeze, but killing the entire desktop is pretty clearly not OK on desktops. So maybe it's not OK on servers, either. I have no strong opinion here. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 11:42 AM Robbie Harwood wrote: > > > I believe you are assuming the consequent when you suggest that kernel > developers should be somehow fixing this in userspace. > > To back up: the described problem is the manifestation of an interaction > between swap and the OOM condition. The OOM killer is a > popularly-understood piece of what goes on in the system during OOM, but > it's not like the rest of the kernel can be ignored. (I would argue > that part of the reason it's well understood is their insistence that it > remain simple, but that's getting off into the weeds.) No the problem happens any time a resource becomes constrained: cpu, memory, io. It's not exclusively a swap problem. When swap pressure is part of the problem, it depends on how swap is being used. Heavy IO page out is a good thing. Heavy IO page-in and page-out of the same pages is a bad thing. > > So, several control points here: > > - OOM killer behavior. I think we're in agreement that this isn't the > thing that needs changed. If you mean the kernel oomkiller, yeah probably. That's generally considered to be working correctly. It keeps the kernel functioning, with forward progress being made without any respect whatsoever to user space priorities like system responsiveness. > - Enabling swap. Swap is really slow (by virtue of not being RAM...) > and we don't seem willing to acknowledge that. If we want the system > to be snappy and responsive... we shouldn't be swapping. This is not entirely correct. The chosen workload might be excessive compared to the allocated resources. That does happen, I see it from time to time, but it's not that common because it results in a lot of pain for the user. So they stop doing it. This is an underprovisioned system. If you aren't swapping at all, that means you have allocated more resources than the workload requires. You've over provisioned. This is apparently quite common in the Kubernetes workflow, because Kubernetes doesn't work properly with swap, somehow by design. So their view is, don't create a swap device, just overprovision. Swap is for evicting anonymous pages, pages that aren't backed by any kind of file. If inactive anonymous pages can't be swapped, they have to stay in memory. And when memory is under pressure, the kernel has no choice but to resort to reclaim, i.e. evict pages that are backed by files. This will end up looking a lot like swap thrashing. Another factor is there have been recent improvements in the swap code to make dirty page eviction much better and avoid swap thrashing. You'll need a 5.10 kernel for the most recent work on this. > > - Swap aggressiveness. Suggested by above, people want swap anyway. > (Sometimes it's for hibernation (not supported, but that stops no > one), sometimes it's for... historical reasons? Underprovisioning?) > This could be tuned to the use cases we actually want. The idea of proper resource control is to use swap more effectively, to reduce the heavy swap thrashing. It's not a problem to do dirty page eviction (page out). That frees memory and makes it less likely other processes will thrash. > > - Education. Get people to a point where admins don't deploy swap on > systems that aren't going to hibernate. I'll readily admit this one > might be hardest. That is bad advice. We do need swap. https://chrisdown.name/2018/01/02/in-defence-of-swap.html There's a nice tl;dr at the top and a summary at the bottom. And quite detailed explanation in the middle. > And even possibly the (conceptually) simplest solution of all: > > - Swap usage monitoring as described for oomd... but in the kernel. > This saves you on all the overhead of running in userspace, if nothing > else. This exists in the form of PSI, as well as cgroupsv2: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html memory.swap.current memory.swap.events memory.swap.high memory.swap.max > But what really bothers me here is that, to my knowledge, no one has > tried to actually make any of these happen in the kernel. There's a > vague perception of what "the kernel devs" want, as if they're some > other, but... has anyone asked? If so, we should be able to quote what > the response was, and a good design proposal should include it as an > explanation of why that route wasn't taken. I'm not even sure what you're asking for. There is no such thing as a one size fits all set of policies for resource control. There are kernel-side components for this, as well as user space, to implement a policy. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
> If your desktop doesn't segregate apps and services into cgroups, > systemd-oomd will kill the entire desktop whenever anything uses too > much memory, because the desktop is going to be running in the same > cgroup as the apps that it launches. So I think desktop spins (other > than KDE) ought to opt out of this. It should be good for all Fedora > editions, though (including Workstation, Server, Atomic, CoreOS), and > also for KDE spin. How will this work on headless systems like Fedora Server, Atomic, and CoreOS? Will it be expected that users manually create their own cgroups? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 2:03 PM Michael Catanzaro wrote: > > On Tue, Dec 22, 2020 at 6:55 pm, Tom Seewald wrote: > > Overall I like the change for desktop use, but I'm not sure it > > currently is a good fit for non-Workstation/KDE spins of Fedora. > > If your desktop doesn't segregate apps and services into cgroups, > systemd-oomd will kill the entire desktop whenever anything uses too > much memory, because the desktop is going to be running in the same > cgroup as the apps that it launches. So I think desktop spins (other > than KDE) ought to opt out of this. It should be good for all Fedora > editions, though (including Workstation, Server, Atomic, CoreOS), and > also for KDE spin. > To be clear, the "opt-out", as it were, is just to not include systemd-oomd in the comps groups for those variants. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 6:55 pm, Tom Seewald wrote: Overall I like the change for desktop use, but I'm not sure it currently is a good fit for non-Workstation/KDE spins of Fedora. If your desktop doesn't segregate apps and services into cgroups, systemd-oomd will kill the entire desktop whenever anything uses too much memory, because the desktop is going to be running in the same cgroup as the apps that it launches. So I think desktop spins (other than KDE) ought to opt out of this. It should be good for all Fedora editions, though (including Workstation, Server, Atomic, CoreOS), and also for KDE spin. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 1:41 pm, Robbie Harwood wrote: - OOM killer behavior. I think we're in agreement that this isn't the thing that needs changed. Let's back up. The choice is between earlyoom (status quo) or systemd-oomd (future). We're not going to get rid of our userspace OOM killer, because we've found a userspace killer is necessary to avoid locking up when under memory pressure. That ship sailed when we decided to ship earlyoom. (Well, in theory, we *could* change our minds about that... but we should not, because earlyoom has significantly improved our responsiveness under low memory conditions.) earlyoom was always recognized as a stopgap measure until systemd was ready to take over. - Enabling swap. Swap is really slow (by virtue of not being RAM...) and we don't seem willing to acknowledge that. If we want the system to be snappy and responsive... we shouldn't be swapping. - Swap aggressiveness. Suggested by above, people want swap anyway. (Sometimes it's for hibernation (not supported, but that stops no one), sometimes it's for... historical reasons? Underprovisioning?) This could be tuned to the use cases we actually want. - Education. Get people to a point where admins don't deploy swap on systems that aren't going to hibernate. I'll readily admit this one might be hardest. And even possibly the (conceptually) simplest solution of all: - Swap usage monitoring as described for oomd... but in the kernel. This saves you on all the overhead of running in userspace, if nothing else. I don't think we need to be discussing swap at all. Any amount of swap means you won't have good performance under memory pressure. We don't create a disk-based swap partition by default anymore, and swap on zram is relatively fast, so there's no reason to consider swap as part of this discussion. We are discussing defaults, after all. If you're going to use a non-default configuration, that's fine of course, but it shouldn't affect our decisions on how Fedora should work by default. Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
> On Mon, Dec 21, 2020 at 1:51 pm, Aleksei Bavshin wrote: > > Hm good point. I think only GNOME and KDE create systemd scopes when > launching apps; systemd-oomd is not going to work well in other > desktops. Probably other desktop spins should opt-out of this change > for now. > > Michael Does this change apply to *all* Fedora releases? Overall I like the change for desktop use, but I'm not sure it currently is a good fit for non-Workstation/KDE spins of Fedora. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
Dan Čermák writes: > Robbie Harwood writes: > >> Ben Cotton writes: >> >>> For swap based actions, systemd-oomd will monitor the system-wide swap >>> space and act when available swap falls below the configured >>> threshold, starting with the cgroups with the highest swap usage to >>> the least. Keeping some amount of swap (if enabled) available will >>> prevent the kernel OOM killer from killing processes unpredictably and >>> spending an unbounded amount of time afterwards. >> >> -1 from me. If the kernel behavior is a problem, fix it - don't kludge >> around it in userspace. > > That is unlikely to happen. As far as I know, the kernel devs see the > kernel oom killer as a kernel self protection measure and want it to > fire as the last resort (and they are quite hesitant to touch > userspace). I believe you are assuming the consequent when you suggest that kernel developers should be somehow fixing this in userspace. To back up: the described problem is the manifestation of an interaction between swap and the OOM condition. The OOM killer is a popularly-understood piece of what goes on in the system during OOM, but it's not like the rest of the kernel can be ignored. (I would argue that part of the reason it's well understood is their insistence that it remain simple, but that's getting off into the weeds.) So, several control points here: - OOM killer behavior. I think we're in agreement that this isn't the thing that needs changed. - Enabling swap. Swap is really slow (by virtue of not being RAM...) and we don't seem willing to acknowledge that. If we want the system to be snappy and responsive... we shouldn't be swapping. - Swap aggressiveness. Suggested by above, people want swap anyway. (Sometimes it's for hibernation (not supported, but that stops no one), sometimes it's for... historical reasons? Underprovisioning?) This could be tuned to the use cases we actually want. - Education. Get people to a point where admins don't deploy swap on systems that aren't going to hibernate. I'll readily admit this one might be hardest. And even possibly the (conceptually) simplest solution of all: - Swap usage monitoring as described for oomd... but in the kernel. This saves you on all the overhead of running in userspace, if nothing else. But what really bothers me here is that, to my knowledge, no one has tried to actually make any of these happen in the kernel. There's a vague perception of what "the kernel devs" want, as if they're some other, but... has anyone asked? If so, we should be able to quote what the response was, and a good design proposal should include it as an explanation of why that route wasn't taken. Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
Robbie Harwood writes: > Ben Cotton writes: > >> For swap based actions, systemd-oomd will monitor the system-wide swap >> space and act when available swap falls below the configured >> threshold, starting with the cgroups with the highest swap usage to >> the least. Keeping some amount of swap (if enabled) available will >> prevent the kernel OOM killer from killing processes unpredictably and >> spending an unbounded amount of time afterwards. > > -1 from me. If the kernel behavior is a problem, fix it - don't kludge > around it in userspace. That is unlikely to happen. As far as I know, the kernel devs see the kernel oom killer as a kernel self protection measure and want it to fire as the last resort (and they are quite hesitant to touch userspace). That's the reason why there has been recently quite some development around various userspace oom killers, because it becomes more and more apparent, that the kernel is not going to fix this problem. Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
Ben Cotton writes: > For swap based actions, systemd-oomd will monitor the system-wide swap > space and act when available swap falls below the configured > threshold, starting with the cgroups with the highest swap usage to > the least. Keeping some amount of swap (if enabled) available will > prevent the kernel OOM killer from killing processes unpredictably and > spending an unbounded amount of time afterwards. -1 from me. If the kernel behavior is a problem, fix it - don't kludge around it in userspace. Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Tue, Dec 22, 2020 at 10:45 am, Vít Ondruch wrote: And I wonder what will be the behavior for applications, which I start from my terminal? The most typical example for me is running GVim from gnome-terminal. Each gnome-terminal tab runs in its own cgroup: │ │ │ ├─app-org.gnome.Terminal.slice │ │ │ │ ├─vte-spawn-1d1d5519-71d2-4035-929a-8a9ae5bc3822.scope │ │ │ │ │ ├─7848 bash │ │ │ │ │ └─7889 less /etc/hosts │ │ │ │ ├─vte-spawn-03fe4cc9-0400-4723-ac9b-e929b850ca02.scope │ │ │ │ │ ├─7892 bash │ │ │ │ │ ├─7927 systemd-cgls │ │ │ │ │ └─7928 less │ │ │ │ └─gnome-terminal-server.service │ │ │ │ └─7843 /usr/libexec/gnome-terminal-server Firefox does this too. WebKit doesn't, but it won't be hard to fix. Not sure about Chrome. Point is, apps have all the tools they need to use cgroups to ensure out-of-control subprocesses don't cause the main process to be killed, and our important default apps (Firefox and gnome-terminal) actually do this. Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On 21.12.2020 17:28, Ben Cotton wrote: Provide a better experience for Fedora users in out-of-memory (OOM) situations by enabling systemd-oomd by default. Earlyoom maintainer here. I think it's too early to switch to systemd-oomd, because it was just merged to the systemd codebase and is still an experimental feature. In earlyoom we have a list of processes that cannot be killed (eg. dnf/packagekit/etc.), so it is absolutely safe to use as a default userspace OOM solution. We currently don't know anything about the systemd-oomd safety for regular use on end user desktops. We can't even test it on the current stable Fedora release. That's why I think this change need to be postponed to Fedora 35 (opt-in in F34 and default in F35). -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
Dne 21. 12. 20 v 23:20 Aleksei Bavshin napsal(a): On 12/21/20 1:53 PM, Neal Gompa wrote: On Mon, Dec 21, 2020 at 4:52 PM Aleksei Bavshin wrote: On 12/21/20 8:28 AM, Ben Cotton wrote: == Documentation == https://www.freedesktop.org/software/systemd/man/systemd-oomd.html/> https://www.freedesktop.org/software/systemd/man/oomctl.html https://www.freedesktop.org/software/systemd/man/oomd.conf.html Be aware that if you intend to enable monitoring and actions on user.slice, user-$UID.slice, or their ancestor cgroups, it is highly recommended that your programs be managed by the systemd user manager to prevent running too many processes under the same session scope (and thus avoid a situation where memory intensive tasks trigger systemd-oomd to kill everything under the cgroup). If you're using a desktop environment like GNOME, it already spawns many session components with the systemd user manager. This makes me slightly very concerned. According to cgls, I have most of the apps running directly under user-$UID.slice -> session-X.scope. That includes a compositor (sway) and a few applications that consume uncomfortably close to 100% of available memory (firefox, thunderbird, clangd, gcc, etc...). My understanding is that unless I configure all of the above to run under dedicated scopes, an attempt to run a memory-intensive task would make systemd-oomd terminate the whole user slice with all my apps. Is there anything that could be improved for that scenario? I don't expect that all our desktop users would start using `systemd-run --scope --user` to launch their applications. My understanding is that we intend to do exactly that for you automatically when you open an application through a desktop file. So it should be fine from that perspective. Should I mention that this is happening not in GNOME and neither make nor vim are using desktop files to start subprocesses? And there are dozens of application launchers in Fedora repositories that aren't aware of systemd or cgroups. And I wonder what will be the behavior for applications, which I start from my terminal? The most typical example for me is running GVim from gnome-terminal. Vít I'm raising that point because I'd like to have at least a set of recommendations for any alternative environments. So far it seems like the impact outside of major DEs and standard desktop workflows wasn't considered. Don't take this as an antagonism to the proposal, I see the benefits of systemd-oomd and will likely use it myself with a few shell aliases, patches and config changes. It's just that as a maintainer of one of those alternative desktop environments I have no idea how to make that work in default configuration. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org OpenPGP_0x0CE09EE79917B87C.asc Description: application/pgp-keys OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 2:20 pm, Aleksei Bavshin wrote: It's just that as a maintainer of one of those alternative desktop environments I have no idea how to make that work in default configuration. You're going to want to create a new systemd scope for each application that you launch. I'd start by reading https://lwn.net/Articles/834329/ for an overview. https://gitlab.gnome.org/GNOME/gnome-desktop/-/blob/master/libgnome-desktop/gnome-systemd.c shows how we do it. If you use glib, then https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1596 will make this a lot easier! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On 12/21/20 1:53 PM, Neal Gompa wrote: On Mon, Dec 21, 2020 at 4:52 PM Aleksei Bavshin wrote: On 12/21/20 8:28 AM, Ben Cotton wrote: == Documentation == https://www.freedesktop.org/software/systemd/man/systemd-oomd.html https://www.freedesktop.org/software/systemd/man/oomctl.html https://www.freedesktop.org/software/systemd/man/oomd.conf.html Be aware that if you intend to enable monitoring and actions on user.slice, user-$UID.slice, or their ancestor cgroups, it is highly recommended that your programs be managed by the systemd user manager to prevent running too many processes under the same session scope (and thus avoid a situation where memory intensive tasks trigger systemd-oomd to kill everything under the cgroup). If you're using a desktop environment like GNOME, it already spawns many session components with the systemd user manager. This makes me slightly very concerned. According to cgls, I have most of the apps running directly under user-$UID.slice -> session-X.scope. That includes a compositor (sway) and a few applications that consume uncomfortably close to 100% of available memory (firefox, thunderbird, clangd, gcc, etc...). My understanding is that unless I configure all of the above to run under dedicated scopes, an attempt to run a memory-intensive task would make systemd-oomd terminate the whole user slice with all my apps. Is there anything that could be improved for that scenario? I don't expect that all our desktop users would start using `systemd-run --scope --user` to launch their applications. My understanding is that we intend to do exactly that for you automatically when you open an application through a desktop file. So it should be fine from that perspective. Should I mention that this is happening not in GNOME and neither make nor vim are using desktop files to start subprocesses? And there are dozens of application launchers in Fedora repositories that aren't aware of systemd or cgroups. I'm raising that point because I'd like to have at least a set of recommendations for any alternative environments. So far it seems like the impact outside of major DEs and standard desktop workflows wasn't considered. Don't take this as an antagonism to the proposal, I see the benefits of systemd-oomd and will likely use it myself with a few shell aliases, patches and config changes. It's just that as a maintainer of one of those alternative desktop environments I have no idea how to make that work in default configuration. -- With best regards, Aleksei Bavshin OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 1:51 pm, Aleksei Bavshin wrote: This makes me slightly very concerned. According to cgls, I have most of the apps running directly under user-$UID.slice -> session-X.scope. That includes a compositor (sway) and a few applications that consume uncomfortably close to 100% of available memory (firefox, thunderbird, clangd, gcc, etc...). Hm good point. I think only GNOME and KDE create systemd scopes when launching apps; systemd-oomd is not going to work well in other desktops. Probably other desktop spins should opt-out of this change for now. Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 4:52 PM Aleksei Bavshin wrote: > > > > On 12/21/20 8:28 AM, Ben Cotton wrote: > > == Documentation == > > > > https://www.freedesktop.org/software/systemd/man/systemd-oomd.html > > https://www.freedesktop.org/software/systemd/man/oomctl.html > > https://www.freedesktop.org/software/systemd/man/oomd.conf.html > > > Be aware that if you intend to enable monitoring and actions on user.slice, > > user-$UID.slice, or their ancestor cgroups, it is highly recommended that > > your programs be managed by the systemd user manager to prevent running too > > many processes under the same session scope (and thus avoid a situation > > where memory intensive tasks trigger systemd-oomd to kill everything under > > the cgroup). If you're using a desktop environment like GNOME, it already > > spawns many session components with the systemd user manager. > > This makes me slightly very concerned. According to cgls, I have most of > the apps running directly under user-$UID.slice -> session-X.scope. That > includes a compositor (sway) and a few applications that consume > uncomfortably close to 100% of available memory (firefox, thunderbird, > clangd, gcc, etc...). > > My understanding is that unless I configure all of the above to run > under dedicated scopes, an attempt to run a memory-intensive task would > make systemd-oomd terminate the whole user slice with all my apps. > > Is there anything that could be improved for that scenario? I don't > expect that all our desktop users would start using `systemd-run --scope > --user` to launch their applications. > My understanding is that we intend to do exactly that for you automatically when you open an application through a desktop file. So it should be fine from that perspective. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On 12/21/20 8:28 AM, Ben Cotton wrote: == Documentation == https://www.freedesktop.org/software/systemd/man/systemd-oomd.html https://www.freedesktop.org/software/systemd/man/oomctl.html https://www.freedesktop.org/software/systemd/man/oomd.conf.html Be aware that if you intend to enable monitoring and actions on user.slice, user-$UID.slice, or their ancestor cgroups, it is highly recommended that your programs be managed by the systemd user manager to prevent running too many processes under the same session scope (and thus avoid a situation where memory intensive tasks trigger systemd-oomd to kill everything under the cgroup). If you're using a desktop environment like GNOME, it already spawns many session components with the systemd user manager. This makes me slightly very concerned. According to cgls, I have most of the apps running directly under user-$UID.slice -> session-X.scope. That includes a compositor (sway) and a few applications that consume uncomfortably close to 100% of available memory (firefox, thunderbird, clangd, gcc, etc...). My understanding is that unless I configure all of the above to run under dedicated scopes, an attempt to run a memory-intensive task would make systemd-oomd terminate the whole user slice with all my apps. Is there anything that could be improved for that scenario? I don't expect that all our desktop users would start using `systemd-run --scope --user` to launch their applications. -- With best regards, Aleksei Bavshin OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 2:35 PM Anita Zhang wrote: > > I think so. I designed the systemd-oomd interface to be general enough to > eventually support sending d-bus signals in addition to or in place of > killing cgroups with SIGKILL. But I'm uncertain if it can be implemented > before the freeze. We don't have this implemented with earlyoom today, so no worries. It just would have been very helpful to have it since it makes things nicer for Flatpaks and such. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, 2020-12-21 at 12:44 -0600, Michael Catanzaro wrote: > On Mon, Dec 21, 2020 at 9:59 am, Davide Cavalca via devel > wrote: > > We had thought about that, but one concern was migrating custom > > configuration that one might have for earlyoom, which could be > > tricky. > > If we're ok with unconditionally migrating to oomd with its default > > config, that should be pretty straightforward to do. > > Yup, that's fine. It's expected that custom configuration goes away > during a major architectural change like this. > In that case we should document how to preserve earlyoom in the F34 release notes, similar to how we have documentation on how to opt out of systemd-resolved: have a systemd preset that explicitly enables earlyoom and disables oomd. And update the Change Proposal to include modifying the default presets? -- Michel Alexandre Salim profile: https://keyoxide.org/mic...@michel-slm.name chat via email: https://delta.chat/ GPG key: 5DCE 2E7E 9C3B 1CFF D335 C1D7 8B22 9D2F 7CCC 04F2 signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
I think so. I designed the systemd-oomd interface to be general enough to eventually support sending d-bus signals in addition to or in place of killing cgroups with SIGKILL. But I'm uncertain if it can be implemented before the freeze. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 9:59 am, Davide Cavalca via devel wrote: We had thought about that, but one concern was migrating custom configuration that one might have for earlyoom, which could be tricky. If we're ok with unconditionally migrating to oomd with its default config, that should be pretty straightforward to do. Yup, that's fine. It's expected that custom configuration goes away during a major architectural change like this. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 12:59 PM Davide Cavalca via devel wrote: > > On Mon, 2020-12-21 at 10:57 -0600, Michael Catanzaro wrote: > > On Mon, Dec 21, 2020 at 11:28 am, Ben Cotton > > wrote: > > > == Upgrade/compatibility impact == > > > > > > Existing systems running earlyoom will not be modified. One can > > > transition to systemd-oomd via: > > > > > > sudo systemctl disable --now earlyoom > > > sudo systemctl enable --now systemd-oomd > > > Systems that were previously not running earlyoom will have > > > systemd-oomd enabled by default. > > > > I suggest we enable systemd-oomd for everyone on upgrade, to follow > > our > > design goal of ensuring upgraded systems match fresh installs as > > closely as practical. > > We had thought about that, but one concern was migrating custom > configuration that one might have for earlyoom, which could be tricky. > If we're ok with unconditionally migrating to oomd with its default > config, that should be pretty straightforward to do. > I think I'd rather unconditionally just move everything to oomd. It's easier to focus on shoring up the experience with oomd than to worry about earlyoom vs oomd experiences forever. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, 2020-12-21 at 10:57 -0600, Michael Catanzaro wrote: > On Mon, Dec 21, 2020 at 11:28 am, Ben Cotton > wrote: > > == Upgrade/compatibility impact == > > > > Existing systems running earlyoom will not be modified. One can > > transition to systemd-oomd via: > > > > sudo systemctl disable --now earlyoom > > sudo systemctl enable --now systemd-oomd > > Systems that were previously not running earlyoom will have > > systemd-oomd enabled by default. > > I suggest we enable systemd-oomd for everyone on upgrade, to follow > our > design goal of ensuring upgraded systems match fresh installs as > closely as practical. We had thought about that, but one concern was migrating custom configuration that one might have for earlyoom, which could be tricky. If we're ok with unconditionally migrating to oomd with its default config, that should be pretty straightforward to do. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 11:28 am, Ben Cotton wrote: == Upgrade/compatibility impact == Existing systems running earlyoom will not be modified. One can transition to systemd-oomd via: sudo systemctl disable --now earlyoom sudo systemctl enable --now systemd-oomd Systems that were previously not running earlyoom will have systemd-oomd enabled by default. I suggest we enable systemd-oomd for everyone on upgrade, to follow our design goal of ensuring upgraded systems match fresh installs as closely as practical. Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, Dec 21, 2020 at 11:29 AM Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/EnableSystemdOomd > > == Summary == > > Provide a better experience for Fedora users in out-of-memory (OOM) > situations by enabling > [https://www.freedesktop.org/software/systemd/man/systemd-oomd.html > systemd-oomd] by default. Actions taken by systemd-oomd operate on a > per-cgroup level, aligning well with the life cycle of systemd units. > systemd-oomd primarily uses [https://facebookmicrosites.github.io/psi/ > Linux pressure stall information (PSI)] to make decisions based on > wasted productivity due to resource shortages; in addition to that, it > also supports swap based actions. > > == Owners == > > * Name: [[User:anitazha|Anita Zhang]], [[User:Dcavalca|Davide > Cavalca]], [[User:Salimma|Michel Salim]], [[User:Htejun|Tejun Heo]], > [[User:3ki|Rik van Riel]] > * Email: the.anita...@gmail.com, dcava...@fb.com, > mic...@michel-slm.name, hte...@fb.com, r...@fb.com > > > == Detailed description == > > The primary mechanism used by systemd-oomd for detecting when the > system is out of memory is memory pressure. Memory pressure measures > the percentage of time a cgroup has “wasted” due to lack of memory. > This includes time spent reclaiming free memory, faulting in recently > resident pages, and loading in anonymous pages from swap. When a > monitored cgroup’s memory pressure exceeds the specified thresholds, > systemd-oomd will perform action(s) on the targeted cgroup’s > descendants, starting from the cgroups with the most reclaim scans. > Reclaim activity is used here, rather than the largest consumer, as it > reflects values set in the cgroup memory controller for memory > protection (such as memory.low). > > For memory pressure configuration, this will be > ManagedOOMMemoryPressure=kill and ManagedOOMMemoryPressureLimit=4% on > user@.service to have systemd-oomd send SIGKILLs to all processes > under a selected cgroup when total memory pressure on all tasks > exceeds 4% for 10 seconds. > > For swap based actions, systemd-oomd will monitor the system-wide swap > space and act when available swap falls below the configured > threshold, starting with the cgroups with the highest swap usage to > the least. Keeping some amount of swap (if enabled) available will > prevent the kernel OOM killer from killing processes unpredictably and > spending an unbounded amount of time afterwards. > > For swap configuration, this will be SwapUsedLimitPercent=90% in > oomd.conf and ManagedOOMSwap=kill on -.slice (root cgroup slice) to > have systemd-oomd send SIGKILLs to all processes under a cgroup when > swap used exceeds 90%. > > > == Benefit to Fedora == > > * Addressing the issue of improving user feedback in > https://pagure.io/fedora-workstation/issue/202, systemd-oomd currently > logs to the journal if pressure or swap action is about to occur. > There are also debug logs, for each process that is sent a SIGKILL, > that can be bumped up in priority. Further notification mechanisms > (i.e. over dbus) can also be implemented depending on feedback. > * While systemd-oomd is simpler in configuration to the oomd used at > Facebook, the algorithm is largely the same. As such, the following > case study can be used as an example of how PSI and cgroup killing can > release memory not normally resolved with process killing and lead to > better utilization: > https://facebookincubator.github.io/oomd/docs/oomd-casestudy.html > * OOM killing in userspace, before the kernel OOM killer kicks in, has > been shown to be effective at keeping a system functional. An OOM kill > in the kernel is slow, possibly leading to an unbounded amount of time > swapping in and out pages and evicting the page cache. > * PSI based actions, versus looking at raw memory consumption numbers, > better reflect memory protection policies set for cgroup resource > control limits (e.g. memory.low). > > == Scope == > > * Proposal owners: > ** Implement and land additional refinements to systemd-oomd > *** Remove swap as a hard requirement to running systemd-oomd > *** Expand ManagedOOM*= properties to user units (currently only > usable on system units) > *** Configurable memory pressure time window knob > ** Enable oomd by default with sensible configuration > ** Test days > ** Aid with documentation > * Other developers: > ** systemd: review PRs as needed > * Release engineering: https://pagure.io/releng/issue/9913 > * Policies and guidelines: N/A > * Trademark approval: N/A > > == Upgrade/compatibility impact == > > Existing systems running earlyoom will not be modified. One can > transition to systemd-oomd via: > > sudo systemctl disable --now earlyoom > sudo systemctl enable --now systemd-oomd > Systems that were previously not running earlyoom will have > systemd-oomd enabled by default. > > == How to test == > > systemd 247 build for Fedora includes all the artifacts for > systemd-oomd. It is disabled by default but can be started with: > > sudo
Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
https://fedoraproject.org/wiki/Changes/EnableSystemdOomd == Summary == Provide a better experience for Fedora users in out-of-memory (OOM) situations by enabling [https://www.freedesktop.org/software/systemd/man/systemd-oomd.html systemd-oomd] by default. Actions taken by systemd-oomd operate on a per-cgroup level, aligning well with the life cycle of systemd units. systemd-oomd primarily uses [https://facebookmicrosites.github.io/psi/ Linux pressure stall information (PSI)] to make decisions based on wasted productivity due to resource shortages; in addition to that, it also supports swap based actions. == Owners == * Name: [[User:anitazha|Anita Zhang]], [[User:Dcavalca|Davide Cavalca]], [[User:Salimma|Michel Salim]], [[User:Htejun|Tejun Heo]], [[User:3ki|Rik van Riel]] * Email: the.anita...@gmail.com, dcava...@fb.com, mic...@michel-slm.name, hte...@fb.com, r...@fb.com == Detailed description == The primary mechanism used by systemd-oomd for detecting when the system is out of memory is memory pressure. Memory pressure measures the percentage of time a cgroup has “wasted” due to lack of memory. This includes time spent reclaiming free memory, faulting in recently resident pages, and loading in anonymous pages from swap. When a monitored cgroup’s memory pressure exceeds the specified thresholds, systemd-oomd will perform action(s) on the targeted cgroup’s descendants, starting from the cgroups with the most reclaim scans. Reclaim activity is used here, rather than the largest consumer, as it reflects values set in the cgroup memory controller for memory protection (such as memory.low). For memory pressure configuration, this will be ManagedOOMMemoryPressure=kill and ManagedOOMMemoryPressureLimit=4% on user@.service to have systemd-oomd send SIGKILLs to all processes under a selected cgroup when total memory pressure on all tasks exceeds 4% for 10 seconds. For swap based actions, systemd-oomd will monitor the system-wide swap space and act when available swap falls below the configured threshold, starting with the cgroups with the highest swap usage to the least. Keeping some amount of swap (if enabled) available will prevent the kernel OOM killer from killing processes unpredictably and spending an unbounded amount of time afterwards. For swap configuration, this will be SwapUsedLimitPercent=90% in oomd.conf and ManagedOOMSwap=kill on -.slice (root cgroup slice) to have systemd-oomd send SIGKILLs to all processes under a cgroup when swap used exceeds 90%. == Benefit to Fedora == * Addressing the issue of improving user feedback in https://pagure.io/fedora-workstation/issue/202, systemd-oomd currently logs to the journal if pressure or swap action is about to occur. There are also debug logs, for each process that is sent a SIGKILL, that can be bumped up in priority. Further notification mechanisms (i.e. over dbus) can also be implemented depending on feedback. * While systemd-oomd is simpler in configuration to the oomd used at Facebook, the algorithm is largely the same. As such, the following case study can be used as an example of how PSI and cgroup killing can release memory not normally resolved with process killing and lead to better utilization: https://facebookincubator.github.io/oomd/docs/oomd-casestudy.html * OOM killing in userspace, before the kernel OOM killer kicks in, has been shown to be effective at keeping a system functional. An OOM kill in the kernel is slow, possibly leading to an unbounded amount of time swapping in and out pages and evicting the page cache. * PSI based actions, versus looking at raw memory consumption numbers, better reflect memory protection policies set for cgroup resource control limits (e.g. memory.low). == Scope == * Proposal owners: ** Implement and land additional refinements to systemd-oomd *** Remove swap as a hard requirement to running systemd-oomd *** Expand ManagedOOM*= properties to user units (currently only usable on system units) *** Configurable memory pressure time window knob ** Enable oomd by default with sensible configuration ** Test days ** Aid with documentation * Other developers: ** systemd: review PRs as needed * Release engineering: https://pagure.io/releng/issue/9913 * Policies and guidelines: N/A * Trademark approval: N/A == Upgrade/compatibility impact == Existing systems running earlyoom will not be modified. One can transition to systemd-oomd via: sudo systemctl disable --now earlyoom sudo systemctl enable --now systemd-oomd Systems that were previously not running earlyoom will have systemd-oomd enabled by default. == How to test == systemd 247 build for Fedora includes all the artifacts for systemd-oomd. It is disabled by default but can be started with: sudo systemctl enable --now systemd-oomd At this point you can decide which units to set properties on. For example, to enable swap-based killing on all units below the root slice: sudo systemctl edit --force -- -.slice [Slice] ManagedOOMSwap=kill # save and
Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
https://fedoraproject.org/wiki/Changes/EnableSystemdOomd == Summary == Provide a better experience for Fedora users in out-of-memory (OOM) situations by enabling [https://www.freedesktop.org/software/systemd/man/systemd-oomd.html systemd-oomd] by default. Actions taken by systemd-oomd operate on a per-cgroup level, aligning well with the life cycle of systemd units. systemd-oomd primarily uses [https://facebookmicrosites.github.io/psi/ Linux pressure stall information (PSI)] to make decisions based on wasted productivity due to resource shortages; in addition to that, it also supports swap based actions. == Owners == * Name: [[User:anitazha|Anita Zhang]], [[User:Dcavalca|Davide Cavalca]], [[User:Salimma|Michel Salim]], [[User:Htejun|Tejun Heo]], [[User:3ki|Rik van Riel]] * Email: the.anita...@gmail.com, dcava...@fb.com, mic...@michel-slm.name, hte...@fb.com, r...@fb.com == Detailed description == The primary mechanism used by systemd-oomd for detecting when the system is out of memory is memory pressure. Memory pressure measures the percentage of time a cgroup has “wasted” due to lack of memory. This includes time spent reclaiming free memory, faulting in recently resident pages, and loading in anonymous pages from swap. When a monitored cgroup’s memory pressure exceeds the specified thresholds, systemd-oomd will perform action(s) on the targeted cgroup’s descendants, starting from the cgroups with the most reclaim scans. Reclaim activity is used here, rather than the largest consumer, as it reflects values set in the cgroup memory controller for memory protection (such as memory.low). For memory pressure configuration, this will be ManagedOOMMemoryPressure=kill and ManagedOOMMemoryPressureLimit=4% on user@.service to have systemd-oomd send SIGKILLs to all processes under a selected cgroup when total memory pressure on all tasks exceeds 4% for 10 seconds. For swap based actions, systemd-oomd will monitor the system-wide swap space and act when available swap falls below the configured threshold, starting with the cgroups with the highest swap usage to the least. Keeping some amount of swap (if enabled) available will prevent the kernel OOM killer from killing processes unpredictably and spending an unbounded amount of time afterwards. For swap configuration, this will be SwapUsedLimitPercent=90% in oomd.conf and ManagedOOMSwap=kill on -.slice (root cgroup slice) to have systemd-oomd send SIGKILLs to all processes under a cgroup when swap used exceeds 90%. == Benefit to Fedora == * Addressing the issue of improving user feedback in https://pagure.io/fedora-workstation/issue/202, systemd-oomd currently logs to the journal if pressure or swap action is about to occur. There are also debug logs, for each process that is sent a SIGKILL, that can be bumped up in priority. Further notification mechanisms (i.e. over dbus) can also be implemented depending on feedback. * While systemd-oomd is simpler in configuration to the oomd used at Facebook, the algorithm is largely the same. As such, the following case study can be used as an example of how PSI and cgroup killing can release memory not normally resolved with process killing and lead to better utilization: https://facebookincubator.github.io/oomd/docs/oomd-casestudy.html * OOM killing in userspace, before the kernel OOM killer kicks in, has been shown to be effective at keeping a system functional. An OOM kill in the kernel is slow, possibly leading to an unbounded amount of time swapping in and out pages and evicting the page cache. * PSI based actions, versus looking at raw memory consumption numbers, better reflect memory protection policies set for cgroup resource control limits (e.g. memory.low). == Scope == * Proposal owners: ** Implement and land additional refinements to systemd-oomd *** Remove swap as a hard requirement to running systemd-oomd *** Expand ManagedOOM*= properties to user units (currently only usable on system units) *** Configurable memory pressure time window knob ** Enable oomd by default with sensible configuration ** Test days ** Aid with documentation * Other developers: ** systemd: review PRs as needed * Release engineering: https://pagure.io/releng/issue/9913 * Policies and guidelines: N/A * Trademark approval: N/A == Upgrade/compatibility impact == Existing systems running earlyoom will not be modified. One can transition to systemd-oomd via: sudo systemctl disable --now earlyoom sudo systemctl enable --now systemd-oomd Systems that were previously not running earlyoom will have systemd-oomd enabled by default. == How to test == systemd 247 build for Fedora includes all the artifacts for systemd-oomd. It is disabled by default but can be started with: sudo systemctl enable --now systemd-oomd At this point you can decide which units to set properties on. For example, to enable swap-based killing on all units below the root slice: sudo systemctl edit --force -- -.slice [Slice] ManagedOOMSwap=kill # save and
