Re: [systemd-devel] journald deleting logs on LiveOS boots
On Wed, Jun 19, 2019 at 6:15 AM Lennart Poettering wrote: > > On Di, 18.06.19 20:34, Chris Murphy (li...@colorremedies.com) wrote: > > > When I boot Fedora 30/Rawhide Workstation (LiveOS install media) in a > > VM with ~2GiB memory, using any combination of systemd, dracut, or > > udev debugging, events are missing from the journal. > > > > systemd-241-7.gita2eaa1c.fc30.x86_64 > > systemd-242-3.git7a6d834.fc31.x86_64 > > > > 'journalctl -b -o short-monotonic' starts at ~22s monotonic time. i.e. > > all the messages before that are not in the journal. This problem > > doesn't happen with netinstalls. Fedora LiveOS boots setup a memory > > based persistent overlay, and /var/log/journal exists and > > what do you mean by "memory based persistent overlay"? if its in > memory it's not persistant, is it? /me is confused... I agree it is an oxymoron. The device-mapper rw ext4 volume where all writes go is memory based, as the install media and the base image are read-only. Since this volume is rw, and it's mounted at /, that makes /var/log/journal rw, and appears to be persistent from the systemd-journald point of view, so it uses it instead of /run/log/journal [9.127691] localhost systemd[1]: Starting Flush Journal to Persistent Storage... > > is this LVM stuff or overlayfs? Neither, only device mapper. [root@localhost-live liveuser]# lvs [root@localhost-live liveuser]# vgs [root@localhost-live liveuser]# pvs [root@localhost-live liveuser]# dmsetup status live-base: 0 13635584 linear live-rw: 0 13635584 snapshot 163328/67108864 648 [root@localhost-live liveuser]# All of this is setup by dracut with the boot parameter 'rd.live.image' > > > systemd-journald tries to flush there, whereas on netinstalls > > /var/log/journal does not exist. > > > > Using systemd.log_target=kmsg I discovered that systemd-journald is > > deleting logs in the LiveOS case, but I don't know why, / has ~750M > > free > > > > [ 24.910792] systemd-journald[922]: Vacuuming... > > [ 24.921802] systemd-journald[922]: Deleted archived journal > > /var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee/system@818f3f40f19849e08a1b37b9c1e304f1-0001-00058ba31bec725e.journal > > (8.0M). > > [ 24.921808] systemd-journald[922]: Vacuuming done, freed 8.0M of > > archived journals from > > /var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee. > > > > I filed a bug here and have nominated it as a blocker bug for the > > F31 cycle. > > Note that journald logs early on how much disk space it assumes to > have free, you might see that in dmesg? (if not, boot with > systemd.journald.forward_to_kmsg=1 on the kernel cmdline) Using systemd.log_level=debug and systemd.journald.forward_to_kmsg=1 I only see: [4.224599] systemd-journald[327]: Fixed min_use=1.0M max_use=145.8M max_size=18.2M min_size=512.0K keep_free=218.7M n_max_files=100 If I boot with defaults (no debug options), I now see the following, which is otherwise lost (not in the journal and not in dmesg, despite systemd.journald.forward_to_kmsg=1) [9.124045] localhost systemd-journald[933]: Runtime journal (/run/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee) is 8.0M, max 146.6M, 138.6M free. ...snip... [9.179050] localhost systemd-journald[933]: Time spent on flushing to /var is 17.824ms for 936 entries. [9.179050] localhost systemd-journald[933]: System journal (/var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee) is 8.0M, max 8.0M, 0B free. Why does it say 0B free? [root@localhost-live liveuser]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 1.5G 0 1.5G 0% /dev tmpfs1.5G 4.0K 1.5G 1% /dev/shm tmpfs1.5G 1.1M 1.5G 1% /run tmpfs1.5G 0 1.5G 0% /sys/fs/cgroup /dev/sr0 1.9G 1.9G 0 100% /run/initramfs/live /dev/mapper/live-rw 6.4G 5.5G 879M 87% / tmpfs1.5G 4.0K 1.5G 1% /tmp vartmp 1.5G 0 1.5G 0% /var/tmp tmpfs294M 0 294M 0% /run/user/0 tmpfs294M 4.0K 294M 1% /run/user/1000 [root@localhost-live liveuser]# free -m totalusedfree shared buff/cache available Mem: 2932 2231754 1 9542435 Swap: 0 0 0 [root@localhost-live liveuser]# cat /etc/systemd/journald.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See journald.conf(5) for details. [Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitIntervalSec=30
[systemd-devel] Delegate v1 cgroup controller permissions
I have a service which runs as an unprivileged user (User=foo) with delegated cgroup (Delegate=true) that wants to use the "memory" and "cpu" controllers. Systemd is using the hybrid mode with both v1 and v2 cgroups, and the controllers are assigned to the v1 groups. Before I can use the "cpu" or "memory" cgroups I have to force the permissions of them because the delegated permissions are only applied in the unified hierarchy. Doing this requires root which is a problem because we don't want to give this service root permissions. I have read https://systemd.io/CGROUP_DELEGATION and note that the hybrid mode "is a stopgap" and "has no future" but I am forced to use it because the distros that we have to use (fedora) are set up that way (I have yet to see any system use the unified v2 mode exclusively). So I'm having to bother with hybrid mode even though I don't have enough free time ;) I have read in the same article that delegation "won’t pass ownership of the legacy controller hierarchies" and "think twice before delegating cgroup v1 controllers to less privileged containers." I get that it isn't the preferred mechanism with systemd but we just want to manage access to resources (cpu and memory) allocated to subtasks from within our application. So is there a way to tell systemd (or some other way) to set the v1 cgroup permissions so they are usable by the delegated user without having to give the user process root privileges ? pEpkey.asc Description: application/pgp-keys ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Unset option in drop-in override
On Wed, Jun 19, 2019, at 6:19 AM, Lennart Poettering wrote: > On Di, 18.06.19 22:05, Dominick C. Pastore (dominickpast...@dcpx.org) wrote: > > > Hello, > > > > I think this is an easy question but I can't find the answer > > anywhere. > > Which version of systemd are you using? Version 232. > > I'm having some trouble with a unit file drop-in overrride. I have a > > package that uses some options that are incompatible with the LXD container > > I'm running it in. Specifically, the following two options: > > > > [Service] > > IOSchedulingClass=realtime > > CPUSchedulingPolicy=rr > > > > So I used "systemd edit" to try and override these using the following: > > > > [Service] > > IOSchedulingClass= > > CPUSchedulingPolicy= > > > > The problem is, then I get the messages "Failed to parse IO > > scheduling class, ignoring: " and "Failed to parse CPU scheduling > > policy, ignoring: " in the journal. > > In current versions (since 239) this should work fine. Please upgrade > (or ask your distro to backport > 617d253afad10ca9c7861eb3a2b381a8f95cf4ad. > > You might be able to do the following though: > >IOSchedulingClass=be >CPUSchedulingPolicy=other > > which are the default values. I didn't test that though, I am not sure > if the kernel will let you set the parameters to the values already > set without complaining. Thanks, that's good to know. I'll give the default values a try first and switch to the backport if it still complains. > Alternatively, just copy the unit file from /usr/lib to /etc, and > comment the relevant lines. That way you override the whole file. This is indeed what I am currently doing, but I'm concerned it will mask an update to the original unit file. I'll give the other suggestions a try. Thanks for the advice. Dominick ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] journald deleting logs on LiveOS boots
On Di, 18.06.19 20:34, Chris Murphy (li...@colorremedies.com) wrote: > When I boot Fedora 30/Rawhide Workstation (LiveOS install media) in a > VM with ~2GiB memory, using any combination of systemd, dracut, or > udev debugging, events are missing from the journal. > > systemd-241-7.gita2eaa1c.fc30.x86_64 > systemd-242-3.git7a6d834.fc31.x86_64 > > 'journalctl -b -o short-monotonic' starts at ~22s monotonic time. i.e. > all the messages before that are not in the journal. This problem > doesn't happen with netinstalls. Fedora LiveOS boots setup a memory > based persistent overlay, and /var/log/journal exists and what do you mean by "memory based persistent overlay"? if its in memory it's not persistant, is it? /me is confused... is this LVM stuff or overlayfs? > systemd-journald tries to flush there, whereas on netinstalls > /var/log/journal does not exist. > > Using systemd.log_target=kmsg I discovered that systemd-journald is > deleting logs in the LiveOS case, but I don't know why, / has ~750M > free > > [ 24.910792] systemd-journald[922]: Vacuuming... > [ 24.921802] systemd-journald[922]: Deleted archived journal > /var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee/system@818f3f40f19849e08a1b37b9c1e304f1-0001-00058ba31bec725e.journal > (8.0M). > [ 24.921808] systemd-journald[922]: Vacuuming done, freed 8.0M of > archived journals from > /var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee. > > I filed a bug here and have nominated it as a blocker bug for the > F31 cycle. Note that journald logs early on how much disk space it assumes to have free, you might see that in dmesg? (if not, boot with systemd.journald.forward_to_kmsg=1 on the kernel cmdline) > A related question is whether it's even appropriate or > /var/log/journal to exist on LiveOS boots, rather than just have > journals go to /run/log/journal? Obviously we must have the entire > journal for LiveOS boots, from monotonic time 0.0, no matter the debug > options chosen, I'd say for at least 5 minutes? Almost immediately > dropping the first 30s just because I've got debug options set is a > big problem for troubleshooting other problems with early startup of > install images. There is not much point in using /var if /run works too and /var isn't persistent anyway... Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Unset option in drop-in override
On Di, 18.06.19 22:05, Dominick C. Pastore (dominickpast...@dcpx.org) wrote: > Hello, > > I think this is an easy question but I can't find the answer > anywhere. Which version of systemd are you using? > > I'm having some trouble with a unit file drop-in overrride. I have a package > that uses some options that are incompatible with the LXD container I'm > running it in. Specifically, the following two options: > > [Service] > IOSchedulingClass=realtime > CPUSchedulingPolicy=rr > > So I used "systemd edit" to try and override these using the following: > > [Service] > IOSchedulingClass= > CPUSchedulingPolicy= > > The problem is, then I get the messages "Failed to parse IO > scheduling class, ignoring: " and "Failed to parse CPU scheduling > policy, ignoring: " in the journal. In current versions (since 239) this should work fine. Please upgrade (or ask your distro to backport 617d253afad10ca9c7861eb3a2b381a8f95cf4ad. You might be able to do the following though: IOSchedulingClass=be CPUSchedulingPolicy=other which are the default values. I didn't test that though, I am not sure if the kernel will let you set the parameters to the values already set without complaining. Alternatively, just copy the unit file from /usr/lib to /etc, and comment the relevant lines. That way you override the whole file. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Static IP address on wandering Wi-Fi client
Hi Marcel On 6/19/19 5:14 AM, Marcel Holtmann wrote: Hi Johann, And frankly IP configuration needs to move into the network technology daemons like iwd for example. What's the argument here for that reasoning as in why not consolidate all network configuration ( ethernet/wifi/vrrp/vpn's etc.. ) to a single place ( networkd )? Why do you want to keep the network configuration and management space fragmented ( as opposed to consolidated ) ? because that is what the standards are doing. WiFi for example can transport the IP address inline. You just need to confirm it with DHCP later. It is how the standards are written that you want to really move DHCP one level down. The same applies to cellular connections where all the IP level configuration comes via the cellular network and not DHCP. I see that's what the standard specifies but why not merge IWD/EAD with systemd ( systemd-wirelessd/systemd-etherauthd and or tightly integrate it with networkd )? What are the benefits you see keeping these as separated components as opposed to be part of the service management framework for consolidated core/baseOS The IWD/EAD are Linux only projects right ( as in it's not going to be replacing wpa supplicant on other *nix like for example the BSD stack )? Regards Jóhann B. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Allocating resource to achieve predictable run times
On Mi, 19.06.19 09:25, John Lane (j...@lane.uk.net) wrote: > Well I got to the bottom of it and wanted to close this out. I was > prototyping on a laptop and the cpu (i7-7820HQ) did funky things to save > power. I tried setting the performance governor but it didn't help. > > I ran the exact same tests on some servers and got the expected > behaviour on all of them. > > Regarding isolcpus, I did try that, although I noted from the kernel > docs that it is a deprecated option and it suggests that cpuset should > be used instead. But I don't believe systemd (yet) supports cpuset? cpuset became a cgroups feature a long time ago, but it's really really awful semanticswise in cgroupsv1. In cgroupsv2 this is being worked on and should be fixed. As soon as that's done we'll support it. That said, I don't really see how cpuset could ever provide what isolcpus does, i.e. control what cpus the kernel's own threads are scheduled on. I mean, cpuset is inherentlich an "a-posteriori" thing: to schedule something on a cpu you need its pid, but kernel threads come and go, and for those you really want to configure the cpuset mask "a-priori"... So I must say I don't understand the comment in the docs really, I don't see how cpuset and isolcpus could be equivalent. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
