Re: [systemd-devel] [EXT] Re: consider dropping defrag of journals on btrfs
On Tue, Feb 9, 2021 at 8:02 AM Phillip Susi wrote: > > > Chris Murphy writes: > > > Basically correct. It will merge random writes such that they become > > sequential writes. But it means inserts/appends/overwrites for a file > > won't be located with the original extents. > > Wait, I thoguht that was only true for metadata, not normal file data > blocks? Well, maybe it becomes true for normal data if you enable > compression. Or small files that get leaf packed into the metadata > chunk. Both data and metadata. > > If it's really combining streaming writes from two different files into > a single interleaved write to the disk, that would be really silly. It's not interleaving. It uses delayed allocation to make random writes into sequential writes. It's tries harder to keep file blocks together for the nossd case. It's a bit more opportunistic with the ssd mount option. And it also depends on the pattern of the writer. There's btrfs heatmap to get a visual idea of these behaviors. https://github.com/knorrie/btrfs-heatmap -- Chris Murphy ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@fuse.service is masked
On Tue, Feb 9, 2021 at 7:19 PM Mike Gilbert wrote: > > On Tue, Feb 9, 2021 at 7:05 PM Reindl Harald wrote: > > > > > > > > Am 09.02.21 um 23:18 schrieb Mike Gilbert: > > > On Tue, Feb 9, 2021 at 11:47 AM Reindl Harald > > > wrote: > > >> > > >> > > >> > > >> Am 09.02.21 um 17:13 schrieb Mike Gilbert: > > >>> On Tue, Feb 9, 2021 at 6:17 AM Reindl Harald > > >>> wrote: > > > > > > > > Am 08.02.21 um 23:42 schrieb Mike Gilbert: > > > On Mon, Feb 8, 2021 at 2:31 PM Reindl Harald > > > wrote: > > >>> I think removing this symlink would prevent /sys/fs/fuse/connections > > >>> from being mounted and the fuse module from being loaded > > >>> unconditionally on boot > > >> > > >> no > > >> > > >> https://bugzilla.redhat.com/show_bug.cgi?id=1909805#c6 > > > > > > It almost works for me on Gentoo Linux. > > > To test, I first had to reconfigure my kernel to build FUSE as a > > > module (I normally have it built-in). > > > I then removed the sys-fs-fuse-connections.mount symlink from > > > sysinit.target.wants. > > > After rebooting with the new kernel, the FUSE module is not loaded and > > > /sys/fs/fuse/connections is not mounted. > > > > > > Unfortunately, mounting FUSE-based file systems does not work until I > > > manually run "modprobe fuse". > > > It seems that my kernel does not auto-load the module, despite the > > > static /dev/fuse node. The kernel is probably missing a call to > > > __request_module(). > > > > > > Given that the kernel doesn't auto-load the module on demand, leaving > > > the sysinit.target.wants symlink in place seems like the safe thing to > > > do. > > > > but for sure not on a stripped down machine running a iptables-nft > > ruleset, a socket-activated sshd and nohting else > > > > if it's me for server setups the "fuse" kernel-module could be in > > "kernel-modules" which is not installed and needed for virtualized > > guests > > > > the point is that all this setups where happy without fuse loaded from > > 2008 to 2021 and you can't even avoid it with F33 at all, no matter > > what > > you delete or mask > > > > a active masked unit - seriously? :-) > > > > [root@rawhide ~]# systemctl status sys-module-fuse.device > > sys-fs-fuse-connections.mount > > ● sys-module-fuse.device - /sys/module/fuse > > Loaded: masked (Reason: Unit sys-module-fuse.device is masked.) > > Active: active (plugged) since Mon 2021-02-08 19:33:18 CET; > > 1min > > 42s ago > > Device: /sys/module/fuse > > >>> > > >>> I think something else on your system is loading the fuse kernel > > >>> module, which activates sys-module-fuse.device, and tries to start > > >>> sys-fs-fuse-connections.mount. It appears systemd doesn't really > > >>> support masking device units, which are generated by udev events. > > >>> > > >>> You should probably try to track down exactly what else is loading the > > >>> fuse module, and disable that. > > >> > > >> this is a bare setup with *nothing* enabled at all > > > > > > Off the top of my head, maybe fuse is getting loaded by an entry in > > > modules-load.d. > > > > no > > > > [root@rawhide ~]# ls /etc/modules-load.d/ > > total 0 > > > > > Also, vmware tools might utilize FUSE in some way. > > > > no > > > > [root@rawhide ~]# system-errors.sh > > Feb 10 00:59:22 rawhide systemd[1]: sys-module-fuse.device: Failed to > > enqueue SYSTEMD_WANTS= job, ignoring: Unit sys-fs-fuse-connections.mount > > is masked. > > [root@rawhide ~]# systemctl status vmtoolsd.service > > ● vmtoolsd.service - VMware Tools > > Loaded: loaded (/etc/systemd/system/vmtoolsd.service; disabled; > > vendor preset: enabled) > > Active: inactive (dead) > > > > even that file from the vmtools package was deleted long before my > > initial post of this thread > > > > [root@rawhide ~]# cat /usr/lib/systemd/system/run-vmblock\x2dfuse.mount > > cat: /usr/lib/systemd/system/run-vmblockx2dfuse.mount: No such file or > > directory > > > > > If you're unable to figure out what is loading it, you might replace > > > /sbin/modprobe with a wrapper script to log all processes that call > > > it > > there is nothing left but systemd which also don't go the normal way > > otherwise this below would prevent loading the kernel module > > > > modprobe won't load it in that case > > > > [root@rawhide ~]# cat /etc/modprobe.d/blacklist-lounge-vm.conf | grep fuse > > blacklist fuse > > install fuse /usr/bin/true > > > > The blacklist is only applied if you call "modprobe -b". Possibly > something else is calling modprobe without the -b option. Also, it might be loaded by something in the initramfs (assuming you have one). Anyway, I think I will stop spoon feeding you ideas at this point. I'm sure you are capable of poking around files on your own syste
Re: [systemd-devel] sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@fuse.service is masked
On Tue, Feb 9, 2021 at 7:05 PM Reindl Harald wrote: > > > > Am 09.02.21 um 23:18 schrieb Mike Gilbert: > > On Tue, Feb 9, 2021 at 11:47 AM Reindl Harald > > wrote: > >> > >> > >> > >> Am 09.02.21 um 17:13 schrieb Mike Gilbert: > >>> On Tue, Feb 9, 2021 at 6:17 AM Reindl Harald > >>> wrote: > > > > Am 08.02.21 um 23:42 schrieb Mike Gilbert: > > On Mon, Feb 8, 2021 at 2:31 PM Reindl Harald > > wrote: > >>> I think removing this symlink would prevent /sys/fs/fuse/connections > >>> from being mounted and the fuse module from being loaded > >>> unconditionally on boot > >> > >> no > >> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1909805#c6 > > > > It almost works for me on Gentoo Linux. > > To test, I first had to reconfigure my kernel to build FUSE as a > > module (I normally have it built-in). > > I then removed the sys-fs-fuse-connections.mount symlink from > > sysinit.target.wants. > > After rebooting with the new kernel, the FUSE module is not loaded and > > /sys/fs/fuse/connections is not mounted. > > > > Unfortunately, mounting FUSE-based file systems does not work until I > > manually run "modprobe fuse". > > It seems that my kernel does not auto-load the module, despite the > > static /dev/fuse node. The kernel is probably missing a call to > > __request_module(). > > > > Given that the kernel doesn't auto-load the module on demand, leaving > > the sysinit.target.wants symlink in place seems like the safe thing to > > do. > > but for sure not on a stripped down machine running a iptables-nft > ruleset, a socket-activated sshd and nohting else > > if it's me for server setups the "fuse" kernel-module could be in > "kernel-modules" which is not installed and needed for virtualized guests > > the point is that all this setups where happy without fuse loaded from > 2008 to 2021 and you can't even avoid it with F33 at all, no matter what > you delete or mask > > a active masked unit - seriously? :-) > > [root@rawhide ~]# systemctl status sys-module-fuse.device > sys-fs-fuse-connections.mount > ● sys-module-fuse.device - /sys/module/fuse > Loaded: masked (Reason: Unit sys-module-fuse.device is masked.) > Active: active (plugged) since Mon 2021-02-08 19:33:18 CET; 1min > 42s ago > Device: /sys/module/fuse > >>> > >>> I think something else on your system is loading the fuse kernel > >>> module, which activates sys-module-fuse.device, and tries to start > >>> sys-fs-fuse-connections.mount. It appears systemd doesn't really > >>> support masking device units, which are generated by udev events. > >>> > >>> You should probably try to track down exactly what else is loading the > >>> fuse module, and disable that. > >> > >> this is a bare setup with *nothing* enabled at all > > > > Off the top of my head, maybe fuse is getting loaded by an entry in > > modules-load.d. > > no > > [root@rawhide ~]# ls /etc/modules-load.d/ > total 0 > > > Also, vmware tools might utilize FUSE in some way. > > no > > [root@rawhide ~]# system-errors.sh > Feb 10 00:59:22 rawhide systemd[1]: sys-module-fuse.device: Failed to > enqueue SYSTEMD_WANTS= job, ignoring: Unit sys-fs-fuse-connections.mount > is masked. > [root@rawhide ~]# systemctl status vmtoolsd.service > ● vmtoolsd.service - VMware Tools > Loaded: loaded (/etc/systemd/system/vmtoolsd.service; disabled; > vendor preset: enabled) > Active: inactive (dead) > > even that file from the vmtools package was deleted long before my > initial post of this thread > > [root@rawhide ~]# cat /usr/lib/systemd/system/run-vmblock\x2dfuse.mount > cat: /usr/lib/systemd/system/run-vmblockx2dfuse.mount: No such file or > directory > > > If you're unable to figure out what is loading it, you might replace > > /sbin/modprobe with a wrapper script to log all processes that call > > it > there is nothing left but systemd which also don't go the normal way > otherwise this below would prevent loading the kernel module > > modprobe won't load it in that case > > [root@rawhide ~]# cat /etc/modprobe.d/blacklist-lounge-vm.conf | grep fuse > blacklist fuse > install fuse /usr/bin/true > The blacklist is only applied if you call "modprobe -b". Possibly something else is calling modprobe without the -b option. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@fuse.service is masked
On Tue, Feb 9, 2021 at 11:47 AM Reindl Harald wrote: > > > > Am 09.02.21 um 17:13 schrieb Mike Gilbert: > > On Tue, Feb 9, 2021 at 6:17 AM Reindl Harald wrote: > >> > >> > >> > >> Am 08.02.21 um 23:42 schrieb Mike Gilbert: > >>> On Mon, Feb 8, 2021 at 2:31 PM Reindl Harald > >>> wrote: > > I think removing this symlink would prevent /sys/fs/fuse/connections > > from being mounted and the fuse module from being loaded > > unconditionally on boot > > no > > https://bugzilla.redhat.com/show_bug.cgi?id=1909805#c6 > >>> > >>> It almost works for me on Gentoo Linux. > >>> To test, I first had to reconfigure my kernel to build FUSE as a > >>> module (I normally have it built-in). > >>> I then removed the sys-fs-fuse-connections.mount symlink from > >>> sysinit.target.wants. > >>> After rebooting with the new kernel, the FUSE module is not loaded and > >>> /sys/fs/fuse/connections is not mounted. > >>> > >>> Unfortunately, mounting FUSE-based file systems does not work until I > >>> manually run "modprobe fuse". > >>> It seems that my kernel does not auto-load the module, despite the > >>> static /dev/fuse node. The kernel is probably missing a call to > >>> __request_module(). > >>> > >>> Given that the kernel doesn't auto-load the module on demand, leaving > >>> the sysinit.target.wants symlink in place seems like the safe thing to > >>> do. > >> > >> but for sure not on a stripped down machine running a iptables-nft > >> ruleset, a socket-activated sshd and nohting else > >> > >> if it's me for server setups the "fuse" kernel-module could be in > >> "kernel-modules" which is not installed and needed for virtualized guests > >> > >> the point is that all this setups where happy without fuse loaded from > >> 2008 to 2021 and you can't even avoid it with F33 at all, no matter what > >> you delete or mask > >> > >> a active masked unit - seriously? :-) > >> > >> [root@rawhide ~]# systemctl status sys-module-fuse.device > >> sys-fs-fuse-connections.mount > >> ● sys-module-fuse.device - /sys/module/fuse > >>Loaded: masked (Reason: Unit sys-module-fuse.device is masked.) > >>Active: active (plugged) since Mon 2021-02-08 19:33:18 CET; 1min > >> 42s ago > >>Device: /sys/module/fuse > > > > I think something else on your system is loading the fuse kernel > > module, which activates sys-module-fuse.device, and tries to start > > sys-fs-fuse-connections.mount. It appears systemd doesn't really > > support masking device units, which are generated by udev events. > > > > You should probably try to track down exactly what else is loading the > > fuse module, and disable that. > > this is a bare setup with *nothing* enabled at all Off the top of my head, maybe fuse is getting loaded by an entry in modules-load.d. Also, vmware tools might utilize FUSE in some way. If you're unable to figure out what is loading it, you might replace /sbin/modprobe with a wrapper script to log all processes that call it. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Should services be able to run without /proc?
Hi!
So this is kind of a follow-up to the thread in [1], and the
corresponding PR in [2].
In short, the PR made some changes to allow for cases where /proc was
not available in the mount namespace of the service, and added a test
[3] to make sure that this would work. This test was later removed and
rewritten to block /sys instead [4], because it turned out that having
/proc unavailable sometimes caused problems with close_all_fds(), which
is called in exec_child() after namespaces have been set up.
On current master, services that don't have /proc mounted don't work at
all anymore, since find_executable_full() ends up opening the given path
and calling access_fd() on the resulting fd, and access_fd uses
/proc/self/fd/* to turn the fd back into a path it can call access() on.
As far as I can tell, the reason for not using access on the path
directly is that access_fd is more elegant since it avoids a potential
race condition.
In addition to this, setup_private_users() also needs access to
/proc/$pid/{uid_map, gid_map, setgroups} to do its job.
Given all this, I guess my question is whether it is still desirable to
allow units to run without /proc, especially given that ProtectProc and
ProcSubset exist now.* If not, it might be nice to just always mount
/proc if it wouldn't otherwise be there (i.e. if RootImage/RootDirectory
is used); currently, MountAPIVFS=yes is basically a required option
because of this. (I guess you could mount proc manually, but then you
can't use ProtectProc/ProcSubset.) I'm a bit unhappy about this, because
MountAPIVFS also mounts /sys and /dev, and then you need separate
options just to protect those again. Either way, maybe it would be good
to explicitly state this requirement in the documentation?
Anyway, I hope that this was okay to post here, I don't really know a
lot about this and maybe there are good reasons for why things are the
way they are. I'd be happy about feedback though.
Cheers,
Antonius
* Using both ProtectProc=ptraceable and ProcSubset=pid really doesn't
let a lot of things through, and I don't think those interfere with any
of the functions described above. The only thing I'm unsure about is
setup_private_users(), since that spawns off a child process which then
goes and writes to /proc/$parent_pid/, but I guess children can ptrace
their parents? At least it seemed to work when I just tested it.
[1]:
https://lists.freedesktop.org/archives/systemd-devel/2017-April/038634.html
[2]: https://github.com/systemd/systemd/pull/5985
[3]: https://github.com/systemd/systemd/pull/6017
[4]:
https://github.com/systemd/systemd/commit/054d871d41039fcfc1a4a661c979941b9660c9e6
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@fuse.service is masked
On Tue, Feb 9, 2021 at 6:17 AM Reindl Harald wrote: > > > > Am 08.02.21 um 23:42 schrieb Mike Gilbert: > > On Mon, Feb 8, 2021 at 2:31 PM Reindl Harald wrote: > >>> I think removing this symlink would prevent /sys/fs/fuse/connections > >>> from being mounted and the fuse module from being loaded > >>> unconditionally on boot > >> > >> no > >> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1909805#c6 > > > > It almost works for me on Gentoo Linux. > > To test, I first had to reconfigure my kernel to build FUSE as a > > module (I normally have it built-in). > > I then removed the sys-fs-fuse-connections.mount symlink from > > sysinit.target.wants. > > After rebooting with the new kernel, the FUSE module is not loaded and > > /sys/fs/fuse/connections is not mounted. > > > > Unfortunately, mounting FUSE-based file systems does not work until I > > manually run "modprobe fuse". > > It seems that my kernel does not auto-load the module, despite the > > static /dev/fuse node. The kernel is probably missing a call to > > __request_module(). > > > > Given that the kernel doesn't auto-load the module on demand, leaving > > the sysinit.target.wants symlink in place seems like the safe thing to > > do. > > but for sure not on a stripped down machine running a iptables-nft > ruleset, a socket-activated sshd and nohting else > > if it's me for server setups the "fuse" kernel-module could be in > "kernel-modules" which is not installed and needed for virtualized guests > > the point is that all this setups where happy without fuse loaded from > 2008 to 2021 and you can't even avoid it with F33 at all, no matter what > you delete or mask > > a active masked unit - seriously? :-) > > [root@rawhide ~]# systemctl status sys-module-fuse.device > sys-fs-fuse-connections.mount > ● sys-module-fuse.device - /sys/module/fuse > Loaded: masked (Reason: Unit sys-module-fuse.device is masked.) > Active: active (plugged) since Mon 2021-02-08 19:33:18 CET; 1min > 42s ago > Device: /sys/module/fuse I think something else on your system is loading the fuse kernel module, which activates sys-module-fuse.device, and tries to start sys-fs-fuse-connections.mount. It appears systemd doesn't really support masking device units, which are generated by udev events. You should probably try to track down exactly what else is loading the fuse module, and disable that. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] consider dropping defrag of journals on btrfs
Chris Murphy writes: > And I agree 8MB isn't a big deal. Does anyone complain about journal > fragmentation on ext4 or xfs? If not, then we come full circle to my > second email in the thread which is don't defragment when nodatacow, > only defragment when datacow. Or use BTRFS_IOC_DEFRAG_RANGE and > specify 8MB length. That does seem to consistently no op on nodatacow > journals which have 8MB extents. Ok, I agree there. > The reason I'm dismissive is because the nodatacow fragment case is > the same as ext4 and XFS; the datacow fragment case is both > spectacular and non-deterministic. The workload will matter where Your argument seems to be that it's no worse than ext4 and so if we don't defrag there, why on btrfs? Lennart seems to be arguing that the only reason systemd doesn't defrag on ext4 is because the ioctl is harder to use. Maybe it should defrag as well, so he's asking for actual performance data to evaluate whether the defrag is pointless or whether maybe ext4 should also start doing a defrag. At least I think that's his point. Personally I agree ( and showed the calculations in a previous post ) that 8 MB/fragment is only going to have a negligiable impact on performance and so isn't worth bothering with a defrag, but he has asked for real world data... > And also, only defragmenting on rotation strikes me as leaving > performance on the table, right? If there is concern about fragmented No, because fragmentation only causes additional latency on HDD, not SSD. > But it sounds to me like you want to learn what the performance is of > journals defragmented with BTFS_IOC_DEFRAG specifically? I don't think > it's interesting because you're still better off leaving nodatacow > journals alone, and something still has to be done in the datacow Except that you're not. Your definition of better off appears to be only on SSD and only because it is preferable to have fewer writes than less fragmentation. On HDD defragmenting is a good thing. Lennart seems to want real world performance data to evaluate just *how* good and whether it's worth the bother, at least for HDD. For SSDs, I believe he agreed that it may as well be shut off there since it provides no benefit, but your patch kills it on HDDs as well. > Is there a test mode for journald to just dump a bunch of random stuff > into the journal to age it? I don't want to wait weeks to get a dozen > journal files. The cause of the fragmentation is slowly appending to the file over time, so if you dump a bunch of data in too quickly you would eliminate the fragmentation. You might try: while true ; do logger "This is a test log message to act as filler" ; sleep 1 ; done To speed things up a little bit. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Re: [EXT] Re: consider dropping defrag of journals on btrfs
>>> Phillip Susi schrieb am 09.02.2021 um 15:53 in >>> Nachricht <87o8gtz3m1@vps.thesusis.net>: > Chris Murphy writes: > >> Basically correct. It will merge random writes such that they become >> sequential writes. But it means inserts/appends/overwrites for a file >> won't be located with the original extents. > > Wait, I thoguht that was only true for metadata, not normal file data > blocks? Well, maybe it becomes true for normal data if you enable > compression. Or small files that get leaf packed into the metadata > chunk. > > If it's really combining streaming writes from two different files into > a single interleaved write to the disk, that would be really silly. Why? The idea of BtrFS was that any block written (or at least any block that is used "frequently enough") will be in RAM cache, so the actual location of a block does not matter. Perfomance-killing synchronous random writes would benefit instead. Like that (AFAIK). ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [EXT] Re: consider dropping defrag of journals on btrfs
Chris Murphy writes: > Basically correct. It will merge random writes such that they become > sequential writes. But it means inserts/appends/overwrites for a file > won't be located with the original extents. Wait, I thoguht that was only true for metadata, not normal file data blocks? Well, maybe it becomes true for normal data if you enable compression. Or small files that get leaf packed into the metadata chunk. If it's really combining streaming writes from two different files into a single interleaved write to the disk, that would be really silly. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Re: Antw: Antw: Re: Re: Antw: [EXT] Re: Still confused with socket activation
>>> Michael Chapman schrieb am 09.02.2021 um 11:28 in Nachricht <74f975f6-1ef2-ee64-bf95-415a5626...@very.puzzling.org>: > On Tue, 9 Feb 2021, Ulrich Windl wrote: > [...] >> OK, I tried (staring libvirtd.service with ‑‑listen and without ‑‑timout): >> Feb 09 10:59:23 h18 libvirtd[42540]: ‑‑listen parameter not permitted with >> systemd activation sockets, see 'man libvirtd' for further guidance >> Feb 09 10:59:23 h18 systemd[1]: libvirtd.service: Main process exited, >> code=exited, status=6/NOTCONFIGURED >> Feb 09 10:59:23 h18 systemd[1]: Failed to start Virtualization daemon. > > That must be because you're still passing through sockets from systemd. > > When `libvirtd.service` is started, any active socket units with > `Service=libvirtd.service` will be passed to the service. When libvirt is > started with `‑‑listen`, it checks that no sockets were passed to it. > > If you don't want libvirt to be socket‑activated, you have to make sure > ALL of libvirt's sockets are stopped. Masking them is a good idea too, > but stopping them is what's important. Actually I was 99% sure they all were stopped. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: Re: Antw: [EXT] Re: Still confused with socket activation
>>> Michael Chapman schrieb am 09.02.2021 um 10:17 in Nachricht : > On Tue, 9 Feb 2021, Ulrich Windl wrote: > [...] >> At what timne exactly? When pacemaker starts, or when the systemd using is >> about to be started? > > Pacemaker adds the drop‑in just before it starts the resource, and it > removes the drop‑in just after it stops the resource. It's entire purpose > is to handle the case when Pacemaker and the service are *simultaneously* > stopped (i.e. by something external to Pacemaker). > > Without the drop‑in, what can happen is that the service is stopped before > Pacemaker thinks it should be stopped... which makes Pacemaker attempt to > recover the resource... which makes every go wrong quickly. Yeah two resource managers is one too many (much?). ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Re: Re: Antw: [EXT] Re: Still confused with socket activation
>>> Andrei Borzenkov schrieb am 09.02.2021 um 10:14 in
Nachricht
:
> On Tue, Feb 9, 2021 at 11:54 AM Ulrich Windl
> wrote:
>>
>> Thanks and "back to the mess": If I use libvirtd.service instead of
>> libvirtd-tls.socket, it does *not* open the TLS socket, even though the
>> configuration file contains "listen_tls=1"...
>
> libvirtd --listen
>
> Did you read the link I gave you on the pacemaker list?
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1750340#c0
>
> quoting
>
> --><--
> Thus if the mgmt app / admin wants to use TCP/TLS sockets they have two
> choices
>
> - To continue the old approach (setting --listen in
> /etc/sysconfig/libvirtd), then they MUST use 'systemctl mask ...' for
> all the socket units listed above, before libvirtd.service is started.
> --><--
>
> Does it not work?
The point is that masking is still required ("nail down system with silver
nails once you killed it, just to make sure it doesn't resurrect") as
"disabling" the sockets is not enough.
Now it works.
Regards,
Ulrich
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: [EXT] Re: sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@fuse.service is masked
Am 09.02.21 um 08:54 schrieb Ulrich Windl: Reindl Harald schrieb am 08.02.2021 um 19:01 in Nachricht : Am 08.02.21 um 18:27 schrieb Lennart Poettering: On So, 07.02.21 22:43, Reindl Harald (h.rei...@thelounge.net) wrote: https://bugzilla.redhat.com/show_bug.cgi?id=1909805 In response to your actual issue, ignoring all the nasty wording: Masking is a last resort thing, you really want to use that only after having investigated everything. You use it here anyway to mask out a really low‑level system thing, hence you might get warnings about this. You can of course mask sys‑fs‑fuse‑connections.mount too who needs anything related to fuse at every boot? fuse is nothing in common used fuse is not used unconditional directly after boot on a server‑vm with no fuse business [root@testserver:~]$ lsmod | grep fuse fuse 163840 1 First it seems you fuse module is used by another module, and then in SLES15 SP2 with libvirt and two running Xen PVMs, I see no fuse being used or even loaded. So maybe find out who requests fuse... nobody and nothing requests it except systemd is triggering the loading don't compare your SLES with a completly stripped down Fedora 33 only running sshd - with F32 for now masking "sys-fs-fuse-connections.mount" is enough, on F33 not [harry@srv-rhsoft:~]$ rawhide [root@rawhide ~]# lsmod Module Size Used by nft_counter16384 4 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_multiport 20480 1 xt_conntrack 16384 1 nf_conntrack 163840 1 xt_conntrack nf_defrag_ipv6 24576 1 nf_conntrack nf_defrag_ipv4 16384 1 nf_conntrack nft_compat 20480 3 nf_tables 241664 9 nft_compat,nft_counter nfnetlink 16384 2 nft_compat,nf_tables crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel16384 0 vmw_balloon28672 0 vmw_vmci 90112 1 vmw_balloon vmxnet369632 0 ip_tables 28672 0 crc32c_intel 24576 1 vmw_pvscsi 32768 1 fuse 163840 0 [root@rawhide ~]# system-errors.sh Feb 9 12:03:32 rawhide systemd[1]: sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit sys-fs-fuse-connections.mount is masked. [root@rawhide ~]# rmmod fuse [root@rawhide ~]# even if you *delete* the "sysinit.target.wants/sys-fs-fuse-connections.mount" no, basic target don't want fuse [harry@srv-rhsoft:~]$ rawhide [root@rawhide ~]# system-errors.sh Feb 8 19:32:20 rawhide systemd[1]: sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit sys-fs-fuse-connections.mount is masked. [root@rawhide ~]# rpm -q --filesbypkg systemd | grep fuse systemd /usr/lib/systemd/system/sys-fs-fuse-connections.mount systemd /usr/lib/systemd/system/sysinit.target.wants/sys-fs-fuse-connections.mount [root@rawhide ~]# rm /usr/lib/systemd/system/sysinit.target.wants/sys-fs-fuse-connections.mount [root@rawhide ~]# rm /usr/lib/systemd/system/sys-fs-fuse-connections.mount [root@rawhide ~]# reboot [root@rawhide ~]# system-errors.sh Feb 8 19:33:19 rawhide systemd[1]: sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit sys-fs-fuse-connections.mount is masked. [root@rawhide ~]# systemctl status sys-module-fuse.device sys-fs-fuse-connections.mount ● sys-module-fuse.device - /sys/module/fuse Loaded: masked (Reason: Unit sys-module-fuse.device is masked.) Active: active (plugged) since Mon 2021-02-08 19:33:18 CET; 1min 42s ago Device: /sys/module/fuse Feb 08 19:33:19 rawhide.vmware.local systemd[1]: sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit sys-fs-fuse-connections.mount is masked. ● sys-fs-fuse-connections.mount Loaded: masked (Reason: Unit sys-fs-fuse-connections.mount is masked.) Active: inactive (dead) [root@rawhide ~]# ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sys-module-fuse.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@fuse.service is masked
Am 08.02.21 um 20:23 schrieb Mike Gilbert: On Mon, Feb 8, 2021 at 1:01 PM Reindl Harald wrote: Am 08.02.21 um 18:27 schrieb Lennart Poettering: On So, 07.02.21 22:43, Reindl Harald (h.rei...@thelounge.net) wrote: https://bugzilla.redhat.com/show_bug.cgi?id=1909805 In response to your actual issue, ignoring all the nasty wording: Masking is a last resort thing, you really want to use that only after having investigated everything. You use it here anyway to mask out a really low-level system thing, hence you might get warnings about this. You can of course mask sys-fs-fuse-connections.mount too who needs anything related to fuse at every boot? fuse is nothing in common used fuse is not used unconditional directly after boot on a server-vm with no fuse business [root@testserver:~]$ lsmod | grep fuse fuse 163840 1 my only usecase on 50 machines is every few weeks fuse.sshfs abd what makes people nasty is that for months nobody responds to systemd related issues in the Fedora bugracker if something is usiong fuse it happily loads the kernel module on-demand for years Looking into this a bit closer, it looks like systemd installs a symlink: /usr/lib/systemd/system/sysinit.target.wants/sys-fs-fuse-connections.mount -> ../sys-fs-fuse-connections.mount I think removing this symlink would prevent /sys/fs/fuse/connections from being mounted and the fuse module from being loaded unconditionally on boot no https://bugzilla.redhat.com/show_bug.cgi?id=1909805#c6 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: Antw: Re: Re: Antw: [EXT] Re: Still confused with socket activation
On Tue, 9 Feb 2021, Ulrich Windl wrote: [...] > OK, I tried (staring libvirtd.service with --listen and without --timout): > Feb 09 10:59:23 h18 libvirtd[42540]: --listen parameter not permitted with > systemd activation sockets, see 'man libvirtd' for further guidance > Feb 09 10:59:23 h18 systemd[1]: libvirtd.service: Main process exited, > code=exited, status=6/NOTCONFIGURED > Feb 09 10:59:23 h18 systemd[1]: Failed to start Virtualization daemon. That must be because you're still passing through sockets from systemd. When `libvirtd.service` is started, any active socket units with `Service=libvirtd.service` will be passed to the service. When libvirt is started with `--listen`, it checks that no sockets were passed to it. If you don't want libvirt to be socket-activated, you have to make sure ALL of libvirt's sockets are stopped. Masking them is a good idea too, but stopping them is what's important. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Antw: Re: Re: Antw: [EXT] Re: Still confused with socket activation
>>> "Ulrich Windl" schrieb am 09.02.2021
um
10:28 in Nachricht <602255b402a10003e...@gwsmtp.uni-regensburg.de>:
Andrei Borzenkov schrieb am 09.02.2021 um 10:14 in
> Nachricht
> :
>> On Tue, Feb 9, 2021 at 11:54 AM Ulrich Windl
>> wrote:
>>>
>>> Thanks and "back to the mess": If I use libvirtd.service instead of
>>> libvirtd‑tls.socket, it does *not* open the TLS socket, even though the
>>> configuration file contains "listen_tls=1"...
>
> ...and if I use libvirtd‑tls.socket, it fails on restarting:
> Feb 09 10:20:17 h18 systemd[1]: libvirtd‑tls.socket: Socket service
> libvirtd.service already active, refusing.
> Feb 09 10:20:17 h18 systemd[1]: Failed to listen on Libvirt TLS IP socket.
> Feb 09 10:20:19 h18 pacemaker‑controld[36557]: notice: Result of start
> operation for prm_libvirtd on h18: error
>
>>
>> libvirtd ‑‑listen
>>
>> Did you read the link I gave you on the pacemaker list?
>
> Not yet, but due to your hint I found:
> # If systemd socket activation is disabled, then the following
> # can be used to listen on TCP/TLS sockets
> #LIBVIRTD_ARGS="‑‑listen"
>
> ("back to the mess")
>
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1750340#c0
>>
>> quoting
>>
>> ‑‑><‑‑
>> Thus if the mgmt app / admin wants to use TCP/TLS sockets they have two
>> choices
>>
>> ‑ To continue the old approach (setting ‑‑listen in
>> /etc/sysconfig/libvirtd), then they MUST use 'systemctl mask ...' for
>> all the socket units listed above, before libvirtd.service is started.
>> ‑‑><‑‑
>>
>> Does it not work?
>
> I'll roll‑back and try ;‑)
OK, I tried (staring libvirtd.service with --listen and without --timout):
Feb 09 10:59:23 h18 libvirtd[42540]: --listen parameter not permitted with
systemd activation sockets, see 'man libvirtd' for further guidance
Feb 09 10:59:23 h18 systemd[1]: libvirtd.service: Main process exited,
code=exited, status=6/NOTCONFIGURED
Feb 09 10:59:23 h18 systemd[1]: Failed to start Virtualization daemon.
Is libvirtd.service (as opposed to libvirtd.socket) as "socket activation"? I
thought: "no".
>
> Regards,
> Ulrich
>
>
>
> ___
> systemd‑devel mailing list
> systemd‑de...@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd‑devel
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Re: Re: Antw: [EXT] Re: Still confused with socket activation
>>> Andrei Borzenkov schrieb am 09.02.2021 um 10:14 in
Nachricht
:
> On Tue, Feb 9, 2021 at 11:54 AM Ulrich Windl
> wrote:
>>
>> Thanks and "back to the mess": If I use libvirtd.service instead of
>> libvirtd-tls.socket, it does *not* open the TLS socket, even though the
>> configuration file contains "listen_tls=1"...
...and if I use libvirtd-tls.socket, it fails on restarting:
Feb 09 10:20:17 h18 systemd[1]: libvirtd-tls.socket: Socket service
libvirtd.service already active, refusing.
Feb 09 10:20:17 h18 systemd[1]: Failed to listen on Libvirt TLS IP socket.
Feb 09 10:20:19 h18 pacemaker-controld[36557]: notice: Result of start
operation for prm_libvirtd on h18: error
>
> libvirtd --listen
>
> Did you read the link I gave you on the pacemaker list?
Not yet, but due to your hint I found:
# If systemd socket activation is disabled, then the following
# can be used to listen on TCP/TLS sockets
#LIBVIRTD_ARGS="--listen"
("back to the mess")
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1750340#c0
>
> quoting
>
> --><--
> Thus if the mgmt app / admin wants to use TCP/TLS sockets they have two
> choices
>
> - To continue the old approach (setting --listen in
> /etc/sysconfig/libvirtd), then they MUST use 'systemctl mask ...' for
> all the socket units listed above, before libvirtd.service is started.
> --><--
>
> Does it not work?
I'll roll-back and try ;-)
Regards,
Ulrich
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: Re: Antw: [EXT] Re: Still confused with socket activation
On Tue, 9 Feb 2021, Ulrich Windl wrote: [...] > > > > libvirt can be run without socket activation [2]. I strongly recommend you > > configure it this way if you intend to manage libvirt in Pacemaker. > > Yes, I'd like to! Any pointers? Follow the link. It's all described there. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: Re: Antw: [EXT] Re: Still confused with socket activation
On Tue, 9 Feb 2021, Ulrich Windl wrote: [...] > At what timne exactly? When pacemaker starts, or when the systemd using is > about to be started? Pacemaker adds the drop-in just before it starts the resource, and it removes the drop-in just after it stops the resource. It's entire purpose is to handle the case when Pacemaker and the service are *simultaneously* stopped (i.e. by something external to Pacemaker). Without the drop-in, what can happen is that the service is stopped before Pacemaker thinks it should be stopped... which makes Pacemaker attempt to recover the resource... which makes every go wrong quickly. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: [EXT] Re: Still confused with socket activation
On Tue, Feb 9, 2021 at 11:54 AM Ulrich Windl wrote: > > Thanks and "back to the mess": If I use libvirtd.service instead of > libvirtd-tls.socket, it does *not* open the TLS socket, even though the > configuration file contains "listen_tls=1"... libvirtd --listen Did you read the link I gave you on the pacemaker list? https://bugzilla.redhat.com/show_bug.cgi?id=1750340#c0 quoting --><-- Thus if the mgmt app / admin wants to use TCP/TLS sockets they have two choices - To continue the old approach (setting --listen in /etc/sysconfig/libvirtd), then they MUST use 'systemctl mask ...' for all the socket units listed above, before libvirtd.service is started. --><-- Does it not work? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Re: Antw: [EXT] Re: Still confused with socket activation
>>> Michael Chapman schrieb am 09.02.2021 um 09:15 in Nachricht <20675743-9521-cdca-1c58-d42de7117...@very.puzzling.org>: > On Tue, 9 Feb 2021, Michael Chapman wrote: > [...] >> Note that when you're using Pacemaker to manage a systemd service, you >> should not enable the service in the normal way ‑‑ that is, the service >> should not be started simply by virtue of it being in the Wants= list of >> multi‑user.target. The service is intended to be started and stopped only >> by Pacemaker. > > Ah, there's something else I forgot to mention. > > Since Pacemaker is in charge of the service's lifecycle, you almost > certainly *do not* want it to be socket‑activated. Hi Michael! Thanks and "back to the mess": If I use libvirtd.service instead of libvirtd-tls.socket, it does *not* open the TLS socket, even though the configuration file contains "listen_tls=1"... Support was telling me I should "probably" use libvirtd-tls.socket instead of libvirtd.service... "The wheel has come full circle" said Shakespeare (AFAIR) > > libvirt can be run without socket activation [2]. I strongly recommend you > configure it this way if you intend to manage libvirt in Pacemaker. Yes, I'd like to! Any pointers? > > (I think managing libvirt in Pacemaker is a mistake, mind you. Sure, > manage individual libvirt *guests* in Pacemaker. But managing libvirt as a > whole from Pacemaker seems to be the wrong approach.) As libvirtd has a dependency on virtlockd, and when using indirect locks, virtlockd has a dependency on a (thinking multi-node) cluster-wide filesystem (like OCFS2), you *must* start virtlockd *after* the cluster filesystem had been mountd. Naturally libvirtd cannot be started before virtlockd, so you'll have what I'm trying to manage. As indicated before, I had asked a similar question at "superuser", but nobody had an answer so far... > > [2] https://libvirt.org/daemons.html Regards, Ulrich ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: Re: Antw: [EXT] Re: Still confused with socket activation
>>> Michael Chapman schrieb am 09.02.2021 um 09:09 in Nachricht : > On Tue, 9 Feb 2021, Ulrich Windl wrote: > [...] >> As for the drop‑ins: I neither know what those are expected to do, not who >> adds them at run time. See "documentation"... > > The 50‑pacemaker.conf drop‑ins are, as their name suggests, created by > Pacemaker. Hi Michael, thanks for explaining! > > Specifically, Pacemaker's systemd resource agent [1] creates a drop‑in on > each Pacemaker‑managed systemd service. Consider the situation where both At what timne exactly? When pacemaker starts, or when the systemd using is about to be started? > Pacemaker and the Pacemaker‑managed service are scheduled to be stopped > (e.g. you're rebooting the entire system). Either you want Pacemaker to > stop the service itself (and, perhaps, start the service on some other > node in your cluster), or ‑‑ if the pacemaker resource has management > disabled ‑‑ you want the service to be stopped *after* Pacemaker has been > stopped. Either way, the service needs to be ordered > Before=pacemaker.service. This is precisely what that drop‑in does. But doesn't "Before=pacemaker.service" say the corresponding service is to be *started* *before* pacemaker? If that's the case any ordering attempts inside the pacemaker configuration do not make any sense. Likewise when shutting down: If a systemd service needs some other pacemaker service it would be stopped *after* pacemaker. Sorry I don't understand that. > > Note that when you're using Pacemaker to manage a systemd service, you > should not enable the service in the normal way ‑‑ that is, the service > should not be started simply by virtue of it being in the Wants= list of > multi‑user.target. The service is intended to be started and stopped only > by Pacemaker. Yes, that is what I wanted, and it seems it works after a reboot of the node, but not when pacemaker had been running once. > > For more help on this drop‑in in particular, I think you'd be better off > contacting the Pacemaker developers. > > [1] > https://github.com/ClusterLabs/pacemaker/blob/master/lib/services/systemd.c Actually I had been raising the issue there too (after haing googled the topic). As it seems *nobody* managed to create the configuration I want. Probably I should dump all that libvirt stuff and ose the plain Xen RA until people fixed the mess, making it ready for actual use. Regards, Ulrich ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: [EXT] Re: Still confused with socket activation
On Tue, 9 Feb 2021, Ulrich Windl wrote: [...] > As for the drop-ins: I neither know what those are expected to do, not who > adds them at run time. See "documentation"... The 50-pacemaker.conf drop-ins are, as their name suggests, created by Pacemaker. Specifically, Pacemaker's systemd resource agent [1] creates a drop-in on each Pacemaker-managed systemd service. Consider the situation where both Pacemaker and the Pacemaker-managed service are scheduled to be stopped (e.g. you're rebooting the entire system). Either you want Pacemaker to stop the service itself (and, perhaps, start the service on some other node in your cluster), or -- if the pacemaker resource has management disabled -- you want the service to be stopped *after* Pacemaker has been stopped. Either way, the service needs to be ordered Before=pacemaker.service. This is precisely what that drop-in does. Note that when you're using Pacemaker to manage a systemd service, you should not enable the service in the normal way -- that is, the service should not be started simply by virtue of it being in the Wants= list of multi-user.target. The service is intended to be started and stopped only by Pacemaker. For more help on this drop-in in particular, I think you'd be better off contacting the Pacemaker developers. [1] https://github.com/ClusterLabs/pacemaker/blob/master/lib/services/systemd.c ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: [EXT] Re: Still confused with socket activation
On Tue, 9 Feb 2021, Michael Chapman wrote: [...] > Note that when you're using Pacemaker to manage a systemd service, you > should not enable the service in the normal way -- that is, the service > should not be started simply by virtue of it being in the Wants= list of > multi-user.target. The service is intended to be started and stopped only > by Pacemaker. Ah, there's something else I forgot to mention. Since Pacemaker is in charge of the service's lifecycle, you almost certainly *do not* want it to be socket-activated. libvirt can be run without socket activation [2]. I strongly recommend you configure it this way if you intend to manage libvirt in Pacemaker. (I think managing libvirt in Pacemaker is a mistake, mind you. Sure, manage individual libvirt *guests* in Pacemaker. But managing libvirt as a whole from Pacemaker seems to be the wrong approach.) [2] https://libvirt.org/daemons.html ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: [EXT] Re: consider dropping defrag of journals on btrfs
>>> Chris Murphy schrieb am 09.02.2021 um 06:13 in Nachricht : > On Mon, Feb 8, 2021 at 7:56 AM Phillip Susi wrote: >> >> >> Chris Murphy writes: >> >> >> It sounds like you are arguing that it is better to do the wrong thing >> >> on all SSDs rather than do the right thing on ones that aren't broken. >> > >> > No I'm suggesting there isn't currently a way to isolate >> > defragmentation to just HDDs. >> >> Yes, but it sounded like you were suggesting that we shouldn't even try, >> not just that it isn't 100% accurate. Sure, some SSDs will be stupid >> and report that they are rotational, but most aren't stupid, so it's a >> good idea to disable the defragmentation on drives that report that they >> are non rotational. > > So far I've seen, all USB devices report rotational. All USB flash > drives, and any SSD in an enclosure. > > Maybe some way of estimating rotational based on latency standard > deviation, and stick that in sysfs, instead of trusting device > reporting. But in the meantime, the imperfect rule could be do not > defragment unless it's SCSI/SATA/SAS and it reports it's rotational. OTOH we had set the rotational attribute to zero for years with out negative effects. We did that for SAN storage that actually has spinning discs, but what sense does the rotational attribute make if the logical disk is distributed over 40 physical disks, managed by a smart controller that does I/O scheduling across multiple hosts and logical disks? > > ‑‑ > Chris Murphy > ___ > systemd‑devel mailing list > systemd‑de...@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd‑devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
