Re: [systemd-devel] Ordering issues

[Scott Ellentuch]

[image: Mailtrack]

Sender
notified by
Mailtrack

01/05/23,
09:40:11 PM

On Thu, Jan 5, 2023 at 4:15 AM Lennart Poettering 
wrote:

> On Mi, 04.01.23 13:21, Scott Ellentuch (tuct...@gmail.com) wrote:
>
> > Hi,
> >
> > I'm trying to do something with inconsistent results and wondered where
> I'm
> > going wrong. I recently installed GLUSTERFS onto a set of 4 Raspberry
> PI's,
> > but am having some boot sequencing issues.
> >
> > It started trying to run rsyslogd. Apparently, it was getting started
> > before the gluster mount was done. After a bit of looking, I added into
> > [Unit] -
> >
> > After=glusterd.service
> >
> > Restarted the machine and all good.
>
> Hmm, you actually *want* rsyslog to run after glusterd?
>
> No, I really don't, but because the rsyslog.conf's are on the glusterfs
volume, my hands are a bit tied. I could move them back to the root
filesystem, but defeats why I did that all in the first place.


> The thing is that most system services log, and I presume glusterd
> too, and usually you want that to enter your log framework of choice,
> but that means rsyslog probably should run first, not after glusterd,
> but that means it cannot log onto gluster fs. because that would be a
> cyclic dep.
>
> In my case besides the rsyslog.conf being on the gluster fs, its also
logging to the gluster fs. Double whammy.

> So the next thing I tried was for
> > keepalived . It started up and failed with not being able to see its
> config
> > file (Moved to gluster). I changed -
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target
> > Wants=network-online.target
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > to
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target glusterd.service
> > Wants=network-online.target glusterd.service
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > But it still isn't letting glusterd run before it. (And I did run
> > `systemctl daemon-reload` after the change.
> >
> > How can I track down why this isn't happening in the order I expect?
>
> This all smeels like a mess of cyclic deps. See the system logs
> (journalctl).
>
> Yea, I actually am fuzzy on that right now, I'll be back to it by Tuesday
next week.


> /etc/ must be available during early boot, before you run complex
> services (such as glusterd) off it. Thus it cannot be backed by such
> complex services.
>
>
I don't have ALL of /etc. I just have directories or files, but essentially
the same issue.


> If you want /etc/ to be backed by such complex services, these
> services must run from the initrd, and stick around. But I doubt
> glusterfs is ready for that.
>
> Lennart
>
> --
> Lennart Poettering, Berlin
>

Re: [systemd-devel] Ordering issues

[Scott Ellentuch]

[image: Mailtrack]

Sender
notified by
Mailtrack

01/05/23,
09:34:36 PM

On Thu, Jan 5, 2023 at 1:42 AM Andrei Borzenkov  wrote:

> > Restarted the machine and all good. So the next thing I tried was for
> > keepalived . It started up and failed with not being able to see its
> config
> > file (Moved to gluster).
>
> It checks for /etc/keepalived/keepalived.conf file. If you moved this
> file somewhere else, you presumably need to adjust path to this file in
> unit definition as well.
>
>
File is in the same location, just /etc/keepalived is a link to
/gfs/etc/keepalived .


> > I changed -
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target
> > Wants=network-online.target
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > to
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target glusterd.service
> > Wants=network-online.target glusterd.service
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > But it still isn't letting glusterd run before it.
>
> What exactly does it mean? What is "it" that "isn't letting" and what is
> "it" in "before it" and how you determine that the problem is due to the
> wrong order?
>
>
keepalived appears to run before gluster has completed its mounts since I'm
getting a message about the ConditionFileNotEmpty being empty.

>(And I did run
> > `systemctl daemon-reload` after the change.
> >
>
> daemon-reload applies to units started after it. When you edited unit
> definition your system was already booted so presumably this unit was
> already activated.
>
>
Just covering all bases.


>
> > How can I track down why this isn't happening in the order I expect?
> >
>
> What makes you think the problem is in ordering of units? Have you
> checked journal? systemd logs when it begins and completes activation of
> each unit.
>

I am not near the systems for the next few days due to a funeral, I don't
remember now.

Thanks

[systemd-devel] Unmountable mounts and systemd-fsck@.service conflicting with shutdown.target

[Valentin David]

Hello,

In Ubuntu Core, we have some mounts that cannot be unmounted until we have
switched root.

To simplify, this looks like that:

/ mounts a ro loop devices backed by /some/disk/some/path/image.img
/some/disk mounts a block device (let's say /dev/some-block0p1)

In this case, /some/disk cannot be unmounted.

We do not want to lazily unmount, we cannot get errors if something fails.
(Unless we had a lazy unmount that would only work when read-only)

We do remount /some/disk read-only on shutdown. And in the shutdown
intramfs, we unmount /oldroot/some/disk.

However, we get an error message with systemd trying to unmount it. While
functionally, it does not matter, it is still very problematic to have
error messages.

Using `DefaultDependencies=no` is not enough. I have tried to be clever and
add some-disk.mount to shutdown.target.wants so it would not try to unmount
it. But systemd got confused with conflicts and randomly kills stop jobs
until there is no conflict.

Debugging it, I have found that this is because some-disk.mount depends on
systemd-fsck@some\x2dblock0p1.service. And systemd-fsck@.service conflicts
with shutdown.target.

I wonder if having conflict on shutdown.target really needed. Could we
remove it? (And also add DefaultDepenencies=no to
system-systemd\x2dfsck.slice) With this, mounts with DefaultDependencie=no
do not get unmounted as part of shutdown.target. (They do during
systemd-shutdown)

[systemd-devel] Review request: Adding JSON and table output to systemd-analyze plot

[Joshua Zivkovic]

Hello,

I recently submitted a pull request[1] to add JSON and table output
options to `systemd-analyze plot`, though a couple of weeks have passed
and it's still pending a review. I appreciate many will have been
taking some time out over the Christmas period, I just wanted to draw
attention to it for fear of it getting lost amongst the more recent
pull requests (mine has now fallen off the first page).

Kind regards,
Joshua Zivkovic

[1] https://github.com/systemd/systemd/pull/25790

Re: [systemd-devel] Ordering issues

[Lennart Poettering]

On Mi, 04.01.23 13:21, Scott Ellentuch (tuct...@gmail.com) wrote:

> Hi,
>
> I'm trying to do something with inconsistent results and wondered where I'm
> going wrong. I recently installed GLUSTERFS onto a set of 4 Raspberry PI's,
> but am having some boot sequencing issues.
>
> It started trying to run rsyslogd. Apparently, it was getting started
> before the gluster mount was done. After a bit of looking, I added into
> [Unit] -
>
> After=glusterd.service
>
> Restarted the machine and all good.

Hmm, you actually *want* rsyslog to run after glusterd?

The thing is that most system services log, and I presume glusterd
too, and usually you want that to enter your log framework of choice,
but that means rsyslog probably should run first, not after glusterd,
but that means it cannot log onto gluster fs. because that would be a
cyclic dep.

> So the next thing I tried was for
> keepalived . It started up and failed with not being able to see its config
> file (Moved to gluster). I changed -
>
> [Unit]
> Description=Keepalive Daemon (LVS and VRRP)
> After=network-online.target
> Wants=network-online.target
> # Only start if there is a configuration file
> ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
>
> to
>
> [Unit]
> Description=Keepalive Daemon (LVS and VRRP)
> After=network-online.target glusterd.service
> Wants=network-online.target glusterd.service
> # Only start if there is a configuration file
> ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
>
> But it still isn't letting glusterd run before it. (And I did run
> `systemctl daemon-reload` after the change.
>
> How can I track down why this isn't happening in the order I expect?

This all smeels like a mess of cyclic deps. See the system logs
(journalctl).

/etc/ must be available during early boot, before you run complex
services (such as glusterd) off it. Thus it cannot be backed by such
complex services.

If you want /etc/ to be backed by such complex services, these
services must run from the initrd, and stick around. But I doubt
glusterfs is ready for that.

Lennart

--
Lennart Poettering, Berlin