Re: [systemd-devel] Ordering issues

[Scott Ellentuch]

Hi,

I have a cluster of 4 Raspberry Pis that are responsible for close to 20
different activities (Mrtg, Smokeping, OpenVPN [server and client], Netflow
, etc). When one fails (which they do) if it wasn't the leader they'll
elect a new leader, and that one decides how to redistribute the work. I
want that one to make any config changes needed at the time and signal that
machine to reload/restart/etc.

As an aside, putting an "After" of gfs.mount seems to have done the trick.

Tuc

On Fri, Jan 6, 2023, 12:26 PM Lennart Poettering 
wrote:

> On Do, 05.01.23 21:39, Scott Ellentuch (tuct...@gmail.com) wrote:
>
> > > > Restarted the machine and all good. So the next thing I tried was for
> > > > keepalived . It started up and failed with not being able to see its
> > > config
> > > > file (Moved to gluster).
> > >
> > > It checks for /etc/keepalived/keepalived.conf file. If you moved this
> > > file somewhere else, you presumably need to adjust path to this file in
> > > unit definition as well.
> > >
> > File is in the same location, just /etc/keepalived is a link to
> > /gfs/etc/keepalived .
>
> This sounds like a bad idea. use gluster for storing data, not for
> storing configuration. You are using a cluster storage where you
> should just use configuration management. Don't confuse the two
> concepts. One is totally not like the other, and you cannot replace
> one with the other.
>
> This will create tons of cyclic deps.
>
> This all sounds like a terrible idea, you are actively working on
> making things hard for you.
>
> Lennart
>
> --
> Lennart Poettering, Berlin
>

Re: [systemd-devel] Ordering issues

[Lennart Poettering]

On Do, 05.01.23 21:39, Scott Ellentuch (tuct...@gmail.com) wrote:

> > > Restarted the machine and all good. So the next thing I tried was for
> > > keepalived . It started up and failed with not being able to see its
> > config
> > > file (Moved to gluster).
> >
> > It checks for /etc/keepalived/keepalived.conf file. If you moved this
> > file somewhere else, you presumably need to adjust path to this file in
> > unit definition as well.
> >
> File is in the same location, just /etc/keepalived is a link to
> /gfs/etc/keepalived .

This sounds like a bad idea. use gluster for storing data, not for
storing configuration. You are using a cluster storage where you
should just use configuration management. Don't confuse the two
concepts. One is totally not like the other, and you cannot replace
one with the other.

This will create tons of cyclic deps.

This all sounds like a terrible idea, you are actively working on
making things hard for you.

Lennart

--
Lennart Poettering, Berlin

Re: [systemd-devel] Ordering issues

[Scott Ellentuch]

[image: Mailtrack]

Sender
notified by
Mailtrack

01/05/23,
09:40:11 PM

On Thu, Jan 5, 2023 at 4:15 AM Lennart Poettering 
wrote:

> On Mi, 04.01.23 13:21, Scott Ellentuch (tuct...@gmail.com) wrote:
>
> > Hi,
> >
> > I'm trying to do something with inconsistent results and wondered where
> I'm
> > going wrong. I recently installed GLUSTERFS onto a set of 4 Raspberry
> PI's,
> > but am having some boot sequencing issues.
> >
> > It started trying to run rsyslogd. Apparently, it was getting started
> > before the gluster mount was done. After a bit of looking, I added into
> > [Unit] -
> >
> > After=glusterd.service
> >
> > Restarted the machine and all good.
>
> Hmm, you actually *want* rsyslog to run after glusterd?
>
> No, I really don't, but because the rsyslog.conf's are on the glusterfs
volume, my hands are a bit tied. I could move them back to the root
filesystem, but defeats why I did that all in the first place.


> The thing is that most system services log, and I presume glusterd
> too, and usually you want that to enter your log framework of choice,
> but that means rsyslog probably should run first, not after glusterd,
> but that means it cannot log onto gluster fs. because that would be a
> cyclic dep.
>
> In my case besides the rsyslog.conf being on the gluster fs, its also
logging to the gluster fs. Double whammy.

> So the next thing I tried was for
> > keepalived . It started up and failed with not being able to see its
> config
> > file (Moved to gluster). I changed -
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target
> > Wants=network-online.target
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > to
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target glusterd.service
> > Wants=network-online.target glusterd.service
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > But it still isn't letting glusterd run before it. (And I did run
> > `systemctl daemon-reload` after the change.
> >
> > How can I track down why this isn't happening in the order I expect?
>
> This all smeels like a mess of cyclic deps. See the system logs
> (journalctl).
>
> Yea, I actually am fuzzy on that right now, I'll be back to it by Tuesday
next week.


> /etc/ must be available during early boot, before you run complex
> services (such as glusterd) off it. Thus it cannot be backed by such
> complex services.
>
>
I don't have ALL of /etc. I just have directories or files, but essentially
the same issue.


> If you want /etc/ to be backed by such complex services, these
> services must run from the initrd, and stick around. But I doubt
> glusterfs is ready for that.
>
> Lennart
>
> --
> Lennart Poettering, Berlin
>

Re: [systemd-devel] Ordering issues

[Scott Ellentuch]

[image: Mailtrack]

Sender
notified by
Mailtrack

01/05/23,
09:34:36 PM

On Thu, Jan 5, 2023 at 1:42 AM Andrei Borzenkov  wrote:

> > Restarted the machine and all good. So the next thing I tried was for
> > keepalived . It started up and failed with not being able to see its
> config
> > file (Moved to gluster).
>
> It checks for /etc/keepalived/keepalived.conf file. If you moved this
> file somewhere else, you presumably need to adjust path to this file in
> unit definition as well.
>
>
File is in the same location, just /etc/keepalived is a link to
/gfs/etc/keepalived .


> > I changed -
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target
> > Wants=network-online.target
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > to
> >
> > [Unit]
> > Description=Keepalive Daemon (LVS and VRRP)
> > After=network-online.target glusterd.service
> > Wants=network-online.target glusterd.service
> > # Only start if there is a configuration file
> > ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
> >
> > But it still isn't letting glusterd run before it.
>
> What exactly does it mean? What is "it" that "isn't letting" and what is
> "it" in "before it" and how you determine that the problem is due to the
> wrong order?
>
>
keepalived appears to run before gluster has completed its mounts since I'm
getting a message about the ConditionFileNotEmpty being empty.

>(And I did run
> > `systemctl daemon-reload` after the change.
> >
>
> daemon-reload applies to units started after it. When you edited unit
> definition your system was already booted so presumably this unit was
> already activated.
>
>
Just covering all bases.


>
> > How can I track down why this isn't happening in the order I expect?
> >
>
> What makes you think the problem is in ordering of units? Have you
> checked journal? systemd logs when it begins and completes activation of
> each unit.
>

I am not near the systems for the next few days due to a funeral, I don't
remember now.

Thanks

Re: [systemd-devel] Ordering issues

[Lennart Poettering]

On Mi, 04.01.23 13:21, Scott Ellentuch (tuct...@gmail.com) wrote:

> Hi,
>
> I'm trying to do something with inconsistent results and wondered where I'm
> going wrong. I recently installed GLUSTERFS onto a set of 4 Raspberry PI's,
> but am having some boot sequencing issues.
>
> It started trying to run rsyslogd. Apparently, it was getting started
> before the gluster mount was done. After a bit of looking, I added into
> [Unit] -
>
> After=glusterd.service
>
> Restarted the machine and all good.

Hmm, you actually *want* rsyslog to run after glusterd?

The thing is that most system services log, and I presume glusterd
too, and usually you want that to enter your log framework of choice,
but that means rsyslog probably should run first, not after glusterd,
but that means it cannot log onto gluster fs. because that would be a
cyclic dep.

> So the next thing I tried was for
> keepalived . It started up and failed with not being able to see its config
> file (Moved to gluster). I changed -
>
> [Unit]
> Description=Keepalive Daemon (LVS and VRRP)
> After=network-online.target
> Wants=network-online.target
> # Only start if there is a configuration file
> ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
>
> to
>
> [Unit]
> Description=Keepalive Daemon (LVS and VRRP)
> After=network-online.target glusterd.service
> Wants=network-online.target glusterd.service
> # Only start if there is a configuration file
> ConditionFileNotEmpty=/etc/keepalived/keepalived.conf
>
> But it still isn't letting glusterd run before it. (And I did run
> `systemctl daemon-reload` after the change.
>
> How can I track down why this isn't happening in the order I expect?

This all smeels like a mess of cyclic deps. See the system logs
(journalctl).

/etc/ must be available during early boot, before you run complex
services (such as glusterd) off it. Thus it cannot be backed by such
complex services.

If you want /etc/ to be backed by such complex services, these
services must run from the initrd, and stick around. But I doubt
glusterfs is ready for that.

Lennart

--
Lennart Poettering, Berlin

Re: [systemd-devel] Ordering issues

[Andrei Borzenkov]

On 04.01.2023 21:21, Scott Ellentuch wrote:

Hi,

I'm trying to do something with inconsistent results and wondered where I'm
going wrong. I recently installed GLUSTERFS onto a set of 4 Raspberry PI's,
but am having some boot sequencing issues.

It started trying to run rsyslogd. Apparently, it was getting started
before the gluster mount was done. After a bit of looking, I added into
[Unit] -

After=glusterd.service

Restarted the machine and all good. So the next thing I tried was for
keepalived . It started up and failed with not being able to see its config
file (Moved to gluster).


It checks for /etc/keepalived/keepalived.conf file. If you moved this 
file somewhere else, you presumably need to adjust path to this file in 
unit definition as well.



I changed -

[Unit]
Description=Keepalive Daemon (LVS and VRRP)
After=network-online.target
Wants=network-online.target
# Only start if there is a configuration file
ConditionFileNotEmpty=/etc/keepalived/keepalived.conf

to

[Unit]
Description=Keepalive Daemon (LVS and VRRP)
After=network-online.target glusterd.service
Wants=network-online.target glusterd.service
# Only start if there is a configuration file
ConditionFileNotEmpty=/etc/keepalived/keepalived.conf

But it still isn't letting glusterd run before it. 


What exactly does it mean? What is "it" that "isn't letting" and what is 
"it" in "before it" and how you determine that the problem is due to the 
wrong order?



(And I did run
`systemctl daemon-reload` after the change.



daemon-reload applies to units started after it. When you edited unit 
definition your system was already booted so presumably this unit was 
already activated.




How can I track down why this isn't happening in the order I expect?



What makes you think the problem is in ordering of units? Have you 
checked journal? systemd logs when it begins and completes activation of 
each unit.