Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Philipp Kern]

On 8/20/2019 8:02 AM, Bjørn Mork wrote:
> Bernd Zeimetz  writes:
>> On 8/11/19 12:01 PM, Adam Borowski wrote:
>>>   restart|force-reload)
>>> log_daemon_msg "Restarting $DESC"
>>> do_stop
>>> sleep 1
>>> do_start
>>> log_end_msg $?
>>> ;;
>>>
>>> Yes, this particular case might fail on a pathologically loaded box or with 
>>> a
>>> very slow chip, but I don't know of a way to ask firmware if something is
>>> still winding down, thus what we ship is probably still sanest.
>>
>> yes, for firmware this makes sense, but I've also seen various sysv
>> scripts where people tried to guess the time a service needs to
>> shutdown, so some random sleep was added instead of handling it in a
>> sane way. This issues are luckily fixed forever with systemd - it just
>> knows, whats going on.
> 
> LOL! Please try to be serious.
> 
> https://www.google.com/search?q=%22stop+job+is+running%22

I'd wager that most of this is due to unit files autogenerated by the
sysv generator as well as daemon options not being sufficiently tightly
speced out in native unit files. After all, you do want to give daemons
some time to stop. But at least with systemd you know when the process
has exited.

Also I mostly saw this taking a long time around deactivation of devices
(swap, crypto). (Although I question why you'd disable swap given the
consequence of getting everything back in, but alas.)

Kind regards
Philipp Kern

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Bjørn Mork]

Bernd Zeimetz  writes:
> On 8/11/19 12:01 PM, Adam Borowski wrote:
>>   restart|force-reload)
>> log_daemon_msg "Restarting $DESC"
>> do_stop
>> sleep 1
>> do_start
>> log_end_msg $?
>> ;;
>> 
>> Yes, this particular case might fail on a pathologically loaded box or with a
>> very slow chip, but I don't know of a way to ask firmware if something is
>> still winding down, thus what we ship is probably still sanest.
>
> yes, for firmware this makes sense, but I've also seen various sysv
> scripts where people tried to guess the time a service needs to
> shutdown, so some random sleep was added instead of handling it in a
> sane way. This issues are luckily fixed forever with systemd - it just
> knows, whats going on.

LOL! Please try to be serious.

https://www.google.com/search?q=%22stop+job+is+running%22


Bjørn

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Bernd Zeimetz]

On 8/11/19 12:01 PM, Adam Borowski wrote:
>   restart|force-reload)
> log_daemon_msg "Restarting $DESC"
> do_stop
> sleep 1
> do_start
> log_end_msg $?
> ;;
> 
> Yes, this particular case might fail on a pathologically loaded box or with a
> very slow chip, but I don't know of a way to ask firmware if something is
> still winding down, thus what we ship is probably still sanest.

yes, for firmware this makes sense, but I've also seen various sysv
scripts where people tried to guess the time a service needs to
shutdown, so some random sleep was added instead of handling it in a
sane way. This issues are luckily fixed forever with systemd - it just
knows, whats going on.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Marc Haber]

On Sun, 11 Aug 2019 10:49:57 +0200, Vincent Bernat 
wrote:
>systemd cannot guess if a SysV init script should leave a daemon behind
>or not. Therefore, they are converted as "Type=forking", "Restart=no"
>"GuessMainPID=no" and "RemainAfterExit=yes". So, when a daemon stops
>unexpectedly, it is not restarted and "restart" doesn't work because the
>unit is still active.

Since we have made providing a native systemd service unit optional,
did we document this non-negligible shortcoming for both package
maintainers and our users? 

Shouldn't we ask our package maintainers to provide native systemd
units to make systemd more useful for our systemd users?

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Don Armstrong]

On Sun, 11 Aug 2019, Adam Borowski wrote:
> A SysV init script being naturally a script makes hacking in fixes
> much easier, both for the admin and maintainer. For example,
> restarting connman with systemd means no wifi unless you restart twice
> (or stop, wait, start), this works with sysvinit:
> 
>   restart|force-reload)
> log_daemon_msg "Restarting $DESC"
> do_stop
> sleep 1
> do_start
> log_end_msg $?
> ;;

Probably the sleep 1 should really be in do_stop, and then the parallel
implementation in systemd is just to add

ExecStop=/bin/sleep 1

[This is to avoid `service connman stop; service connman start;` being 
surprising.]

> I don't know of a way to ask firmware if something is still winding
> down

If someone figures out how to do this, you'd just replace the sleep with
such a command.

-- 
Don Armstrong  https://www.donarmstrong.com

"The trouble with you, Ibid" he said, "is that you think you're the
biggest bloody authority on everything"
 -- Terry Pratchet _Pyramids_ p146

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Adam Borowski]

On Sun, Aug 11, 2019 at 10:49:57AM +0200, Vincent Bernat wrote:
>  ❦ 11 août 2019 10:27 +02, Marc Haber :
> > We have, however, failed to make use of that. "systemctl restart" is
> > nearly useless in Debian because a non-negligible part of our daemon
> > packages make systemd think the daemon is running while it is actually
> > not, and systemd does for some reason not to anything in this case.
> >
> > I have watched myself involuntarily changing to a systemctl stop;
> > systemctl start scheme, because this has a way higher chance of doing
> > what I want. I haven't dug in deeply in this matter though.
> 
> systemd cannot guess if a SysV init script should leave a daemon behind
> or not. Therefore, they are converted as "Type=forking", "Restart=no"
> "GuessMainPID=no" and "RemainAfterExit=yes". So, when a daemon stops
> unexpectedly, it is not restarted and "restart" doesn't work because the
> unit is still active.

A SysV init script being naturally a script makes hacking in fixes much
easier, both for the admin and maintainer.  For example, restarting connman
with systemd means no wifi unless you restart twice (or stop, wait, start),
this works with sysvinit:

  restart|force-reload)
log_daemon_msg "Restarting $DESC"
do_stop
sleep 1
do_start
log_end_msg $?
;;

Yes, this particular case might fail on a pathologically loaded box or with a
very slow chip, but I don't know of a way to ask firmware if something is
still winding down, thus what we ship is probably still sanest.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian is one big family.  Including that weird uncle
⢿⡄⠘⠷⠚⠋⠀ and ultra-religious in-laws.
⠈⠳⣄

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Vincent Bernat]

 ❦ 11 août 2019 10:27 +02, Marc Haber :

>>* Better restart semantics and monitoring of services/ways to configure
>>  restart.
>
> We have, however, failed to make use of that. "systemctl restart" is
> nearly useless in Debian because a non-negligible part of our daemon
> packages make systemd think the daemon is running while it is actually
> not, and systemd does for some reason not to anything in this case.
>
> I have watched myself involuntarily changing to a systemctl stop;
> systemctl start scheme, because this has a way higher chance of doing
> what I want. I haven't dug in deeply in this matter though.

systemd cannot guess if a SysV init script should leave a daemon behind
or not. Therefore, they are converted as "Type=forking", "Restart=no"
"GuessMainPID=no" and "RemainAfterExit=yes". So, when a daemon stops
unexpectedly, it is not restarted and "restart" doesn't work because the
unit is still active.
-- 
Tell the truth or trump--but get the trick.
-- Mark Twain, "Pudd'nhead Wilson's Calendar"


signature.asc
Description: PGP signature

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Marc Haber]

On Sat, 10 Aug 2019 19:41:59 -0400, Sam Hartman 
wrote:
>I'm one of the people who has found systemd hugely valuable in server
>environments.

So am I, but I see a bunch of shortcomings.

>Things I've found valuable include:
>
>* Avoiding imperative languages for configuration

This makes using "magic" in startup procedures vastly harder. For
agenda reasons, Upstream doesnt want people to write scripts and makes
that intentionally hard. Because people usually don't follow, this
leads to bash -c constructs in the unit file. I have not see init
script that are on this level of ugliness.

>* Better restart semantics and monitoring of services/ways to configure
>  restart.

We have, however, failed to make use of that. "systemctl restart" is
nearly useless in Debian because a non-negligible part of our daemon
packages make systemd think the daemon is running while it is actually
not, and systemd does for some reason not to anything in this case.

I have watched myself involuntarily changing to a systemctl stop;
systemctl start scheme, because this has a way higher chance of doing
what I want. I haven't dug in deeply in this matter though.

>I'll admit that I do use complex dependencies more on desktop or
>appliance machines than I do on servers.

Would you want to share some examples of that, maybe on the wiki? The
vast amount of systemd documentation is weirdly short of example code.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Sam Hartman]

Hi.

I'm one of the people who has found systemd hugely valuable in server
environments.

Things I've found valuable include:

* Avoiding imperative languages for configuration

* Better configuration of security isolation

* Better restart semantics and monitoring of services/ways to configure
  restart.  I was one of the people who did use inittab to try and
  restart services, found it wanting, and generally found that I had
  inadequate tools for handling service restart.  In some cases, I used
  third party tools; in some cases, I accepted the decrease in
  reliability

* Better separation of configuration especially for cloud and container
  images.  Fragments are helpful, as are things like conditios

* Tools like path-based units which can be incredibly helpful in
  scripting security isolation and putting together simple queues

* Better log filtering and querying through the journal

I'll admit that I do use complex dependencies more on desktop or
appliance machines than I do on servers.  However, I find systemd makes
me much happier as a sysadmin than any tool I've used before.

--Sam

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Russ Allbery]

Simon Richter  writes:

> What I'm not happy with is that we have effectively incorporated systemd
> unit files as an interface into Debian Policy without *explicitly* doing
> so, and that this interface remains "defined by upstream".

> If that is what we want, then we should update Debian Policy.

With my Policy Editor hat on, I'm in complete support of this.  It just
needs cycles for someone to write the language.

-- 
Russ Allbery (r...@debian.org)   

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Simon Richter]

Hi,

On Fri, Aug 09, 2019 at 02:54:55PM +0100, Simon McVittie wrote:

> To a large extent, the design of units and service files *is* systemd.

This is a large part of the systemd criticism as well: the refusal to
commit to an API because it would hinder future development, while at the
same time proposing to make this moving target the official API for service
startup.

Debian Policy is rather explicit about the interface of init scripts,
because that is how services integrate into Debian. I was completely fine
when that interface got extended to allow dependency-based boot using
insserv, and I'm also -mostly- fine with replacing it with something else.

What I'm not happy with is that we have effectively incorporated systemd
unit files as an interface into Debian Policy without *explicitly* doing
so, and that this interface remains "defined by upstream".

If that is what we want, then we should update Debian Policy.

> If
> you don't like the design of systemd, and you set out to implement
> something that implements the same "API" by reading the man pages and
> implementing all the features described there, it's hard to see how you
> could get a result that doesn't have at least some of the same properties
> that you don't like about systemd.

I can totally see the benefit of using systemd unit files as init scripts
through start-stop-daemon, because we could have a common "open socket,
chroot, drop privileges" wrapper that way, most of the code is already
there (start-stop-daemon's command line is a descriptive interface) and it
would still be simple enough to fully understand in a single day.

If I went and implemented the "API" by implementing all the features in the
man pages, I would indeed get the single property I like the least about
systemd: that its specification is always in flux.

I am fairly sure it would be possible to define a more powerful API than
init scripts. Alas, that hasn't happened yet, because nobody is willing to
create a normative reference.

   Simon

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Alf Gaida]

On 09.08.19 12:06, Ansgar wrote:
>
> Having sysvinit might make things a bit easier for Hurd/kFreeBSD, but
> it's not an absolute requirement for such a port to exist.
>
> Ansgar
>
Thanks Ansgar, this is the user deep in me - i like things to easy as
possible. More verbose: I will apply all patches for the Hurd or
kFreeBSD as soon as possible in my packages and upstream if possible and
will continue to do so. The other thing is: God damn, i want to able to
run a Hurd or kFreeBSD desktop system on my hardware as soon as possible
:D - it was possible for a mere human with kfreebsd before, right now it
is not so easy anymore


Alf

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Alf Gaida]

On 09.08.19 15:51, Tomas Pospisek wrote:
>
> FWIW (I mean it, this is just anecdotical evidence): I have been
> recently upgrading a lot of containers and host and I have been unable
> to make lxc guest with systemd inits even start.
>
> Also, I have been having problems with ssh sessions taking 25 seconds to
> start on the remote side because of systemd and pam trying to initialize
> some systemd user session.
>
> I gave up on understanding both of those problems after an hour or two
> of research on each. After all, machines were down, automation was not
> working I had to get the stuff running again.
>
> So at this instant I can't see sysv going away because there's too many
> things not working in practice on my systems with systemd.
> *t

Tom, you are not alone. I use LXD with lots of containers - and i had

a) to use Ubuntu as host because LXD is not in debian yet

b) LXC is more fun with an LXD daemon


Some services still have problems, right now in all of my installations
i have a problem with redis - it just don't work with systemd inside of
the container. Filed a bug, the bug was closed without a solution in
debian, there is no solution in upstream nor in systemd - so my solution
was:

a) give some silent swearwords to the involved parties

b) give a fuck about the reaction of the redis debian maintainer

c) delete the not working systemd service file and activate the service
over the fallback in init.d


More verbose: I used the f* word before and i will continue to do so.
Beside of that:

* i don't care about any init systems as long they work predictable

* for my use cases systemd works far more predictable than sysv

* there will be some things that are not optimal in systemd right now -
so what, let fix them

* same for sysv - as long sysv scripts are in debian package they should
be as bug free as possible


The ultima ratio for me (and i guess for the vast majority of user) is:
I like working software - so lets build working software. It might be
that some things right now don't work like expected, so improve these
things if possible.


Cheers Alf

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Jeremy Stanley]

On 2019-08-09 07:00:41 +0200 (+0200), Vincent Bernat wrote:
>  ❦  8 août 2019 21:47 +02, Simon Richter :
> 
> >> inetd performance is very low because it needs to spawn one instance for
> >> each connection. systemd socket activation has absolutely 0 overhead
> >> except on the first connection (where systemd needs to start the
> >> service).
> >
> > If you specify "wait" instead of "nowait" for a TCP service, your service
> > is handed the listening socket, and can continue to accept more connections
> > on that socket.
> [...]
> 
> My bad, I didn't know that.
> 
> > This feature went unused not because noone thought of it, but because there
> > is no real world use case that benefits from it. Either the service is used
> > regularly, then it makes sense to keep the daemon in memory, or it isn't,
> > then the reduced complexity for one-shot services (to the point where you
> > can use a shell script as a service) is the benefit.
> 
> Reality seems different. Almost nothing was using inetd (tftpd is the
> only daemon I have in mind), while many daemons adopted systemd socket
> activation. For example, OpenSSH (optional), Docker (by default), CUPS
> (by default), libvirt (by default, several services), Dovecot (by
> default). It seems the key difference is that socket-activated services
> are regular services. They can be started manually if we want them to be
> and they inherit everything systemd is able to provide to services.

On the other hand, I still run the following out of inetd (with
nowait by the way) on a bunch of systemd-based systems: cfingerd,
ident2, simpleproxy, tcpd. I'll readily admit it's because I've just
not found the time to work out how to configure systemd to serve
them instead (last I checked the packages for these didn't include a
service file), but I've also not felt particularly compelled to as
it's really convenient just to be able to put a line in
/etc/inetd.conf and HUP it.

Systemd having socket activation doesn't automatically make inetd
obsolete.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Simon McVittie]

On Fri, 09 Aug 2019 at 17:12:17 +0800, Benda Xu wrote:
> Simon Richter  writes:
> > For that to happen, we'd have to define .service files as an API
> > though, which would feature-freeze them, and I'm not sure the systemd
> > people would be happy about that.
> 
> Thank you for sharing your thoughts.  I also think adding a
> compatibility layer is the best way to move forward.
> 
> With more upstream packages providing systemd-.service by default, it is
> more expansive for the systemd team to break the existing .service API.
> We could consider it to be freezed in practice, or at least backward
> compatible.

Frozen is not the same as backwards compatible. systemd can add new
functionality, and other projects can start to depend on that new
functionality, without it being a backwards compatibility break.

All these examples "go back in time" a bit so that I can talk about systemd
features that literally do exist, but please imagine that they didn't exist
yet and were going to be added in a future systemd version.

When a file format / feature set is backwards-compatible but not frozen,
you can have this situation:

- systemd adds an option, like RuntimeDirectory=
- service "foo" starts relying on that option for correct functionality
- the hypothetical compatibility layer does not support that option
- foo doesn't work correctly

or this:

- systemd adds a feature that is not reflected in .service files, like
  support for creating named pipes with "p" in tmpfiles.d(5) files
- service "foo" ships a tmpfiles.d file that makes use of that, and
  the invocation of foo in foo.service relies on the creation of that
  named pipe for correct functionality
- the hypothetical compatibility layer does not integrate with tmpfiles.d,
  or does integrate with tmpfiles.d but doesn't support "p"
- foo doesn't work correctly

or this:

- systemd adds support for activation of D-Bus services that have a
  SystemdService configured
- dbus changes its ExecStart command line to include --systemd-activation,
  which makes dbus-daemon delegate activation of those services to systemd
- the hypothetical compatibility layer does not implement activation of
  systemd services (it would be hard for it to do so - it isn't systemd)
- dbus doesn't work correctly

The compatibility layer might not implement a feature because nobody has
implemented it yet, or might never implement it because the authors of
that compatibility layer don't like it, or might never implement it
because it's unimplementable without basically being systemd.

To a large extent, the design of units and service files *is* systemd. If
you don't like the design of systemd, and you set out to implement
something that implements the same "API" by reading the man pages and
implementing all the features described there, it's hard to see how you
could get a result that doesn't have at least some of the same properties
that you don't like about systemd.

smcv

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Tomas Pospisek]

Am 07.08.19 um 19:00 schrieb Marc Haber:
> On Wed, 7 Aug 2019 14:44:01 +0100, Ian Jackson
>  wrote:
>> Marc Haber writes ("Re: do packages depend on lexical order or 
>> {daily,weekly,monthly} cron jobs?"):
>>> We have already thrown sysvinit away.
>>
>> No, we have not.
> 
> We have given up on so many ideas that sysvinit has come with that it
> doesn't make sense to stick to Tradition on this count.

FWIW (I mean it, this is just anecdotical evidence): I have been
recently upgrading a lot of containers and host and I have been unable
to make lxc guest with systemd inits even start.

Also, I have been having problems with ssh sessions taking 25 seconds to
start on the remote side because of systemd and pam trying to initialize
some systemd user session.

I gave up on understanding both of those problems after an hour or two
of research on each. After all, machines were down, automation was not
working I had to get the stuff running again.

So at this instant I can't see sysv going away because there's too many
things not working in practice on my systems with systemd.
*t

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Ansgar]

Alf Gaida writes:
> We need sysvinit for some non-linux things

No: Hurd existed for a long time without using sysvinit/sysv-rc.  I
think sysvinit was only ported to Hurd in 2013 or 2014 (I didn't search
much, but found a Summer of Code application from 2013 for this).

Having sysvinit might make things a bit easier for Hurd/kFreeBSD, but
it's not an absolute requirement for such a port to exist.

Ansgar

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Benda Xu]

Dear Simon,

Simon Richter  writes:

> The sanest thing we could do in Debian is to teach start-stop-daemon
> to parse systemd .service files and pull its command line arguments
> from there, so we could use service definitions as init scripts with a
> #! line.
>
> For that to happen, we'd have to define .service files as an API
> though, which would feature-freeze them, and I'm not sure the systemd
> people would be happy about that.

Thank you for sharing your thoughts.  I also think adding a
compatibility layer is the best way to move forward.

With more upstream packages providing systemd-.service by default, it is
more expansive for the systemd team to break the existing .service API.
We could consider it to be freezed in practice, or at least backward
compatible.

Yours,
Benda

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Anthony DeRobertis]

On 8/7/19 4:14 AM, Marc Haber wrote:

Imo, there should be a possibility in a systemd timer to switch on the
"old" output-to-e-mail behavior. This is probably something that
systemd upstream would never implement, so we'd end up with a wrapper
that is called by the systemd timer unit.



You can sort of do this with a few systemd units, at least for failed 
jobs using the unit's OnFailure= action. 
https://wiki.archlinux.org/index.php/Systemd/Timers#MAILTO has details. 
That'd probably be a nice thing to ship with systemd-cron.

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Vincent Bernat]

 ❦  8 août 2019 21:47 +02, Simon Richter :

>> inetd performance is very low because it needs to spawn one instance for
>> each connection. systemd socket activation has absolutely 0 overhead
>> except on the first connection (where systemd needs to start the
>> service).
>
> If you specify "wait" instead of "nowait" for a TCP service, your service
> is handed the listening socket, and can continue to accept more connections
> on that socket.
[...]

My bad, I didn't know that.

> This feature went unused not because noone thought of it, but because there
> is no real world use case that benefits from it. Either the service is used
> regularly, then it makes sense to keep the daemon in memory, or it isn't,
> then the reduced complexity for one-shot services (to the point where you
> can use a shell script as a service) is the benefit.

Reality seems different. Almost nothing was using inetd (tftpd is the
only daemon I have in mind), while many daemons adopted systemd socket
activation. For example, OpenSSH (optional), Docker (by default), CUPS
(by default), libvirt (by default, several services), Dovecot (by
default). It seems the key difference is that socket-activated services
are regular services. They can be started manually if we want them to be
and they inherit everything systemd is able to provide to services.
-- 
Localise input and output in subroutines.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Please stop hating on sysvinit, systemd… and let go of the hating anyway (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Martin Steigerwald]

Russ Allbery - 08.08.19, 20:33:58 CEST:
> Ondřej Surý  writes:
> > So, just to clarify…  so, it’s ok to hate systemd, but it’s not ok
> > to
> > hate sysvinit (spaghetti of shell scripts)?
> 
> Personally, I'd be happy if people would just stop hating on any free
> software in general.  Even buggy free software is someone's effort,
> released into the world hopefully for our benefit.  These
> conversations would be so much easier to have if they were framed in
> terms of costs and benefits, different use cases, and personal
> preferences, rather than making supposedly-objective statements
> grounded in strong personal opinions about the merits or
> worthlessness of some piece of software.

Wonderful, Russ.

Definitely a +1 from me.

What if… just what if if SysVInit and Systemd are just different and none 
of them is even inherently better than the other. What if… just what if 
this is similar to KDE and GNOME, Linux and FreeBSD, Amiga and Atari, 
Ruby and Python, Vi and Emacs?

What if, just what if I just let go of all the judgments? Just for now. 
And see what happens then.

And of cause I can say I like this or that due to this and that… but an 
apparent other can say I like that or this due to that or this. What if, 
just what if in the end its just arbitrary? Its just opinions. What if, 
just what if none of this is *the* truth?

I believe this mind set would help a lot to just agree to disagree and 
move on with life.

Thanks,
-- 
Martin

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Michael Stone]

On Thu, Aug 08, 2019 at 01:08:41PM -0700, Russ Allbery wrote:

Simon Richter  writes:


In the same way, we could have had automatic restart of services through
sysvinit easily: an include mechanism that allows additional inittab
lines to be pulled from /etc/inittab.d/* would be trivial to
implement. That it hasn't been done is not because no one has thought
about it in the last thirty years, but because its use is rather
limited.


Er, well... speaking as someone who tried to keep daemons running via
inittab and then gave up and tried using monit for a while, and then
switched to djb daemontools (all back before systemd existed), I'm not
sure this is quite right.  Those who tried to use this facility quickly
discovered that sysvinit was rather bad at it, and therefore started using
something else.


Yup. Regardless of those who keep insisting that systemd doesn't add 
anything to a server environment, people happily using it for servers 
disagree. I assume those who insist that sysvinit did everything right 
simply either forget the problems it had or just accepted its 
limitations.



Note that this is not a statement about Linux sysvinit specifically -- my
experiences were originally on Solaris.  The flaws in using inittab to
keep services running were universal among UNIX implementations.


And until we got rid of legacy limitations, there was no way forward 
because everything was met with "but this is how its always worked and 
you'll never manage to change things on those other systems".

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Russ Allbery]

Simon Richter  writes:

> In the same way, we could have had automatic restart of services through
> sysvinit easily: an include mechanism that allows additional inittab
> lines to be pulled from /etc/inittab.d/* would be trivial to
> implement. That it hasn't been done is not because no one has thought
> about it in the last thirty years, but because its use is rather
> limited.

Er, well... speaking as someone who tried to keep daemons running via
inittab and then gave up and tried using monit for a while, and then
switched to djb daemontools (all back before systemd existed), I'm not
sure this is quite right.  Those who tried to use this facility quickly
discovered that sysvinit was rather bad at it, and therefore started using
something else.

Problems included difficulty controlling respawn rates, sometimes-silent
permanent termination of the service when something went wrong, no support
for per-service fragments (easy to implement or not, no one implemented
it), weird behavior when daemons got into a crash loop, no good way to
externally control whether the service was temporarily disabled or not,
problems specifying and controlling start order, weird behavior with
standard input and output, and weirdness when changing the configuration.
djb started daemontools via a script in inittab and suggested just
rebooting the machine after initial configuration because getting init to
reload its configuration properly was too unpredictable.

Note that this is not a statement about Linux sysvinit specifically -- my
experiences were originally on Solaris.  The flaws in using inittab to
keep services running were universal among UNIX implementations.

systemd is *much* better at these things, as is runit, upstart, and
basically every other init system that came along after sysvinit.  This is
a very widely-known flaw in the sysvinit design and basically every
implementation that everyone else fixed.

-- 
Russ Allbery (r...@debian.org)   

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Simon Richter]

Hi,

On Thu, Aug 08, 2019 at 08:15:29PM +0200, Vincent Bernat wrote:

> inetd performance is very low because it needs to spawn one instance for
> each connection. systemd socket activation has absolutely 0 overhead
> except on the first connection (where systemd needs to start the
> service).

If you specify "wait" instead of "nowait" for a TCP service, your service
is handed the listening socket, and can continue to accept more connections
on that socket.

This feature went unused not because noone thought of it, but because there
is no real world use case that benefits from it. Either the service is used
regularly, then it makes sense to keep the daemon in memory, or it isn't,
then the reduced complexity for one-shot services (to the point where you
can use a shell script as a service) is the benefit.

In the same way, we could have had automatic restart of services through
sysvinit easily: an include mechanism that allows additional inittab lines
to be pulled from /etc/inittab.d/* would be trivial to implement. That it
hasn't been done is not because no one has thought about it in the last
thirty years, but because its use is rather limited.

SystemV init does not define service monitoring, because "the process still
exists" is a useless metric, and proper monitoring tests service
availability instead. There is no point in automatically restarting a
crashed service either -- a crash has a reason that needs to be
investigated. There is no service that is at the same time so important
that it must not go down under any circumstances, and at the same time so
unimportant that problems with it do not warrant human attention.

The most useful thing systemd provides to server operators is a shorthand
form for specifying some items in the process environment. Where an init
script needs to set up the environment in the right order (e.g. go into the
chroot before dropping permissions), having a descriptive language there
takes care of some of the complexity -- but that is functionally equivalent
to passing these values as arguments to start-stop-daemon.

All of these things have been proposed multiple times over the last thirty
years. They look useful on the outset, but the real-world use is limited.

   Simon

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Ondřej Surý]

I pretty much agree with everything you just said.

O.
--
Ondřej Surý 

> On 8 Aug 2019, at 20:34, Russ Allbery  wrote:
> 
> Ondřej Surý  writes:
> 
>> So, just to clarify…  so, it’s ok to hate systemd, but it’s not ok to
>> hate sysvinit (spaghetti of shell scripts)?
> 
> Personally, I'd be happy if people would just stop hating on any free
> software in general.  Even buggy free software is someone's effort,
> released into the world hopefully for our benefit.  These conversations
> would be so much easier to have if they were framed in terms of costs and
> benefits, different use cases, and personal preferences, rather than
> making supposedly-objective statements grounded in strong personal
> opinions about the merits or worthlessness of some piece of software.
> 
> Far, far beyond init systems, I wish we would be less denegrating and
> dismissive even of small pieces of software with obvious problems that
> people propose packaging.  We can say that something needs various
> improvements to make sense to include in Debian without hating on it.
> Maybe that sort of positive and thankful attitude would then carry over to
> larger and more controversial pieces of software.
> 
> (And yes, I wish upstreams would take that same advice, but I know how to
> preach to this audience and have less ability to preach to other
> audiences, so)
> 
> -- 
> Russ Allbery (r...@debian.org)   
> 

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Russ Allbery]

Ondřej Surý  writes:

> So, just to clarify…  so, it’s ok to hate systemd, but it’s not ok to
> hate sysvinit (spaghetti of shell scripts)?

Personally, I'd be happy if people would just stop hating on any free
software in general.  Even buggy free software is someone's effort,
released into the world hopefully for our benefit.  These conversations
would be so much easier to have if they were framed in terms of costs and
benefits, different use cases, and personal preferences, rather than
making supposedly-objective statements grounded in strong personal
opinions about the merits or worthlessness of some piece of software.

Far, far beyond init systems, I wish we would be less denegrating and
dismissive even of small pieces of software with obvious problems that
people propose packaging.  We can say that something needs various
improvements to make sense to include in Debian without hating on it.
Maybe that sort of positive and thankful attitude would then carry over to
larger and more controversial pieces of software.

(And yes, I wish upstreams would take that same advice, but I know how to
preach to this audience and have less ability to preach to other
audiences, so)

-- 
Russ Allbery (r...@debian.org)   

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Vincent Bernat]

 ❦  8 août 2019 19:10 +02, Simon Richter :

> For servers, the benefit is rather limited. There is no local user who
> makes system-wide policy decisions, and hardware is not changing
> dynamically either. The actual services provided are either implemented as
> daemons (i.e. not microservices), or proper microservices run inside a
> site-wide microservice framework such as Docker+Kubernetes, because a
> system-local framework such as systemd is too limited to express the
> dependencies. System boot time is significantly less important than service
> response time, so "socket activation" for daemons is not useful in this
> context either. We have had this for -literally- decades in the form of
> inetd, and it has not been widely used, precisely for that reason.

inetd performance is very low because it needs to spawn one instance for
each connection. systemd socket activation has absolutely 0 overhead
except on the first connection (where systemd needs to start the
service).
-- 
After all, all he did was string together a lot of old, well-known quotations.
-- H. L. Mencken, on Shakespeare


signature.asc
Description: PGP signature

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Alf Gaida]

On Thu, 8 Aug 2019 19:10:42 +0200
Simon Richter  wrote:
> For servers, the benefit is rather limited. There is no local user who
i don't agree - systemd just work™ in the most cases. Without changing
a bit.
> The "desktop" and "server" use cases are so vastly different that it
> doesn't make sense to try to squeeze both into a single design,
No. They aren't.
> It is way more useful to have multiple alternatives that fit their
> respective use cases well than one thing that attempts to do
> everything.
No.
> > So then the question boils down to what kind of feature
> > development sysvinit *in Debian* is willing to do to do that. If the
No, it don't. We need sysvinit for some non-linux things and some
enthusiasts that are not switched to devuan yet - it would be sane to
abandon sysvinit, kfreebsd and the hurd alltogether or split them out.
Only my point of view.
> The sanest thing we could do in Debian is to teach start-stop-daemon
> to parse systemd .service files and pull its command line arguments
No. The sanest thing would be to consider systemd as winning system and
improve it - and let sysv bitrot.

> For that to happen, we'd have to define .service files as an API
> though, which would feature-freeze them, and I'm not sure the systemd
> people would be happy about that.
No one would be happy about. Blocking upstream development or trying to
be more clever than upstream is generally a bad idea. And i really like
the idea.
> System-wide systemd-only services past the early boot stage are just
> bad design, do not currently exist and we shouldn't encourage their
> creation.
And why. elaborate it a bit more ...

Cheers Alf

PS: I'm not a systemd fanboi - i just use it - and i'm happy that it
goes out of my way most of the time - on desktop and servers. And i'm
an old fart, so i know the time before systemd well - to be honest, if
someone would turn back time i would abandon Linux and go back to some
sane system like Windows. I just don't want one minute on sysv anymore.
In Linux it is dead for the vast majority - and i'm thankful for.

-- 
Alf Gaida
BDBF C688 EFAD BA89 5A9F  464B CD28 0A0B 4D72 827C

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Simon Richter]

Hi,

On Thu, Aug 08, 2019 at 02:48:48PM +0200, Philipp Kern wrote:

> As a lot of the conflict between sysvinit and systemd was about the
> philosophy.

I wouldn't say "philosophy". These are different technical designs, and
each design has certain capabilities and limitations. It is not possible to
build a design that has the strengths of both but none of the weaknesses,
because you'd have to solve the halting problem for that.

For a desktop environment it makes a lot of sense to have a microservice
architecture such as the one that systemd provides. Typical desktop use
cases are "connect these Bluetooth headphones" and "allow the user on the
local console to select an active network card and configure settings".

The network device configuration case is a simple policy decision. The
logged-in user is the owner of the system and also the only stakeholder, so
in this case it makes sense to provide an interface that allows a
non-privileged user to perform a privileged action, limited by the
expressiveness of the interface. The network configuration API does not
allow setting up a complex routing table with multiple weighted
alternatives, because that is not part of the use case, and that is fine.

The bluetooth headset use case goes a bit beyond policy. There is still a
system-wide element (hardware access only for the local user), but most of
the software stack involved lives in userspace now and is orchestrated as
microservices, including a mechanism to topple the entire stack on error.
That is a sensible design choice -- if the user walks out of range with
their headset, there is no sensible way to provide audio service, and the
existing kernel audio APIs do not have an appropriate signaling mechanism.

For servers, the benefit is rather limited. There is no local user who
makes system-wide policy decisions, and hardware is not changing
dynamically either. The actual services provided are either implemented as
daemons (i.e. not microservices), or proper microservices run inside a
site-wide microservice framework such as Docker+Kubernetes, because a
system-local framework such as systemd is too limited to express the
dependencies. System boot time is significantly less important than service
response time, so "socket activation" for daemons is not useful in this
context either. We have had this for -literally- decades in the form of
inetd, and it has not been widely used, precisely for that reason.

The "desktop" and "server" use cases are so vastly different that it
doesn't make sense to try to squeeze both into a single design, because it
will not be good at either. There are other use cases that have
requirements that are not adequately serviced by either, for example
embedded systems or containers.

It is way more useful to have multiple alternatives that fit their
respective use cases well than one thing that attempts to do everything.

> So then the question boils down to what kind of feature
> development sysvinit *in Debian* is willing to do to do that. If the
> answer is "we really want the shell scripts as they have been since
> the beginning of time - and that is the scope of sysvinit" (which
> would not be true either, I know), then we cannot have that
> discussion either.

It is pretty close to the truth. The SystemV init design boils down to

 - there are two phases of system boot: kernel services (rcS.d) and user
   services (rc[1-5].d). User services can depend on kernel services.
 - configuration files are in an imperative language

The systemd design on the other hand:

 - there is a dependency tree of services that may depend on each other
 - configuration files are descriptive, with an exception for generators,
   which are imperative, but generate descriptive configuration fragments

The scope of sysvinit is to provide a minimal framework and then get out of
the way. Daemons can expect a working file system and to be able to create
listener sockets on INADDR_ANY or IN6ADDR_ANY. Anything beyond that is the
responsibility of the daemon.

The NTP daemon for some reason binds one socket per configured IP address,
so it needs to watch for network configuration events. Offloading that
responsibility into a common component would probably be possible, but
would not save any complexity because the main task of coordinating these
changes with the efforts to adjust the system clock would still be there.
As a result, it doesn't make any sense to make ntpd dependent on systemd
for its socket handling, and the only thing systemd is useful for is the
same thing that sysvinit also does: start the daemon some time after early
boot is complete.

The sanest thing we could do in Debian is to teach start-stop-daemon to
parse systemd .service files and pull its command line arguments from
there, so we could use service definitions as init scripts with a #! line.

For that to happen, we'd have to define .service files as an API though,
which would feature-freeze them, and I'm not sure the systemd people 

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Martin Steigerwald]

Philipp Kern - 08.08.19, 14:48:48 CEST:
> On 2019-08-08 14:43, Holger Levsen wrote:
> > On Thu, Aug 08, 2019 at 02:35:13PM +0200, Ondřej Surý wrote:
> >> And there’s the problem.  If we keep with sysvinit as a baseline of
> >> features provided by the init, we end up with just every init
> >> script
> >> having something like this: [...]
> > 
> > it seems several people in this thread have missed the fact, that
> > sysvinit in Debian is maintained well again, eg there have been 17
> > uploads of it so far in 2019, see
> > https://tracker.debian.org/pkg/sysvinit/news/
> > 
> > (so I think the above fixes could all be made in one central place.)
> 
> As a lot of the conflict between sysvinit and systemd was about the
> philosophy. So then the question boils down to what kind of feature
> development sysvinit *in Debian* is willing to do to do that. If the
> answer is "we really want the shell scripts as they have been since
> the beginning of time - and that is the scope of sysvinit" (which
> would not be true either, I know), then we cannot have that
> discussion either.
> 
> That's also to some degree why I think a solution to this problem is
> for the init diversity folks to figure out and we should not block on
> that. And that seems fine given the scope they have set for
> themselves.

I'd like to mention that people in the debian init diversity group not 
only work on sysvinit, but also on runit, elogind for example.

I believe there is not need to resolve the difference in philosophy. I 
switched all my systems to sysvinit meanwhile. One is on OpenRC. And so 
far I do not miss any features and enjoy the predictability. Probably 
only restarting services on failure – runit / s6 provide that. thinkfan 
fails sometimes for example, but with Systemd I did not even notice it. 
I prefer when whatever manages services to be invisible and I prefer 
policy to be in code I can easily review as an admin of my system.

So for me: No new features is actually a good thing. Software 
development got faster and faster and faster… but did things really get 
better? They for sure got more complex and more difficult to understand.

I bet it again it depends on different view points. And so there is a 
benefit to just agree to disagree. Sysvinit does not need to become like 
Systemd or vice versa.

Of course I do not object careful development of a feature here and 
there… and I am considering to switch to an init system that can restart 
services for my main laptop as well. Maybe runit, maybe… let's see. It 
has no urgent priority tough.

Thanks,
-- 
Martin

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Holger Levsen]

On Thu, Aug 08, 2019 at 02:48:48PM +0200, Philipp Kern wrote:
[...] 
> That's also to some degree why I think a solution to this problem is for the
> init diversity folks to figure out and we should not block on that. And that
> seems fine given the scope they have set for themselves.

I agree with this, also the unquoted part.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Philipp Kern]

On 2019-08-08 14:43, Holger Levsen wrote:

On Thu, Aug 08, 2019 at 02:35:13PM +0200, Ondřej Surý wrote:

And there’s the problem.  If we keep with sysvinit as a baseline of
features provided by the init, we end up with just every init script
having something like this: [...]


it seems several people in this thread have missed the fact, that
sysvinit in Debian is maintained well again, eg there have been 17
uploads of it so far in 2019, see
https://tracker.debian.org/pkg/sysvinit/news/

(so I think the above fixes could all be made in one central place.)


As a lot of the conflict between sysvinit and systemd was about the 
philosophy. So then the question boils down to what kind of feature 
development sysvinit *in Debian* is willing to do to do that. If the 
answer is "we really want the shell scripts as they have been since the 
beginning of time - and that is the scope of sysvinit" (which would not 
be true either, I know), then we cannot have that discussion either.


That's also to some degree why I think a solution to this problem is for 
the init diversity folks to figure out and we should not block on that. 
And that seems fine given the scope they have set for themselves.


Kind regards
Philipp Kern

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Holger Levsen]

On Thu, Aug 08, 2019 at 02:35:13PM +0200, Ondřej Surý wrote:
> And there’s the problem.  If we keep with sysvinit as a baseline of 
> features provided by the init, we end up with just every init script
> having something like this: [...]

it seems several people in this thread have missed the fact, that
sysvinit in Debian is maintained well again, eg there have been 17
uploads of it so far in 2019, see
https://tracker.debian.org/pkg/sysvinit/news/

(so I think the above fixes could all be made in one central place.)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Ondřej Surý]

> On 8 Aug 2019, at 14:08, Philipp Kern  wrote:
> 
> On 2019-08-08 13:47, Ondřej Surý wrote:
>>> Please stop hating on sysvinit
>> So, just to clarify…  so, it’s ok to hate systemd, but it’s not ok to
>> hate sysvinit (spaghetti of shell scripts)?
> 
> I don't think that's a constructive line of argument.

Maybe, but the email subject seemed too ironic to just let it be.

> At the same time it's not a race to the bottom in terms of features. I think 
> our baseline should be thinking in terms of the features of the default we 
> have.

And there’s the problem.  If we keep with sysvinit as a baseline of features 
provided by the init, we end up with just every init script having something 
like this:

do_tmpfiles() {
local type path mode user group age argument
if [ -r "$1" ]; then
if [ -x /bin/systemd-tmpfiles ]; then
/bin/systemd-tmpfiles --create "$1"
else
while read type path mode user group age argument; do
case "$type" in
d)
mkdir -p "$path";
chmod "$mode" "$path";
chown "$user:$group" "$path";
;;
\#*)
;;
*)
log_warning_msg "tmpfile.d type '$type' is not 
supported yet"
;;
esac
done < "$1"
fi
else
log_warning_msg "tmpfiles.d file '$1' doesn't exist or is not readable"
fi
}

> I don't have a great answer about the added maintenance cost that sysvinit 
> support puts on maintainers and thus they are rejecting certain changes. I 
> would like to say that I appreciate the work but personally not care, but I 
> have learned the hard way that just keeping things working is a ton of work. 
> And if you don't pay that cost stuff keeps rotting. Some of that could be 
> addressed with better integration testing. But at the same time it does not 
> answer the question on who pays the cost of rebasing changes, especially with 
> more upstream packages providing base services building more around reliable 
> systemd services. I feel like the answer is temporarily throwing out those 
> parts if needed or accept that they are broken until they get fixed, but that 
> might not be universally accepted, I guess.

The problem here that “sysvinit” or “systemd” are vague terms.  For some 
packages, sysvinit really means sysvrc, for other packages systemd means much 
more than just the service script (service activation, isolation, etc…) which 
is hard to mimick without systemd and friends.  For yet another group of 
packages (postfix, cyrus-imapd), it could have meant not having to develop own 
supervisor and just plug it into systemd.

All my packages still have sysvrc scripts, but it’s easy to dislike maintaining 
sysvinit script when you look at the difference:

   6 pkg-dns/bind9-9.14/debian/bind9.named.default
 145 pkg-dns/bind9-9.14/debian/bind9.named.init

vs

   6 pkg-dns/bind9-9.14/debian/bind9.named.default
  16 pkg-dns/bind9-9.14/debian/bind9.named.service
   1 pkg-dns/bind9-9.14/debian/bind9.named.tmpfile

And I can certainly understand why the maintainers would reject big patchset 
for policykit - the maintenance burden is not “hey, accept this patch”, but 
they would have to take care of the patchset every time something breaks, or 
there’s security issue.  Diverting too much from the upstream has 
non-negligible risks.

Ondrej
--
Ondřej Surý
ond...@sury.org

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Philipp Kern]

On 2019-08-08 13:47, Ondřej Surý wrote:

Please stop hating on sysvinit


So, just to clarify…  so, it’s ok to hate systemd, but it’s not ok to
hate sysvinit (spaghetti of shell scripts)?


I don't think that's a constructive line of argument. At the same time 
it's not a race to the bottom in terms of features. I think our baseline 
should be thinking in terms of the features of the default we have.


I don't have a great answer about the added maintenance cost that 
sysvinit support puts on maintainers and thus they are rejecting certain 
changes. I would like to say that I appreciate the work but personally 
not care, but I have learned the hard way that just keeping things 
working is a ton of work. And if you don't pay that cost stuff keeps 
rotting. Some of that could be addressed with better integration 
testing. But at the same time it does not answer the question on who 
pays the cost of rebasing changes, especially with more upstream 
packages providing base services building more around reliable systemd 
services. I feel like the answer is temporarily throwing out those parts 
if needed or accept that they are broken until they get fixed, but that 
might not be universally accepted, I guess.


Kind regards
Philipp Kern

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Jim Popovitch]

On Thu, 2019-08-08 at 13:47 +0200, Ondřej Surý wrote:
> So, just to clarify…  so, it’s ok to hate systemd, but it’s
> not ok to hate sysvinit (spaghetti of shell scripts)?
> 

One has a spaghetti of shell scripts, the other has a kimchi of log
commands and hidden config files.

I think "out of sight, out of mind" comes into play here.

-Jim P.

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Ondřej Surý]

> Please stop hating on sysvinit

So, just to clarify…  so, it’s ok to hate systemd, but it’s not ok to hate 
sysvinit (spaghetti of shell scripts)?

O.
--
Ondřej Surý
ond...@sury.org



> On 7 Aug 2019, at 15:44, Ian Jackson  wrote:
> 
> Marc Haber writes ("Re: do packages depend on lexical order or 
> {daily,weekly,monthly} cron jobs?"):
>> We have already thrown sysvinit away.
> 
> No, we have not.
> 
> https://tracker.debian.org/pkg/sysvinit
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=sysvinit#_2_4_5
> https://tracker.debian.org/pkg/insserv
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=insserv#_4_3_5
> https://www.chiark.greenend.org.uk/pipermail/debian-init-diversity/2019-July/date.html
> 
> Ian.
> 

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Alf Gaida]

On 07.08.19 19:00, Marc Haber wrote:

> Marc, who is trying to have neutral view on systemd and has managed to
> be seen as a fanboi by systemd haters and as a hater by the systemd
> community, and now fully expects the CoC to be used to be silenced


Marc, i don't hope so - you are not alone. For me systemd is one of the
best things ever happend to Linux. That doesn't mean that all people has
to be happy about. Long story short: I can't hear it anymore - there are
reasons why systemd exist and why it is superior over some other init
systems for some use cases. It might be an incident that all the my use
cases are covered today. And there will be other use cases that are not
covered yet or never will be covered.


I read in my early youth that the australians say: "Show me!" if they
don't belive in something - right o, for me and my needs systemd has
delivered.


Cheers Alf

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Philipp Kern]

On 2019-08-07 18:51, Jeremy Stanley wrote:

On 2019-08-07 10:19:00 +0200 (+0200), Marc Haber wrote:

On Mon, 05 Aug 2019 22:29:41 +0200, Philipp Kern wrote:

[...]

> I'd still expect a Cloud/Compute provider to offer default
> images in any case that could be preconfigured appropriately.

I am one of those customers who almost never uses the images
offered by the hoster whereever this is possible. Guilty as
charged, your honor.

[...]

As soon as you start depending on multiple providers to spread out
the risk for your workloads, it almost becomes a necessity to bring
your own images if you want to be able to build consistent systems
across disparate providers. Even if they haven't unnecessarily
tampered with official distro images themselves, there's no
guarantee that the Debian images they offer are for the same point
releases/snapshot dates and so on.


Yeah, and that's fair. But as you professionalize that arrangement and 
mature the offering, you also end up finding all these kinds of fixes 
yourself to optimize your infrastructure. Probably around the same time 
where you are thinking about your own Debian image release pipeline.


Although I have to say that I find it annoying that every sysadmin out 
there needs to learn these things (and also the whole "what are all the 
bits you need to remove from an image" from a different thread) the hard 
way. Yes, it's a way to make us all individually more valuable but 
maintaining something on best practices on how to maintain a Debian 
desktop, laptop, or server would be great.


Kind regards
Philipp Kern

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Adam Borowski]

On Wed, Aug 07, 2019 at 07:00:36PM +0200, Marc Haber wrote:
> On Wed, 7 Aug 2019 14:44:01 +0100, Ian Jackson
>  wrote:
> >Marc Haber writes ("Re: do packages depend on lexical order or 
> >{daily,weekly,monthly} cron jobs?"):
> >> We have already thrown sysvinit away.
> >
> >No, we have not.
> 
> We have given up on so many ideas that sysvinit has come with that it
> doesn't make sense to stick to Tradition on this count.

That would work better if fixes to non-systemd were not actively opposed. 
It's a wee bit nasty when for example a good working _tested_ patchset for
the utopia (policykit, etc) stack was prepared and ready in January 2018,
re-proposed multiple times but stonewalled every time, an unsanctioned NMU
was met with a call to demote the NMUer from DD status (because "hostile"),
there was a last-minute demand to change the approach (from policykit
linking to multiple backends to ABI-compatible alternative to libsystemd),
which was then rejected as too intrusive; this led to GUIs in Buster having
been released in a non-usable state (many packages non-instalable,
shutdown/etc not working, etc), and even now src:elogind is kept from
testing for not implementing some journald functionality (there's no
journal...).

(I'm sitting outside some hotel on the wrong continent mooching their WiFi
thus I can't investigate the last part at the moment, but...)


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Have you accepted Khorne as your lord and saviour?
⠈⠳⣄

Re: Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Marc Haber]

On Wed, 7 Aug 2019 14:44:01 +0100, Ian Jackson
 wrote:
>Marc Haber writes ("Re: do packages depend on lexical order or 
>{daily,weekly,monthly} cron jobs?"):
>> We have already thrown sysvinit away.
>
>No, we have not.

We have given up on so many ideas that sysvinit has come with that it
doesn't make sense to stick to Tradition on this count.

Greetings
Marc, who is trying to have neutral view on systemd and has managed to
be seen as a fanboi by systemd haters and as a hater by the systemd
community, and now fully expects the CoC to be used to be silenced
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Jeremy Stanley]

On 2019-08-07 10:19:00 +0200 (+0200), Marc Haber wrote:
> On Mon, 05 Aug 2019 22:29:41 +0200, Philipp Kern wrote:
[...]
> > I'd still expect a Cloud/Compute provider to offer default
> > images in any case that could be preconfigured appropriately.
> 
> I am one of those customers who almost never uses the images
> offered by the hoster whereever this is possible. Guilty as
> charged, your honor.
[...]

As soon as you start depending on multiple providers to spread out
the risk for your workloads, it almost becomes a necessity to bring
your own images if you want to be able to build consistent systems
across disparate providers. Even if they haven't unnecessarily
tampered with official distro images themselves, there's no
guarantee that the Debian images they offer are for the same point
releases/snapshot dates and so on.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Please stop hating on sysvinit (was Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?)

[Ian Jackson]

Marc Haber writes ("Re: do packages depend on lexical order or 
{daily,weekly,monthly} cron jobs?"):
> We have already thrown sysvinit away.

No, we have not.

https://tracker.debian.org/pkg/sysvinit
https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=sysvinit#_2_4_5
https://tracker.debian.org/pkg/insserv
https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=insserv#_4_3_5
https://www.chiark.greenend.org.uk/pipermail/debian-init-diversity/2019-July/date.html

Ian.

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Ian Jackson]

Simon McVittie writes ("Re: do packages depend on lexical order or 
{daily,weekly,monthly} cron jobs?"):
> Trying to make this point more clearly because you seem to be confusing
> systemd with systemd-cron:

You are completely right.  I was not aware that systemd-cron was not
part of systemd.

> systemd is entirely capable of running Vixie cron as a daemon (as is done
> in default installations of stretch and buster), or running cronie as
> a daemon. systemd-cron is an alternative to those, using systemd rather
> than a separate daemon as the long-running component that waits for the
> right time.

Right.  And as you say it is therefore systemd-cron whose behaviour is
at issue, not systemd's.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Marc Haber]

On Mon, 05 Aug 2019 22:29:41 +0200, Philipp Kern 
wrote:
>With a systemd timer you can declare conflicts as well as a 
>lineralization if so needed.

How would that be configured? Should all timers generated from
cron.fooly be grouped together (in a target?) and conflict with that
very target?

>I also question the "knowledgeable people will not have too much 
>trouble". Export state as granular as possible and there is no guesswork 
>required. I have no doubt that my co-workers can do this. But I want 
>their life to be as easy as possible.

Amen.

>Similarly I wonder what the external monitoring should be apart from 
>injecting fake jobs around every run-parts unit in this case. Replacing 
>run-parts with something monitoring-aware? Then why not take the tool 
>that already exists (systemd)?

I think Ian is referring to mechanisms (that I have actually seen)
that automatically process incoming e-mails sent from cron in a
central place and create alerts or tickets.

>And finally, the load spikes: Upthread it was mentioned that 
>RandomizedDelaySec exists. Generally this should be sufficient to even s 
>out such effects. I understand that there is a case where you run a lot 
>of unrelated VMs that you cannot control. In other cases, like laptops 
>and desktops, it is very likely much more efficient to generate the load 
>spike and complete the task as fast as possible in order to return to 
>the low-power state of (effectively) waiting for input. I suspect that 
>there is a conflict between the two that could be dealt with by 
>encouraging liberal use of DefaultTimerAccuracySec on the system-level. 

Amen.

>I understand that Debian inherently does not distinguish between the two 
>cases. I'd still expect a Cloud/Compute provider to offer default images 
>in any case that could be preconfigured appropriately.

I am one of those customers who almost never uses the images offered
by the hoster whereever this is possible. Guilty as charged, your
honor.

>I apologize that I think of this in terms of systemd primitives. But the 
>tool was written for a reason and a lot of thought went into it.

I agree with that, no apology needed, imo.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Marc Haber]

On Mon, 5 Aug 2019 16:34:18 +0100, Ian Jackson
 wrote:
>So it seems to me that there are the following options for systemd
>users:
>
>A. Continue to use run-parts.
>
>   Disadvantages: Bundles the output together.
>   Doesn't provide individual status.
>
>   Advantages: No work needed.
>
>B. Run each script as a single systemd timer unit.
>
>   Disadvantages: Runs scripts in parallel (causing load spikes and
>   other undesirable performance effects).  Imposes un-auditable and
>   undebuggable new concurrency/ordering requirements on scripts (that
>   is, scripts must cope with concurrent running and in any order).
>   Ie, effectively, exposes systemd users to new bugs.
>
>   Advantages: Unbundled output and individual status.

In fact, systemd-cron already has an issue open upstream to change
this behavior, see
https://github.com/systemd-cron/systemd-cron/issues/47

I have taken the liberty of commenting a pointer to this discussion
into the upstream issue and also making my case to make tihs
configurable.

>C. Provide a feature in systemd to do what is needed.
>   Advantages: Everything works properly.
>   Disadvantage: Requires writing new code to implement the feature.

Imo, there should be a possibility in a systemd timer to switch on the
"old" output-to-e-mail behavior. This is probably something that
systemd upstream would never implement, so we'd end up with a wrapper
that is called by the systemd timer unit.

>D. Provide a version of run-parts which does per-script reporting.
>   Advantages: Provides the benefits of (c) to sysvinit/cron users
>   too.  Disadvantages: Requires design of a new reporting protocol,
>   and implementation of that protocol in run-parts; to gain the
>   benefits for systemd users, must be implemented in systemd too.
>   (Likewise for cron users.)

We have already thrown sysvinit away. Things are exactly like I
predicted, the init scripts are rotting away quickly. I think we
should stop wasting personpower for sysvinit support, people not
liking systemd have gone to devuan anyway.

>With current code the options are:
>
>A. Things run in series but with concatenated output and no individual
>   status.
>
>B. Things run in parallel, giving load spikes and possible concurrency
>   bugs; vs.
>
>I can see few people who would choose (B).

>From an individual system's POV, with RandomizeDelay the load would be
higher, granted, but from a virtualization environment's POV the load
spike would greatly be evened out when the VMs don't invoke their
run-parts in the same second. I know virtualization operators who call
the 06:25 peak in their load "The Debian peak".

I would choose Option B in all my use cases, with different
parameters: As a server jockey, I'd use a high RandomizeDelay to even
out the impact on the infrastructure, and on my personal laptop, I'd
probably use a very small RandomizeDelay so that the box can return to
power-save idle faster.

What keeps me from doing this now is the lack of e-mail output.

>People who don't care much about paying attention to broken cron
>stuff, or people who wouldn't know how to fix it, are better served by
>(A).  It provides a better experience.

I disagree.

>Knowledgeable people will not have too much trouble interpreting
>combined output, and maybe have external monitoring arrangements
>anyway.  Conversely, heisenbugs and load spikes are still undesirable.
>So they should also choose (A).

I disagree.

>IOW reliability and proper operation is more important than separated
>logging and status reporting.

I would choose B to help in debugging the scripts. Better force those
errors now to have them ironed out for bullseye. The current behavior
of run-parts doing things in lexical order and serialized is basically
a historically caused coincidence.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Bill Allombert]

On Tue, Aug 06, 2019 at 05:19:14PM +0200, Philipp Kern wrote:
> On 2019-08-06 13:43, Bill Allombert wrote:
> > On Mon, Aug 05, 2019 at 10:29:41PM +0200, Philipp Kern wrote:
> > > And finally, the load spikes: Upthread it was mentioned that
> > > RandomizedDelaySec exists. Generally this should be sufficient to
> > > even out
> > > such effects. I understand that there is a case where you run a lot of
> > > unrelated VMs that you cannot control. In other cases, like laptops
> > > and
> > > desktops, it is very likely much more efficient to generate the load
> > > spike
> > > and complete the task as fast as possible in order to return to the
> > > low-power state of (effectively) waiting for input.
> > 
> > This assumes the system has enough RAM and other ressources to complete
> > all the jobs in parallel.
> 
> We are talking about cron.*. I feel like that's more of a scheduling
> problem. If you run a constrained machine, I'm sure you are already doing
> customization to accommodate to the shortage and you would not want the
> background cron jobs to kill your production workload?

One background cron job at once might be OK, all at once less so, even
on a full server.

Some of the cron jobs are IO intensive, some other RAM intensive
(think updatedb, Web indexing software like xapian-omega etc).

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Sam Hartman]

> "Ian" == Ian Jackson  writes:


I think this is the big question.

Because of modern thinking about power optimization, if I were designing
this interface today I'd design it to be concurrent.
In particular, you want to use as much concurrency as you can to
maximize utilization of CPU and IO without introducing unnecessary
resource thrashing.

We've generally concluded that short periods of very high activity are
better in terms of power utilization than longer periods of moderate
activity.


So, I think this feature request is coming about exactly because some
people desire the concurrency and are questioning whether the classical
interface design still meets modern needs.

Of course we can continue to provide the same semantics.  That's what
systemd-cron does today.

The interesting question here is whether that's actually desirable.

Starting by presuming that it is sidesteps the entire discussion.

--Sam

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Simon McVittie]

On Tue, 06 Aug 2019 at 16:58:31 +0100, Ian Jackson wrote:
> The Debian systemd maintainers are of course free to enable
> parallelism for systemd users;

Trying to make this point more clearly because you seem to be confusing
systemd with systemd-cron:

Whether to run cron jobs in lexical order or in parallel is not a systemd
decision, because systemd does not read or understand cron jobs. The
component that reads and interprets cron jobs and has a feature request
involving potentially running {daily,etc.} cron jobs in parallel is
systemd-cron, a third-party project that translates cron jobs into
systemd units.

systemd is entirely capable of running Vixie cron as a daemon (as is done
in default installations of stretch and buster), or running cronie as
a daemon. systemd-cron is an alternative to those, using systemd rather
than a separate daemon as the long-running component that waits for the
right time.

If there are flaws in how systemd-cron translates cron jobs into systemd
units, then those are bugs to be fixed in systemd-cron, not in systemd.

(In other words, this is one of those systems involving composing smaller
modules that people sometimes say systemd opposes or makes impossible.)

smcv

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Ian Jackson]

Philipp Kern writes ("Re: do packages depend on lexical order or 
{daily,weekly,monthly} cron jobs?"):
> On 2019-08-05 17:34, Ian Jackson wrote:
> > With current code the options are:
> > 
> > A. Things run in series but with concatenated output and no individual
> >status.
> > 
> > B. Things run in parallel, giving load spikes and possible concurrency
> >bugs; vs.
> > 
> > I can see few people who would choose (B).
...
> > IOW reliability and proper operation is more important than separated
> > logging and status reporting.
> 
> If we are in agreement that concurrency must happen with proper locking 
> and not depend on accidental lineralization then identifying those 
> concurrency bugs is actually a worthwhile goal in order to achieve 
> reliability, is it not?

Given that the concurrency is not actually desirable, it seems foolish
to change the execution model to a concurrent one and thereby impose
a lot of additional requirements on these scripts.

Obviously I agree that bugs, where there are bugs, should be fixed.

However, I do not agree that because bugs should be fixed,
specifications should be changed to declare buggy, things which are
currently not buggy at all.

> I thought you would be the first to acknowledge 
> that bugs are worth fixing rather than sweeping them under the rug.

I confess I was quite annoyed when I read this the first time.  It
still irritates me.

I am not suggesting that anything should be "swept under the rug".
The existing scripts, which do not have any locking, are completely
correct.  They are entitled to make the assumptions that they
currently do, because that is what the specification says and because
that is how the current callers work.

Concurrency in particular is a particularly awkward thing to retrofit,
because code which was not previously concurrent suddenly becomes
unauditably buggy, and because concurrency bugs are very hard to find,
analyse and fix.

Sometimes it is necessary to retrofit concurrency for performance
reasons.  We have suffered a lot of buggy software as a result.  But
in this case concurrency is simply harmful and should be avoided.

>  We already identified that parallelism between the various stages
> is undesirable. With a systemd timer you can declare conflicts as
> well as a lineralization if so needed.

ISTM that a simple solution is for systemd to infer the appropriate
linearisation.  I don't understand why this approach is apparently
being rejected.

> And finally, the load spikes: Upthread it was mentioned that 
> RandomizedDelaySec exists. Generally this should be sufficient to even 
> out such effects.

A predictable scheduler which runs all the jobs in the same order each
time, sequentially, and without gaps, is clearly better than one which
runs them at random times and hopes that the resulting schedule is a
good one.


My bottom line:

The Debian systemd maintainers are of course free to enable
parallelism for systemd users; notwithstanding that everyone
apparently acknowledges that parallelism here is undesirable.

But it would be wrong to change the Debian Policy specification for
cron.fooly script execution.  There is nothing stopping systemd from
implementing the existing specification; and, there is nothing in this
specification which prevents systemd from providing whatever logging
and monitoring features are desired.
i
And there is nothing wrong with the current specification.  The only
difficulty is that systemd doesn't currently implement it to the
systemd maintainers' satisfaction.

So any concurrency bugs (or other infelicities) resulting from
violations by systemd of the currently defined interface, should be
seen as bugs in that hypothetical concurrent systemd cron.fooly
arrangements, not bugs in the cron.fooly scripts provided by packages.

The solution is to improve systemd, not to change policy to divert the
blame for any bugs caused by anyway-unwanted concurrency.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Philipp Kern]

On 2019-08-06 13:43, Bill Allombert wrote:

On Mon, Aug 05, 2019 at 10:29:41PM +0200, Philipp Kern wrote:

And finally, the load spikes: Upthread it was mentioned that
RandomizedDelaySec exists. Generally this should be sufficient to even 
out

such effects. I understand that there is a case where you run a lot of
unrelated VMs that you cannot control. In other cases, like laptops 
and
desktops, it is very likely much more efficient to generate the load 
spike

and complete the task as fast as possible in order to return to the
low-power state of (effectively) waiting for input.


This assumes the system has enough RAM and other ressources to complete
all the jobs in parallel.


We are talking about cron.*. I feel like that's more of a scheduling 
problem. If you run a constrained machine, I'm sure you are already 
doing customization to accommodate to the shortage and you would not 
want the background cron jobs to kill your production workload?


Apart from the fact that systemd actually lets you limit the resource 
usage/declare what to prioritize. If you go the route of separate timers 
even on a per script basis.


Kind regards
Philipp Kern

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Bill Allombert]

On Mon, Aug 05, 2019 at 10:29:41PM +0200, Philipp Kern wrote:
> And finally, the load spikes: Upthread it was mentioned that
> RandomizedDelaySec exists. Generally this should be sufficient to even out
> such effects. I understand that there is a case where you run a lot of
> unrelated VMs that you cannot control. In other cases, like laptops and
> desktops, it is very likely much more efficient to generate the load spike
> and complete the task as fast as possible in order to return to the
> low-power state of (effectively) waiting for input.

This assumes the system has enough RAM and other ressources to complete
all the jobs in parallel.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Philipp Kern]

On 2019-08-05 17:34, Ian Jackson wrote:

With current code the options are:

A. Things run in series but with concatenated output and no individual
   status.

B. Things run in parallel, giving load spikes and possible concurrency
   bugs; vs.

I can see few people who would choose (B).

People who don't care much about paying attention to broken cron
stuff, or people who wouldn't know how to fix it, are better served by
(A).  It provides a better experience.

Knowledgeable people will not have too much trouble interpreting
combined output, and maybe have external monitoring arrangements
anyway.  Conversely, heisenbugs and load spikes are still undesirable.
So they should also choose (A).

IOW reliability and proper operation is more important than separated
logging and status reporting.


If we are in agreement that concurrency must happen with proper locking 
and not depend on accidental lineralization then identifying those 
concurrency bugs is actually a worthwhile goal in order to achieve 
reliability, is it not? I thought you would be the first to acknowledge 
that bugs are worth fixing rather than sweeping them under the rug. We 
already identified that parallelism between the various stages is 
undesirable. With a systemd timer you can declare conflicts as well as a 
lineralization if so needed.


I also question the "knowledgeable people will not have too much 
trouble". Export state as granular as possible and there is no guesswork 
required. I have no doubt that my co-workers can do this. But I want 
their life to be as easy as possible.


Similarly I wonder what the external monitoring should be apart from 
injecting fake jobs around every run-parts unit in this case. Replacing 
run-parts with something monitoring-aware? Then why not take the tool 
that already exists (systemd)?


And finally, the load spikes: Upthread it was mentioned that 
RandomizedDelaySec exists. Generally this should be sufficient to even 
out such effects. I understand that there is a case where you run a lot 
of unrelated VMs that you cannot control. In other cases, like laptops 
and desktops, it is very likely much more efficient to generate the load 
spike and complete the task as fast as possible in order to return to 
the low-power state of (effectively) waiting for input. I suspect that 
there is a conflict between the two that could be dealt with by 
encouraging liberal use of DefaultTimerAccuracySec on the system-level. 
I understand that Debian inherently does not distinguish between the two 
cases. I'd still expect a Cloud/Compute provider to offer default images 
in any case that could be preconfigured appropriately.


I apologize that I think of this in terms of systemd primitives. But the 
tool was written for a reason and a lot of thought went into it.


Kind regards
Philipp Kern

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Ian Jackson]

Marc Haber writes ("Re: do packages depend on lexical order or 
{daily,weekly,monthly} cron jobs?"):
> On Mon, 29 Jul 2019 18:42:34 +0100, Simon McVittie 
> wrote:
> >On Wed, 24 Jul 2019 at 20:14:22 +0200, Marc Haber wrote:
> >> Scripts with such dependencies will probably fail miserably on systems
> >> that are using systemd-cron instead of one of the "classic" cron
> >> packaes
> >
> >I thought so too, but they don't: systemd-cron uses run-parts for
> >cron.{hourly,etc.} too.
> 
> ... this imports many disadvantages of the old scheme into the new
> world. Philipp has explained in this thread very well.

Right.  That makes sense.

So it seems to me that there are the following options for systemd
users:

A. Continue to use run-parts.

   Disadvantages: Bundles the output together.
   Doesn't provide individual status.

   Advantages: No work needed.

B. Run each script as a single systemd timer unit.

   Disadvantages: Runs scripts in parallel (causing load spikes and
   other undesirable performance effects).  Imposes un-auditable and
   undebuggable new concurrency/ordering requirements on scripts (that
   is, scripts must cope with concurrent running and in any order).
   Ie, effectively, exposes systemd users to new bugs.

   Advantages: Unbundled output and individual status.

C. Provide a feature in systemd to do what is needed.
   Advantages: Everything works properly.
   Disadvantage: Requires writing new code to implement the feature.

D. Provide a version of run-parts which does per-script reporting.
   Advantages: Provides the benefits of (c) to sysvinit/cron users
   too.  Disadvantages: Requires design of a new reporting protocol,
   and implementation of that protocol in run-parts; to gain the
   benefits for systemd users, must be implemented in systemd too.
   (Likewise for cron users.)

I was wrong when earlier I wrote that sysvinit users would suffer any
significant trouble, regardless what option above is picked.  The only
negative impact of (B) on sysvinit users is that maybe the cron
scripts they are running become more complex and fragile, in order to
try to satisfy the new concurrency requirement.

I would be interested in (D) if you thought it would be worthwhile.
Maybe the subunit v1 protocol or something :-).

> Maybe systemd-cron could be extended to be locally configurable
> whether to use run-parts, keeping the old semantics, or to generate
> individual timers.

There would be question of the default.

With current code the options are:

A. Things run in series but with concatenated output and no individual
   status.

B. Things run in parallel, giving load spikes and possible concurrency
   bugs; vs.

I can see few people who would choose (B).

People who don't care much about paying attention to broken cron
stuff, or people who wouldn't know how to fix it, are better served by
(A).  It provides a better experience.

Knowledgeable people will not have too much trouble interpreting
combined output, and maybe have external monitoring arrangements
anyway.  Conversely, heisenbugs and load spikes are still undesirable.
So they should also choose (A).

IOW reliability and proper operation is more important than separated
logging and status reporting.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Marc Haber]

On Mon, 29 Jul 2019 18:42:34 +0100, Simon McVittie 
wrote:
>On Wed, 24 Jul 2019 at 20:14:22 +0200, Marc Haber wrote:
>> Scripts with such dependencies will probably fail miserably on systems
>> that are using systemd-cron instead of one of the "classic" cron
>> packaes
>
>I thought so too, but they don't: systemd-cron uses run-parts for
>cron.{hourly,etc.} too.

... this imports many disadvantages of the old scheme into the new
world. Philipp has explained in this thread very well.

Maybe systemd-cron could be extended to be locally configurable
whether to use run-parts, keeping the old semantics, or to generate
individual timers.

systemd itself would need a means to group different timers together
with the constraint that at most X of those can run in parallel.

>On Sun, 28 Jul 2019 at 08:47:49 +0200, Marc Haber wrote:
>> Setting RandomizeDelaySecs sufficiently high on our daily jobs would
>> probably help to chop off the load pike that especially virtualization
>> setups running many Debian instances suffer from at 06:25 or 07:35. I
>> think this could be a net gain worth pursuing.
>
>If you want this, the way to opt-in to it is currently to add a native
>systemd timer, and make the traditional cron script a no-op on systems
>running systemd (to avoid doing the periodic task twice). For example,
>apt and man-db have both done this.

... which takes away flexibility and standardization from the local
admin, adding more things to think about when developing larger
infrastructures. I am not sure whether I like that.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Colin Watson]

On Mon, Jul 29, 2019 at 06:36:00PM +0100, Simon McVittie wrote:
> On Mon, 29 Jul 2019 at 18:26:22 +0100, Simon McVittie wrote:
> > Another reason to break them up is that if the same package installs
> > both a cron script like /etc/cron.daily/man-db and a systemd unit like
> > /lib/systemd/system/man-db.timer, systemd-cron deliberately generates its
> > unit to be named man-db.timer, so that it will be "shadowed" by the native
> > man-db.timer (which is in a higher-precedence directory).
> 
> As a result, systemd-cron doesn't currently do this for daily, etc. cron
> jobs (but it does for cron.d jobs).

It's also a little hard to see how it could rely on shadowing for this,
since it's permitted for e.g. /etc/cron.daily/foo and
/etc/cron.weekly/foo to both exist and do different things.

(This is the case for man-db.  When adding its systemd unit the
submitter of #858022 decided and I agreed that the differences between
the daily and weekly scripts weren't worth representing, and I might yet
change the traditional cron.* scripts to match that decision, but this
wasn't a decision that an automatic conversion system could have taken.)

-- 
Colin Watson   [cjwat...@debian.org]

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Simon McVittie]

On Wed, 24 Jul 2019 at 20:14:22 +0200, Marc Haber wrote:
> Scripts with such dependencies will probably fail miserably on systems
> that are using systemd-cron instead of one of the "classic" cron
> packaes

I thought so too, but they don't: systemd-cron uses run-parts for
cron.{hourly,etc.} too. It only generates individual, parallelizable
systemd timers for scripts in cron.d, which in general run on their own
independent schedules anyway.

On Sun, 28 Jul 2019 at 08:47:49 +0200, Marc Haber wrote:
> Setting RandomizeDelaySecs sufficiently high on our daily jobs would
> probably help to chop off the load pike that especially virtualization
> setups running many Debian instances suffer from at 06:25 or 07:35. I
> think this could be a net gain worth pursuing.

If you want this, the way to opt-in to it is currently to add a native
systemd timer, and make the traditional cron script a no-op on systems
running systemd (to avoid doing the periodic task twice). For example,
apt and man-db have both done this.

smcv

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Simon McVittie]

On Mon, 29 Jul 2019 at 18:26:22 +0100, Simon McVittie wrote:
> On Sun, 28 Jul 2019 at 14:07:59 +0100, Ian Jackson wrote:
> > Why can't systemd run cron.fooly as one big timer job rather than one
> > timer job for each script ?
> 
> Of course it could, but perhaps it shouldn't.
> 
> To avoid ambiguity, this is not a systemd design decision: it's a design
> decision of systemd-cron

Correction: at the moment, systemd-cron *does* run cron.daily, etc. as
one big timer/service pair per interval, using essentially the same
run-parts call as for traditional cron.

If scripts rely on lexical order, then it will have to continue to do this,
and maintainers who want to opt-in to having their periodic script run
outside that sequence on systemd systems will have to write native timer
units.

(Conversely, if scripts don't rely on lexical order, then systemd-cron
would be free to schedule them separately if the trade-offs around
race-to-idle, predictable load spikes, etc. make that desirable.)

> Another reason to break them up is that if the same package installs
> both a cron script like /etc/cron.daily/man-db and a systemd unit like
> /lib/systemd/system/man-db.timer, systemd-cron deliberately generates its
> unit to be named man-db.timer, so that it will be "shadowed" by the native
> man-db.timer (which is in a higher-precedence directory).

As a result, systemd-cron doesn't currently do this for daily, etc. cron
jobs (but it does for cron.d jobs).

smcv

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Simon McVittie]

On Sun, 28 Jul 2019 at 14:07:59 +0100, Ian Jackson wrote:
> Why can't systemd run cron.fooly as one big timer job rather than one
> timer job for each script ?

Of course it could, but perhaps it shouldn't.

To avoid ambiguity, this is not a systemd design decision: it's a design
decision of systemd-cron, a separate package that reads cron jobs and
writes pairs of systemd timer and service units that result in systemd
doing approximately the same things (ana)cron would have done under the
same configuration.

systemd itself does not have cron integration, so packages with
important cron jobs must depend on some cron-daemon implementation. In
current Debian that means either Vixie cron, cronie or bcron, which are
independent daemons, or systemd-cron.

As Philipp Kern pointed out, one reason to break them up is to attribute
log messages and failures to the actual task (popularity-contest.service
or whatever), rather than to some larger unit like cron-weekly.service.
systemd-cron writes cron job output to the system log, rather than sending
it by email like traditional cron implementations, but the general
principle is similar: you can prioritize more easily if the report is
"cron.daily/man-db failed" than if the report is "cron.daily failed".

Another reason to break them up is that if the same package installs
both a cron script like /etc/cron.daily/man-db and a systemd unit like
/lib/systemd/system/man-db.timer, systemd-cron deliberately generates its
unit to be named man-db.timer, so that it will be "shadowed" by the native
man-db.timer (which is in a higher-precedence directory). This avoids
log noise (and inefficiency, but the log noise is more important) from
/etc/cron.daily/man-db starting, noticing that this is a systemd system,
and exiting early so that it won't duplicate the timer unit. This is the
same "shadowing" mechanism that ensures that if a package has a systemd
service dbus.service and an init script /etc/init.d/dbus, then the init
script will be skipped. If every script in cron.daily is shadowed in
this way, no daily systemd-cron timers will be scheduled at all.

systemd-cron *could* generate one big timer unit that activates a "target"
(a group of services), generate a separate service per script as part of
that target, chain them together in lexical order by building a linked
list of Before or After directives, and implement the "don't schedule
this one if there's a native timer" logic itself instead of relying
on shadowing - but right now it doesn't, and what it does now is a lot
simpler. Introducing more complexity might be a necessary evil if real
cron jobs rely on it, but it seems better to avoid that if possible.

smcv

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Philipp Kern]

On 2019-07-28 15:07, Ian Jackson wrote:

Marc Haber writes ("Re: do packages depend on lexical order or
{daily,weekly,monthly} cron jobs?"):

On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
>I worry about additional concurrency.  Unlike ordering bugs,
>concurrency bugs are hard to find by testing.  So running these
>scripts in parallel is risky.
>
>And, I think running cron.fooly scripts in parallel is a bad idea.
>The objective is to run them "at some point", not to get it done as
>soon as possible.  Running them in sequence will save electricity,
>may save wear on components, and will reduce overall impact on other
>uses of the same system.

I fully agree with that. However, moving away from "in sequence" thing
would greatly ease the migration to systemd timers, making it easier
to get away without crond on many systems.


Why can't systemd run cron.fooly as one big timer job rather than one
timer job for each script ?


Of course it could. But it's better for the administrator to export 
state as granular as possible. That way the exit codes are exported into 
global systemd state and you can see exactly what is failing rather than 
a generic "something in cron.$foo failed, good luck finding the right 
logs". systemd also gives you the logs per timer unit, rather than in 
bulk, so the error is trivially visible rather than filtering a long log 
for what went wrong.


Think of this as an alert condition: I want to know if, say, 
popularity-contest failed and treat that with lower urgency than, say, 
debsums. One is a goodie and one is evidence of potential disk 
corruption. If the "cron.daily" script failed, I don't know if this has 
a particular urgency until I crawled the logs.



Obviously, I don't think it is a good idea to break this for
non-systemd users because of difficulties making it work properly
with systemd.  Perhaps I have misunderstood you ?


To be honest, that's something that the compatibility/init diversity 
folks then need to figure out.


Kind regards
Philipp Kern

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Ian Jackson]

Marc Haber writes ("Re: do packages depend on lexical order or 
{daily,weekly,monthly} cron jobs?"):
> On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
> >I worry about additional concurrency.  Unlike ordering bugs,
> >concurrency bugs are hard to find by testing.  So running these
> >scripts in parallel is risky.
> >
> >And, I think running cron.fooly scripts in parallel is a bad idea.
> >The objective is to run them "at some point", not to get it done as
> >soon as possible.  Running them in sequence will save electricity,
> >may save wear on components, and will reduce overall impact on other
> >uses of the same system.
> 
> I fully agree with that. However, moving away from "in sequence" thing
> would greatly ease the migration to systemd timers, making it easier
> to get away without crond on many systems.

Why can't systemd run cron.fooly as one big timer job rather than one
timer job for each script ?

That leaves cron.fooly running in parallel with cron.barly but that is
something these jobs have to cope with anyway ?

> I am wondering whether this is something we should think about giving
> up in future. We have given up so many things since we moved to
> systemd, would it be worth throwing this out of the window as well?

Obviously, I don't think it is a good idea to break this for
non-systemd users because of difficulties making it work properly
with systemd.  Perhaps I have misunderstood you ?

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Bernd Zeimetz]

On 7/28/19 8:47 AM, Marc Haber wrote:
> Setting RandomizeDelaySecs sufficiently high on our daily jobs would
> probably help to chop off the load pike that especially virtualization
> setups running many Debian instances suffer from at 06:25 or 07:35. I
> think this could be a net gain worth pursuing.

definitely. we are able to see the daily logrotate/ run in our power
usage statistics...


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Marc Haber]

On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
 wrote:
>Marc Haber writes ("do packages depend on lexical order or 
>{daily,weekly,monthly} cron jobs?"):
>> Do you people know about packages that have cron jobs that depend on
>> execution order? Would it probably be a good idea to proactively put
>> language into the Policy Manual Chapter 9.5 that scripts put into the
>> /etc/cron.{daily,weekly,monthly} MUST NOT depend on any execution
>> order and MUST be prepared to run in parallel?
>
>"Does not depend on order" and "can be run in parallel" are not the
>same. 

Right. I didn't mean to imply that.

>I worry about additional concurrency.  Unlike ordering bugs,
>concurrency bugs are hard to find by testing.  So running these
>scripts in parallel is risky.
>
>And, I think running cron.fooly scripts in parallel is a bad idea.
>The objective is to run them "at some point", not to get it done as
>soon as possible.  Running them in sequence will save electricity,
>may save wear on components, and will reduce overall impact on other
>uses of the same system.

I fully agree with that. However, moving away from "in sequence" thing
would greatly ease the migration to systemd timers, making it easier
to get away without crond on many systems.

I am wondering whether this is something we should think about giving
up in future. We have given up so many things since we moved to
systemd, would it be worth throwing this out of the window as well?

Setting RandomizeDelaySecs sufficiently high on our daily jobs would
probably help to chop off the load pike that especially virtualization
setups running many Debian instances suffer from at 06:25 or 07:35. I
think this could be a net gain worth pursuing.

Grüße
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Ian Jackson]

Marc Haber writes ("do packages depend on lexical order or 
{daily,weekly,monthly} cron jobs?"):
> I am wondering whether there are packages that actually do depend on
> that and how do find out short of unpacking the entire archive and
> inspecting the scripts.

My feeling is that this situation is
  - quite unlikely
  - very hard to analyse without trying it
  - easy to remediate by combining the two cron jobs into a single
 (perhaps wrapper) script
  - but there is no benefit for changing the rules, because:

> Do you people know about packages that have cron jobs that depend on
> execution order? Would it probably be a good idea to proactively put
> language into the Policy Manual Chapter 9.5 that scripts put into the
> /etc/cron.{daily,weekly,monthly} MUST NOT depend on any execution
> order and MUST be prepared to run in parallel?

"Does not depend on order" and "can be run in parallel" are not the
same.  I worry about additional concurrency.  Unlike ordering bugs,
concurrency bugs are hard to find by testing.  So running these
scripts in parallel is risky.

And, I think running cron.fooly scripts in parallel is a bad idea.
The objective is to run them "at some point", not to get it done as
soon as possible.  Running them in sequence will save electricity,
may save wear on components, and will reduce overall impact on other
uses of the same system.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

[Marc Haber]

Hi,

since cron uses run-parts to execute the scripts from
/etc/cron.{daily,weekly,monthly}, and run-parts is documented to
execute the contents from the directory in lexical sort-order,
packages could theoretically depend on a certain execution order of
cron jobs and fail miserably if the order is violated.

I am wondering whether there are packages that actually do depend on
that and how do find out short of unpacking the entire archive and
inspecting the scripts.

Scripts with such dependencies will probably fail miserably on systems
that are using systemd-cron instead of one of the "classic" cron
packaes since systemd will start all cron jobs in parallel, or if a
feature like RandomizeDelay (see man systemd.timer) is used to reduce
the impact in random order.

Do you people know about packages that have cron jobs that depend on
execution order? Would it probably be a good idea to proactively put
language into the Policy Manual Chapter 9.5 that scripts put into the
/etc/cron.{daily,weekly,monthly} MUST NOT depend on any execution
order and MUST be prepared to run in parallel?

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834