Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Nicholas Geovanis]

On Fri, Apr 21, 2017 at 5:42 PM, Joel Rees  wrote:
 On Sat, Apr 22, 2017 at 4:13 AM, Nicholas Geovanis
 wrote:
> Like numerous linux users I have sometimes lamented coming to terms with
> systemd. My belief is that it's a well-written collection of software
which
> is somewhat over-engineered. It fills a need, sure, though I've managed to
> live and work without it for a long time (been using linux since 1994).
And
> who am I to question Torvalds and Co. on the subject of its suitability
for
> linux and the data center?

Have Linus and Lennart come to a meeting of minds or something?

Yes, they conspired to invade my personal space with systemd ;-)
But as I wrote, the 8-kilo gorilla named Amazon Web Services may
make that a non-issue for the future. It's looking "cloud"-ier every day.
(12 billion $ in 2016. Dwell on that number for a while.)
And Poettering worked on pulseaudio so maybe someday I can learn to forgive
:-)

Fwd: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Joel Rees]

 On Sat, Apr 22, 2017 at 4:13 AM, Nicholas Geovanis
 wrote:
> Like numerous linux users I have sometimes lamented coming to terms with
> systemd. My belief is that it's a well-written collection of software which
> is somewhat over-engineered. It fills a need, sure, though I've managed to
> live and work without it for a long time (been using linux since 1994). And
> who am I to question Torvalds and Co. on the subject of its suitability for
> linux and the data center?

Have Linus and Lennart come to a meeting of minds or something?

So I looked up "Torvalds systemd" and found a slashdot Q/A article with
Torvalds in which someone asked him about systemd.

Interesting.

> So the other day I was on a recently-built Amazon AWS EC2 instance, running
> one of the AWS-branded linux AMIs, fixing things in /etc/init.d. Thinking
> about how AWS might rule the world someday, since they already hold about
> 35-40% of the public cloud
> (http://www.geekwire.com/2017/cloud-report-card-amazon-web-services-12b-juggernaut-microsoft-google-gaining/).
> Then I had one of those "Duh!" moments: There must be on-the-order-of a
> million of linux instances on the planet which are _not_ running systemd, as
> AWS's own linux AMIs do not by default.
>
> It seems to me that this data point has been completely ignored in the
> years-long discussions about systemd's merits, flaws and suitability.
>
> On Mon, Apr 17, 2017 at 4:34 PM, Jonathan Dowland  wrote:
>>
>> On Fri, Apr 14, 2017 at 03:17:00PM +0200, Nicolas George wrote:
>> > Note: systemd is not for end-users, it is for system administrator and
>> > distribution authors.
>>
>> {systemctl,journalctl,etc.} --user beg to differ.
>>
>>
>> --
>> ⢀⣴⠾⠻⢶⣦⠀
>> ⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
>> ⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
>> ⠈⠳⣄ Please do not CC me, I am subscribed to the list.
>
>



--
Joel Rees

I'm imagining I'm a novelist:
http://joel-rees-economics.blogspot.com/2017/01/soc500-00-00-toc.html
More of my delusions:
http://reiisi.blogspot.jp/p/novels-i-am-writing.html


-- 
Joel Rees

I'm imagining I'm a novelist:
http://joel-rees-economics.blogspot.com/2017/01/soc500-00-00-toc.html
More of my delusions:
http://reiisi.blogspot.jp/p/novels-i-am-writing.html

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Nicholas Geovanis]

Like numerous linux users I have sometimes lamented coming to terms with
systemd. My belief is that it's a well-written collection of software which
is somewhat over-engineered. It fills a need, sure, though I've managed to
live and work without it for a long time (been using linux since 1994). And
who am I to question Torvalds and Co. on the subject of its suitability for
linux and the data center?

So the other day I was on a recently-built Amazon AWS EC2 instance, running
one of the AWS-branded linux AMIs, fixing things in /etc/init.d. Thinking
about how AWS might rule the world someday, since they already hold about
35-40% of the public cloud (
http://www.geekwire.com/2017/cloud-report-card-amazon-web-services-12b-juggernaut-microsoft-google-gaining/).
Then I had one of those "Duh!" moments: There must be on-the-order-of a
million of linux instances on the planet which are _not_ running systemd,
as AWS's own linux AMIs do not by default.

It seems to me that this data point has been completely ignored in the
years-long discussions about systemd's merits, flaws and suitability.

On Mon, Apr 17, 2017 at 4:34 PM, Jonathan Dowland  wrote:

> On Fri, Apr 14, 2017 at 03:17:00PM +0200, Nicolas George wrote:
> > Note: systemd is not for end-users, it is for system administrator and
> > distribution authors.
>
> {systemctl,journalctl,etc.} --user beg to differ.
>
>
> --
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
> ⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
> ⠈⠳⣄ Please do not CC me, I am subscribed to the list.
>

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Jonathan Dowland]

On Fri, Apr 14, 2017 at 03:17:00PM +0200, Nicolas George wrote:
> Note: systemd is not for end-users, it is for system administrator and
> distribution authors.

{systemctl,journalctl,etc.} --user beg to differ.


-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Laurent Bigonville]

Greg Wooledge > wrote:

On Fri, Apr 14, 2017 at 03:17:00PM +0200, Nicolas George wrote:
> Le quintidi 25 germinal, an CCXXV, Greg Wooledge a écrit :
> > Some day there will be actual end-user-friendly systemd documentation
> > somewhere, consolidating all of these pieces of wisdom together.  I hope.
>
> Note: systemd is not for end-users, it is for system administrator and
> distribution authors.

The end users of systemd are Linux system administrators.  You and me.
The people on this mailing list.  That's us, the users.  That's why
it's called "debian-user".

If you'd prefer "Some day there will be a system administrator's guide
for systemd", that's an acceptable wording.

There is already extensive documentation about how to administrate systemd:

https://www.freedesktop.org/wiki/Software/systemd/#manualsanddocumentationforusersandadministrators

"The systemd for Administrators Blog Series" worth reading.

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Joel Rees]

On Fri, Apr 14, 2017 at 9:37 PM, Greg Wooledge  wrote:
> [...]
>Don't even get me started on sshd.service vs. ssh.service.  Do you
>have any idea how hard it is to notice that extra/missing "d", and
>figure out why things Simply Do Not Work?

Well, that demonstrates that the concept of tagging a "d" on the end
of a name to indicate the daemon part well predates systemd, and
probably should be reconsidered in a world where short names are no
longer required.

Not sure how that relates to the rest of the issues you are trying to
work through.

-- 
Joel Rees

I'm imagining I'm a novelist:
http://joel-rees-economics.blogspot.com/2017/01/soc500-00-00-toc.html
More of my delusions:
http://reiisi.blogspot.jp/p/novels-i-am-writing.html

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Jonathan de Boyne Pollard]

Greg Wooledge:
> Don't even get me started on sshd.service vs. ssh.service.  Do
> you have any idea how hard it is to notice that extra/missing “d”, 
> and figure out why things Simply Do Not Work?

* http://www.mail-archive.com/supervision@list.skarnet.org/msg01486.html

* https://unix.stackexchange.com/a/303302/5132

Yes.

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Martin Read]

On 14/04/17 14:17, Nicolas George wrote:

Le quintidi 25 germinal, an CCXXV, Greg Wooledge a écrit :

Some day there will be actual end-user-friendly systemd documentation
somewhere, consolidating all of these pieces of wisdom together.  I hope.


Note: systemd is not for end-users, it is for system administrator and
distribution authors.


systemd is absolutely for end-users, because:

* Some systemd-running systems are home desktop computers with a single 
physical user; in this case, the distinction between "administrator" and 
"user" may well only exist as a hallucination of the computer, with no 
basis in the external physical world. If the computer I'm typing this 
e-mail on breaks down, I have to fix it myself.


* systemd can, in any event, be used to manage service-like processes 
that form part of a user's login session, using unit files stored in 
that user's XDG Base Directories.

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Greg Wooledge]

On Fri, Apr 14, 2017 at 03:17:00PM +0200, Nicolas George wrote:
> Le quintidi 25 germinal, an CCXXV, Greg Wooledge a écrit :
> > Some day there will be actual end-user-friendly systemd documentation
> > somewhere, consolidating all of these pieces of wisdom together.  I hope.
> 
> Note: systemd is not for end-users, it is for system administrator and
> distribution authors.

The end users of systemd are Linux system administrators.  You and me.
The people on this mailing list.  That's us, the users.  That's why
it's called "debian-user".

If you'd prefer "Some day there will be a system administrator's guide
for systemd", that's an acceptable wording.

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Dejan Jocic]

On 14-04-17, Nicolas George wrote:
> Le quintidi 25 germinal, an CCXXV, Greg Wooledge a écrit :
> > Some day there will be actual end-user-friendly systemd documentation
> > somewhere, consolidating all of these pieces of wisdom together.  I hope.
> 
> Note: systemd is not for end-users, it is for system administrator and
> distribution authors.
> 
Actually, it should be for end-user too. On personal computer, that
end-user is system administrator. I also find that systemd is very well
documented. But it could be just me. Now, please carry on, enjoyed this
thread very much, learned thing, or two :)

Thank you for your time.

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Nicolas George]

Le quintidi 25 germinal, an CCXXV, Greg Wooledge a écrit :
> Some day there will be actual end-user-friendly systemd documentation
> somewhere, consolidating all of these pieces of wisdom together.  I hope.

Note: systemd is not for end-users, it is for system administrator and
distribution authors.

> 1) To override parts of a distribution's systemd unit locally, you MUST
>use the foo.service.d/ method.  You can't just put the override bits
>into an /etc/systemd/system/foo.service file.  That would be too easy.

foo.service.d/*.confis for overriding bits.
/etc/systemd/system/foo.service is for overriding the whole file.

I find that fairly natural. Otherwise, how would you override the whole
file?

> 2) The files inside foo.service.d/ MUST end with a .conf suffix.  (Cf.
>the wheezy->jessie apache2 upgrade, and having to rename every single
>one of my virtual domain config files AND the symlinks to them.)

After having been bitten by old *.conf~ backup files left by an editor,
I must say I find that restriction quite useful.

> 3) foo.service.d/ must use the CANONICAL service name of whatever it is
>that you're trying to override.  This may not be the same as the
>Debian package name.  For example, the nfs-kernel-server package
>creates a systemd unit named nfs-server.service with an ALIAS of
>nfs-kernel-server.service.  If you try to create override files in
>nfs-kernel-server.service.d/ it will not work correctly.  They have
>to be in nfs-server.service.d/ instead.
> 
>Don't even get me started on sshd.service vs. ssh.service.  Do you
>have any idea how hard it is to notice that extra/missing "d", and
>figure out why things Simply Do Not Work?

On the other hand, if systemd were to read snippets of configuration
with a subtly different name, someone else (or maybe be even yourself!)
would have complained about wasted time because of a stale config
snippet that should not have been read.

I find that strict rules are usually more convenient in the long run.

Note that you can use "systemctl edit" to have an editor started on the
exact correct file.

Regards,

-- 
  Nicolas George


signature.asc
Description: Digital signature

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Greg Wooledge]

On Thu, Apr 13, 2017 at 10:01:25PM +0100, Jonathan de Boyne Pollard wrote:
> ... albeit poorly.  If one wants to run daemontools under systemd, svscanboot 
> is
> not the way; svscanboot is a thing of the past
> http://jdebp.eu./FGA/inittab-is-history.html#svscanboot , and was a source of
> problems long before systemd was invented.

Cool.  I wish my Google searching had stumbled upon that, when I was
trying to figure out how to do all that stuff.

> The world wants you to clean your screen
> http://unix.stackexchange.com/a/233855/5132 , and this is merely one of the 
> ways
> that it makes you do so.

Some day there will be actual end-user-friendly systemd documentation
somewhere, consolidating all of these pieces of wisdom together.  I hope.

My own contributions toward that effort have been riddled with failure and
confusion, for which I am sorry.  I'm honestly *trying*, but this stuff is
really opaque at times.

For instance, just this week I learned three new things:

1) To override parts of a distribution's systemd unit locally, you MUST
   use the foo.service.d/ method.  You can't just put the override bits
   into an /etc/systemd/system/foo.service file.  That would be too easy.

2) The files inside foo.service.d/ MUST end with a .conf suffix.  (Cf.
   the wheezy->jessie apache2 upgrade, and having to rename every single
   one of my virtual domain config files AND the symlinks to them.)

3) foo.service.d/ must use the CANONICAL service name of whatever it is
   that you're trying to override.  This may not be the same as the
   Debian package name.  For example, the nfs-kernel-server package
   creates a systemd unit named nfs-server.service with an ALIAS of
   nfs-kernel-server.service.  If you try to create override files in
   nfs-kernel-server.service.d/ it will not work correctly.  They have
   to be in nfs-server.service.d/ instead.

   Don't even get me started on sshd.service vs. ssh.service.  Do you
   have any idea how hard it is to notice that extra/missing "d", and
   figure out why things Simply Do Not Work?

Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Jonathan de Boyne Pollard]

Greg Wooledge:

> 
> Suppose you want to start DJB's daemontools from a locally created systemd
> unit/service. Here's a file that will do that:
> 

... albeit poorly.  If one wants to run daemontools under systemd, svscanboot is
not the way; svscanboot is a thing of the past
http://jdebp.eu./FGA/inittab-is-history.html#svscanboot , and was a source of
problems long before systemd was invented.


Greg Wooledge:

> 
> (The Linux kernel introduced an entirely new thing called a "cgroup" to
> make this possible. That's how ridiculous self-backgrounding is.)
> 

Control groups are not jobs
http://jdebp.eu./FGA/linux-control-groups-are-not-jobs.html ; they were
introduced to do resource limiting, and the systemd developers have actually
complained quite a lot over the years that control groups did not turn out to be
what they thought they were.


Greg Wooledge:

> 
> $ systemctl status daemontools.service
> 
> * daemontools.service – daemontools supervisor
> Loaded: loaded (/etc/systemd/system/daemontools.service; enabled)
> Active: active (running) since Wed 2017-01-11 03:28:47 EST; 2 months 21
> days ago
> Main PID: 529 (svscanboot)
> CGroup: /system.slice/daemontools.service
> |- 529 /bin/sh /command/svscanboot /dev/ttyS0
> |- 531 svscan /service
> 

... and there is svscanboot being a problem again.  Notice how the main PID is
wrong, and the log output from svscan (when there is some) does not go into the
log that systemctl shows below this.


Greg Wooledge:

> 
> if you want to change the behavior of the Debian default getty@ service to
> make it stop clearing the screen all the damned time,
> 

The world wants you to clean your screen
http://unix.stackexchange.com/a/233855/5132 , and this is merely one of the ways
that it makes you do so.

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[deloptes]

Kevin O'Gorman wrote:

> Are you sure?  On my system, this produces nothing at all.  But the
> directory
> exists and is populated.

It works great in jessie

$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL
+XZ -SECCOMP -APPARMOR

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Nicholas Geovanis]

On Mon, Apr 03, 2017 at 09:36:16AM -0500, Tom Browder wrote:
> But I kind of understand why systemd, but I wish I could find a good
> cookbook description of how to add or modify a new process.

I like the "systemd vs. sysvinit" cheatsheet at
http://linoxide.com/linux-command/systemd-vs-sysvinit-cheatsheet/

.Nick

On Mon, Apr 3, 2017 at 11:14 AM, Kevin O'Gorman  wrote:

>
>
> On Mon, Apr 3, 2017 at 8:39 AM, Greg Wooledge  wrote:
>
>> On Mon, Apr 03, 2017 at 09:36:16AM -0500, Tom Browder wrote:
>> > But I kind of understand why systemd, but I wish I could find a good
>> > cookbook description of how to add or modify a new process.
>>
>> The first hurdle is learning the terminology that systemd uses.  It's
>> not exactly intuitive.
>>
>
>
>> [...]
>>
>> If you want to change your system's "run level" from graphical.target
>> to multi-user.target, run this command as root:
>>
>> # systemctl set-default multi-user.target
>>
>> To see a list of your available targets (assuming no major local changes),
>> use this command:
>>
>> $ find /lib/systemd/ -name '*.target'
>>
>>
> Are you sure?  On my system, this produces nothing at all.  But the
> directory
> exists and is populated.
>
>
>
> --
> Kevin O'Gorman
> #define QUESTION ((bb) || (!bb))   /* Shakespeare */
>
> Please consider the environment before printing this email.
>

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Joshua Schaeffer]

On Mon, Apr 3, 2017 at 10:14 AM, Kevin O'Gorman  wrote:

>
>> To see a list of your available targets (assuming no major local changes),
>> use this command:
>>
>> $ find /lib/systemd/ -name '*.target'
>>
>>
> Are you sure?  On my system, this produces nothing at all.  But the
> directory
> exists and is populated.
>

What version of systemd do you have installed?

#systemd --version

Re: Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Kevin O'Gorman]

On Mon, Apr 3, 2017 at 8:39 AM, Greg Wooledge  wrote:

> On Mon, Apr 03, 2017 at 09:36:16AM -0500, Tom Browder wrote:
> > But I kind of understand why systemd, but I wish I could find a good
> > cookbook description of how to add or modify a new process.
>
> The first hurdle is learning the terminology that systemd uses.  It's
> not exactly intuitive.
>


> [...]
>
> If you want to change your system's "run level" from graphical.target
> to multi-user.target, run this command as root:
>
> # systemctl set-default multi-user.target
>
> To see a list of your available targets (assuming no major local changes),
> use this command:
>
> $ find /lib/systemd/ -name '*.target'
>
>
Are you sure?  On my system, this produces nothing at all.  But the
directory
exists and is populated.



-- 
Kevin O'Gorman
#define QUESTION ((bb) || (!bb))   /* Shakespeare */

Please consider the environment before printing this email.

Systemd services (was Re: If Linux Is About Choice, Why Then ...)

[Greg Wooledge]

On Mon, Apr 03, 2017 at 09:36:16AM -0500, Tom Browder wrote:
> But I kind of understand why systemd, but I wish I could find a good
> cookbook description of how to add or modify a new process.

The first hurdle is learning the terminology that systemd uses.  It's
not exactly intuitive.

Systemd has "run levels" which are used to control which groups of
services are started at boot time.  But it calls these things "targets"
instead of "run levels".

The default target is called "graphical.target", which means that if
a Display Manager is installed and enabled, it will be started.  This
is analogous to run level 5 on Red Hat systems.

You can see your system's default target by running this command:

$ systemctl get-default

The other target (run level) that most people will care about is called
"multi-user.target".  This is analogous to run level 3 on Red Hat
systems.  It's the same as "graphical.target" but without the Display
Manager.

If you want to change your system's "run level" from graphical.target
to multi-user.target, run this command as root:

# systemctl set-default multi-user.target

To see a list of your available targets (assuming no major local changes),
use this command:

$ find /lib/systemd/ -name '*.target'

Now, the trickier part: adding a new managed service.  Systemd calls
these things "units".  It also calls them "services", which are a
specific type of unit.

In the typical case, where you just want to run a daemon, nothing fancy,
what you do is create a file in the directory /etc/systemd/system.
This file's name should end with ".service" to indicate that this is a
service.  The first part of the file's name should be whatever you want
your service to be called.

The .service file is a text file in the "Windows INI" format,
meaning there are sections with [Headers] in square brackets, and
configuration lines within each section.  (See "man systemd.unit" and
"man systemd.service" for full details.)

Suppose you want to start DJB's daemontools from a locally created
systemd unit/service.  Here's a file that will do that:

==
[Unit]
Description=daemontools supervisor
After=getty.target

[Service]
Type=simple
User=root
Group=root
Restart=always
ExecStart=/command/svscanboot /dev/ttyS0
TimeoutSec=0

[Install]
WantedBy=multi-user.target
==

Mostly it's quite simple once you have an example like this.
The ExecStart= line gives the command to be executed.  The Type=
line tells us that this command runs in the foreground, which
every sane daemon should do.  If your daemon is self-backgrounding,
look for an option to tell it to STOP THAT, BAD DAEMON.  Anything
that has been seriously maintained at some time in this millennium
should either run in the foreground by default, or should at least
have an option to run in the foreground.

I cannot stress this enough.  Your daemons should run in the foreground.
Self-backgrounding is BAD.  Unconditional self-backgrounding is grounds
for being deleted and replaced by something that does not suck.

However, because systemd has to live in a world where evil runs free,
it goes out of its way to be accomodating to those who must live
under such a horrible regime.  Therefore, it can be told to manage
even a self-backgrounding daemon.  (The Linux kernel introduced an
entirely new thing called a "cgroup" to make this possible.  That's
how ridiculous self-backgrounding is.)

If your daemon was written in 1985 and self-backgrounds and can't be
changed, and you don't have the time to rewrite it yourself, then read
"man systemd.service" and look for the word "forking".  Be sure to read
all of the recommendations.

Once you've written your .service file, given it a cool name, and saved
it in /etc/systemd/system, there are three more steps.

First, you run this command to tell systemd to look for new unit files
(as root):

# systemctl daemon-reload

Second, you tell systemd that your service should be enabled, which
means it gets started automatically at boot time.  Again, as root:

# systemctl enable your.service

Finally, tell systemd to start the service right now:

# systemctl start your.service

When you run systemctl start (or any other systemctl subcommand), note
that all it does is send a message to systemd, asking systemd to do
stuff.  It doesn't hang around to see what happened.  It doesn't report
status.  It just sends the message.

Some people dislike this.  They want to see a reassuring message to tell
them that yes, their daemon is actually running now.  Usually because
sysvinit used to do that for them.

Since there's no feedback from systemctl, therefore, there's one more
command you should learn.  You can run it without root:

$ systemctl status your.service

This shows you whether things are running or not.  In fact, it tells
you a whole bunch of stuff: the state of the service, how long it has
been in this