Re: [systemd-devel] how to debug failures when trying to lock down services

[Lennart Poettering]

On Do, 30.11.17 10:35, Mantas Mikulėnas (graw...@gmail.com) wrote:

> Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
> /var/log read-only... 

Hmm, it does? It really shouldn't.

I thought the issues were mostly around InaccessiblePaths= not
permitting exclusions, not about ProtectSystem/ReadOnlyPaths...

Have a link?

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to debug failures when trying to lock down services

[Michael Biebl]

2017-11-30 16:07 GMT+01:00 Michael Biebl :
> 2017-11-30 9:35 GMT+01:00 Mantas Mikulėnas :
>> On Thu, Nov 30, 2017 at 10:31 AM, Michael Biebl  wrote:
>>>
>>> 2017-11-30 6:52 GMT+01:00 Mantas Mikulėnas :
>>> > On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl  wrote:
>>> >>
>>> >> [Service]
>>> >> ProtectHome=yes
>>> >> PrivateTmp=yes
>>> >> PrivateDevices=yes
>>> >>
>>> >> ProtectSystem=strict
>>> >> ReadWritePaths=/var/log
>>> >> ReadWritePaths=/var/spool/rsyslog
>>> >> ReadWritePaths=/proc/kmsg
>>> >
>
>>>
>>> I suspect it's related to ProtectSystem=strict, as with
>>> ProtectSystem=full rsyslog seems to start successfully. But this is
>
>> Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
>> /var/log read-only... I think I've seen other people have that problem
>> recently.
>
>
> *facepalm*
> rsyslog.service by default uses StandardOutput=null, so I didn't see
> the error messages in debug mode.
>
> After fixing that, it was rather obvious.
>
> *double facepalm*
> rsyslog writes a pid file in /run and fails to start if it can't write
> the pidfile.  I will raise this upstream that maybe writing a pidfile
> in socket activation / sd_notify mode is not really necessary and it
> should stop doing that.

Filed https://github.com/rsyslog/rsyslog/issues/2143 for that

> For now I used RuntimeDirectory=rsyslog and
> ExecStart=/usr/sbin/rsyslogd -n -i /run/rsyslog/rsyslogd.pid
>
> So the complete rsyslog.service now looks like
>
> [Unit]
> Description=System Logging Service
> Requires=syslog.socket
> Documentation=man:rsyslogd(8)
> Documentation=http://www.rsyslog.com/doc/
>
> [Service]
> Type=notify
> ExecStart=/usr/sbin/rsyslogd -n
> StandardOutput=null
> Restart=on-failure
>
> PrivateTmp=yes
> PrivateDevices=yes
> ProtectHome=yes
> ProtectSystem=strict
> ReadWritePaths=/var/log
> ReadWritePaths=/var/spool/rsyslog
> ReadWritePaths=/proc/kmsg
> ReadWritePaths=/tmp
>
> RuntimeDirectory=rsyslog
>
> CapabilityBoundingSet=CAP_SYSLOG
> CapabilityBoundingSet=CAP_NET_BIND_SERVICE
>
> ExecStart=/usr/sbin/rsyslogd -n -i /run/rsyslog/rsyslogd.pid
>
> [Install]
> WantedBy=multi-user.target
> Alias=syslog.service
>
> Feedback welcome on how to reasonably lock down rsyslog by default
> without breaking commonly used functionality (like remote syslog)
>
> Regards,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [PATCH weston] doc/systemd: system service example

[Pekka Paalanen]

On Thu, 30 Nov 2017 13:29:22 +0100
Lennart Poettering  wrote:

> On Do, 30.11.17 12:09, Pekka Paalanen (ppaala...@gmail.com) wrote:
> 
> > > Hmm, what is this about?
> > > 
> > > This is racy, as the session ID is not really reliably predictable,
> > > and is synthesized in different contexts in different ways, for
> > > example depnding on whether audit is enabled in the kernel it might be
> > > session-1.scope rather than session-c1.scope.  
> > 
> > Hi Lennart,
> > 
> > this is the bit Martyn talked you in person some time ago, maybe Martyn
> > could refresh your memory?  
> 
> Oh, did we? I don't remember, sorry!

Hi Lennart,

no worries. I don't remember what I did early this week...

> > > Piggy-backing on "login" is a bad idea. "login" is a text tool, and
> > > thus the PAM rules for it usually pull in some TTY specific PAM
> > > modules. YOu shoudl really use your own PAM fragment here, and
> > > configure only the bits you need.  
> > 
> > Ok. Is there any guide or example I could point people to, so that they
> > can write their own stuff correctly? Any example I could put into
> > Weston docs?  
> 
> Unfortunately PAM is awful and highly distro-specific. It's not really
> possible to write PAM snippets that work generically on all
> distros. Sorry. The distros even patch PAM differently, so that
> slightly difference constructs are available...

That's the feeling I already got. Following Mantas' suggestion and
commenting it line by line like I did for the service unit is probably
the best we could do then.


Thanks,
pq


pgp731wB5Aclo.pgp
Description: OpenPGP digital signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [PATCH weston] doc/systemd: system service example

[Lennart Poettering]

On Do, 30.11.17 11:16, Martyn Welch (martyn.we...@collabora.co.uk) wrote:

> Debugging suggested that XDG_RUNTIME_DIR was not being created when it
> failed. There are 2 processes setting a PAMName, the failing Weston
> service and the user@.service (IIRC this gets called as part of user
> session creation, which I think was triggered by "User=xuser" in our
> weston.service). The user@.service has "PAMName=systemd-user" where as
> weston.service has "PAMName=login". When user@.service called PAM first
> everything worked as expected, if weston.service called PAM first it
> failed. This was our attempt at forcing the former rather than the
> latter.

Ah yeah, we discussed this recently here:

https://github.com/systemd/systemd/issues/7339

So the problem is that user@.service currently can't be started
independently of logind, it must be logind that starts it for you at
the right moment. If you do anyway, XDG_RUNTIME_DIR won't be set, and
things fall apart. We should fix that though...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [PATCH weston] doc/systemd: system service example

[Lennart Poettering]

On Do, 30.11.17 12:09, Pekka Paalanen (ppaala...@gmail.com) wrote:

> > Hmm, what is this about?
> > 
> > This is racy, as the session ID is not really reliably predictable,
> > and is synthesized in different contexts in different ways, for
> > example depnding on whether audit is enabled in the kernel it might be
> > session-1.scope rather than session-c1.scope.
> 
> Hi Lennart,
> 
> this is the bit Martyn talked you in person some time ago, maybe Martyn
> could refresh your memory?

Oh, did we? I don't remember, sorry!

> > Piggy-backing on "login" is a bad idea. "login" is a text tool, and
> > thus the PAM rules for it usually pull in some TTY specific PAM
> > modules. YOu shoudl really use your own PAM fragment here, and
> > configure only the bits you need.
> 
> Ok. Is there any guide or example I could point people to, so that they
> can write their own stuff correctly? Any example I could put into
> Weston docs?

Unfortunately PAM is awful and highly distro-specific. It's not really
possible to write PAM snippets that work generically on all
distros. Sorry. The distros even patch PAM differently, so that
slightly difference constructs are available...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [PATCH weston] doc/systemd: system service example

[Mantas Mikulėnas]

On Thu, Nov 30, 2017, 12:10 Pekka Paalanen  wrote:

> > > +# Set up a full user session for the user, required by Weston.
> > > +PAMName=login
> >
> > Piggy-backing on "login" is a bad idea. "login" is a text tool, and
> > thus the PAM rules for it usually pull in some TTY specific PAM
> > modules. YOu shoudl really use your own PAM fragment here, and
> > configure only the bits you need.
>
> Ok. Is there any guide or example I could point people to, so that they
> can write their own stuff correctly? Any example I could put into
> Weston docs?
>
> Personally I have no understanding of what PAM does. I just copied
> weston-launch (setuid-root helper for non-systemd systems) that also
> uses "login" for PAM name if it was asked to create a new session(?).
>

Instead of reusing "login", it would be better to start with a copy e.g.
lightdm's or xdm's config, IMHO.

There are three main steps in PAM. Besides "auth" (authentication, which
services simply skip), you also have "account" (authorization and
accounting) which verifies whether the user is allowed to log in – e.g. not
disabled, not locked out, not time-restricted. (For example, SSH pubkey
logins don't use PAM auth, but still have to perform the account
verification.)

Usually there's just one global configuration for "account" (e.g. in
pam.d/common-account) and you can directly include it.

But you also have "session" (session setup), which registers with
systemd-logind, sets up SELinux, prints the motd, and so on. These *do*
vary greatly between service types – e.g. you want pam_motd for 'login' but
not for 'cron'; you want pam_systemd for 'weston' but not for 'ftpd'. So
the "session" part may need to be customized, which is why you should start
with another graphical manager's.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [PATCH weston] doc/systemd: system service example

[Martyn Welch]

On Thu, 2017-11-30 at 12:09 +0200, Pekka Paalanen wrote:
> On Wed, 29 Nov 2017 19:05:07 +0100
> Lennart Poettering  wrote:
> 
> > On Di, 28.11.17 12:14, Pekka Paalanen (ppaala...@gmail.com) wrote:
> > 
> > > +
> > > +[Unit]
> > > +Description=Weston, a Wayland compositor, as a system service
> > > +Documentation=man:weston(1) man:weston.ini(5)
> > > +Documentation=http://wayland.freedesktop.org/
> > > +
> > > +# Make sure we are started after logins are permitted.
> > > +After=systemd-user-sessions.service
> > > +
> > > +# If Plymouth is used, we want to start when it is on its way out.
> > > +After=plymouth-quit-wait.service
> > > +
> > > +# D-Bus is necessary for contacting logind. Logind is required.
> > > +Wants=dbus.socket
> > > +After=dbus.socket
> > > +
> > > +# This scope is created by pam_systemd when logging in as the user.
> > > +# This directive is a workaround to a systemd bug, where the setup of the
> > > +# user session by PAM has some race condition, possibly leading to a 
> > > failure.
> > > +# See README for more details.
> > > +After=session-c1.scope  
> > 
> > Hmm, what is this about?
> > 
> > This is racy, as the session ID is not really reliably predictable,
> > and is synthesized in different contexts in different ways, for
> > example depnding on whether audit is enabled in the kernel it might be
> > session-1.scope rather than session-c1.scope.
> 
> Hi Lennart,
> 
> this is the bit Martyn talked you in person some time ago, maybe Martyn
> could refresh your memory?
> 
> Yes, I am definitely not happy about this directive, but it serves as
> the reminder of the issue Martyn was debugging a long time ago, and
> this was the workaround chosen for the particular project at that time.
> I guessed it's not portable.
> 
> I have it here so it would trigger the discussion, in the hopes that
> someone could recall the details of the fundamental problem. I heard it
> was deemed to be a hard-to-reproduce systemd bug, but I have no other
> details.
> 
> If we could determine the bug doesn't exist anymore, that would be
> awesome and I could just drop this.
> 

First off, apologies if I explained this badly when I attempted to
explain the issue at FOSDEM, I'm still not convinced I truly understand
what's happening here, but let's give it another go...

Testing of the product in the project mentioned by Pekka revealed that
we would sometimes see Weston fail to boot. IIRC, in the order of 2 in
10 times when switching from an initial charging state (that the device
would boot into when power was applied to the device without the power
button being pressed) and the normal running state (when the power
button was then pressed). The charging state is pretty much a small
subset of the normal running state. From memory, the "xuser" session is
not created in that state. The filtered logs that I was given:

2017-01-05T14:08:19.41+00:00 XXX systemd-logind[315]: New seat
seat0.
2017-01-05T14:08:19.672756+00:00 XXX systemd-logind[315]: Watching
system buttons on /dev/input/event0 (power-gpio-keys)
2017-01-05T14:08:20.176354+00:00 XXX systemd[1]: Starting User Manager
for UID 1000...
2017-01-05T14:08:20.394955+00:00 XXX systemd[1]: Starting Weston...
2017-01-05T14:08:21.867999+00:00 XXX systemd-logind[315]: New session c1
of user xuser.
2017-01-05T14:08:21.915400+00:00 XXX systemd-logind[315]: Watching
system buttons on /dev/input/event0 (power-gpio-keys)
2017-01-05T14:08:46.279480+00:00 XXX weston[403]: [14:08:46.157] fatal:
environment variable XDG_RUNTIME_DIR is not set.
2017-01-05T14:08:46.421821+00:00 XXX systemd[1]: Failed to start Weston.
2017-01-05T14:08:46.471701+00:00 XXX systemd-logind[315]: Removed
session c1.
2017-01-05T14:08:47.424030+00:00 XXX systemd[1]: Started User Manager
for UID 1000.
2017-01-05T14:08:47.469949+00:00 XXX systemd-logind[315]: Failed to stop
user service: Transaction is destructive.
2017-01-05T14:08:47.540518+00:00 XXX systemd-logind[315]: Failed to stop
user slice: Transaction is destructive.

Debugging suggested that XDG_RUNTIME_DIR was not being created when it
failed. There are 2 processes setting a PAMName, the failing Weston
service and the user@.service (IIRC this gets called as part of user
session creation, which I think was triggered by "User=xuser" in our
weston.service). The user@.service has "PAMName=systemd-user" where as
weston.service has "PAMName=login". When user@.service called PAM first
everything worked as expected, if weston.service called PAM first it
failed. This was our attempt at forcing the former rather than the
latter.

> > > +# Set up a full user session for the user, required by Weston.
> > > +PAMName=login  
> > 
> > Piggy-backing on "login" is a bad idea. "login" is a text tool, and
> > thus the PAM rules for it usually pull in some TTY specific PAM
> > modules. YOu shoudl really use your own PAM fragment here, and
> > configure only the bits you need.
> 

Oh, so could it just be that we needed something other 

Re: [systemd-devel] [PATCH weston] doc/systemd: system service example

[Pekka Paalanen]

On Wed, 29 Nov 2017 19:05:07 +0100
Lennart Poettering  wrote:

> On Di, 28.11.17 12:14, Pekka Paalanen (ppaala...@gmail.com) wrote:
> 
> > +
> > +[Unit]
> > +Description=Weston, a Wayland compositor, as a system service
> > +Documentation=man:weston(1) man:weston.ini(5)
> > +Documentation=http://wayland.freedesktop.org/
> > +
> > +# Make sure we are started after logins are permitted.
> > +After=systemd-user-sessions.service
> > +
> > +# If Plymouth is used, we want to start when it is on its way out.
> > +After=plymouth-quit-wait.service
> > +
> > +# D-Bus is necessary for contacting logind. Logind is required.
> > +Wants=dbus.socket
> > +After=dbus.socket
> > +
> > +# This scope is created by pam_systemd when logging in as the user.
> > +# This directive is a workaround to a systemd bug, where the setup of the
> > +# user session by PAM has some race condition, possibly leading to a 
> > failure.
> > +# See README for more details.
> > +After=session-c1.scope  
> 
> Hmm, what is this about?
> 
> This is racy, as the session ID is not really reliably predictable,
> and is synthesized in different contexts in different ways, for
> example depnding on whether audit is enabled in the kernel it might be
> session-1.scope rather than session-c1.scope.

Hi Lennart,

this is the bit Martyn talked you in person some time ago, maybe Martyn
could refresh your memory?

Yes, I am definitely not happy about this directive, but it serves as
the reminder of the issue Martyn was debugging a long time ago, and
this was the workaround chosen for the particular project at that time.
I guessed it's not portable.

I have it here so it would trigger the discussion, in the hopes that
someone could recall the details of the fundamental problem. I heard it
was deemed to be a hard-to-reproduce systemd bug, but I have no other
details.

If we could determine the bug doesn't exist anymore, that would be
awesome and I could just drop this.

> > +# Set up a full user session for the user, required by Weston.
> > +PAMName=login  
> 
> Piggy-backing on "login" is a bad idea. "login" is a text tool, and
> thus the PAM rules for it usually pull in some TTY specific PAM
> modules. YOu shoudl really use your own PAM fragment here, and
> configure only the bits you need.

Ok. Is there any guide or example I could point people to, so that they
can write their own stuff correctly? Any example I could put into
Weston docs?

Personally I have no understanding of what PAM does. I just copied
weston-launch (setuid-root helper for non-systemd systems) that also
uses "login" for PAM name if it was asked to create a new session(?).


Thanks,
pq


pgpb72BBsFTNa.pgp
Description: OpenPGP digital signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to debug failures when trying to lock down services

[Mantas Mikulėnas]

On Thu, Nov 30, 2017 at 10:31 AM, Michael Biebl  wrote:

> 2017-11-30 6:52 GMT+01:00 Mantas Mikulėnas :
> > On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl  wrote:
> >>
> >> Hi,
> >>
> >> today I tried to lock down the rsyslog.service that I have on my system.
> >>
> >> For that I first created an override.conf that contained
> >>
> >> [Service]
> >> ProtectHome=yes
> >> PrivateTmp=yes
> >> PrivateDevices=yes
> >>
> >> ProtectSystem=strict
> >> ReadWritePaths=/var/log
> >> ReadWritePaths=/var/spool/rsyslog
> >> ReadWritePaths=/proc/kmsg
> >
> >
> > Are you using imklog or imkmsg? The latter would require the new
> /dev/kmsg
> > interface (which probably conflicts with PrivateDevices= above).
>
> I suspect it's related to ProtectSystem=strict, as with
> ProtectSystem=full rsyslog seems to start successfully. But this is
> just trial and error.

[…]
> Already tried
> ExecStart=
> ExecStart=/usr/bin/strace -f -o /var/log/strace /usr/sbin/rsyslogd -n
>
> but this didn't produce any /var/log/strace log file.
>
>
Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
/var/log read-only... I think I've seen other people have that problem
recently.

Take a look with `ExecStartPre=/usr/bin/findmnt`.

-- 
Mantas Mikulėnas 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] how to debug failures when trying to lock down services

[Michael Biebl]

2017-11-30 6:52 GMT+01:00 Mantas Mikulėnas :
> On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl  wrote:
>>
>> Hi,
>>
>> today I tried to lock down the rsyslog.service that I have on my system.
>>
>> For that I first created an override.conf that contained
>>
>> [Service]
>> ProtectHome=yes
>> PrivateTmp=yes
>> PrivateDevices=yes
>>
>> ProtectSystem=strict
>> ReadWritePaths=/var/log
>> ReadWritePaths=/var/spool/rsyslog
>> ReadWritePaths=/proc/kmsg
>
>
> Are you using imklog or imkmsg? The latter would require the new /dev/kmsg
> interface (which probably conflicts with PrivateDevices= above).

I suspect it's related to ProtectSystem=strict, as with
ProtectSystem=full rsyslog seems to start successfully. But this is
just trial and error.

>>
>> Unfortunately, rsyslog.service failed to start:
>> ● rsyslog.service - System Logging Service
>>Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
>> vendor preset: enabled)
>>   Drop-In: /etc/systemd/system/rsyslog.service.d
>>└─override.conf
>>Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET;
>> 2s ago
>>  Docs: man:rsyslogd(8)
>>http://www.rsyslog.com/doc/
>>   Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited,
>> status=1/FAILURE)
>>  Main PID: 2734 (code=exited, status=1/FAILURE)
>
>
> Well, it does say that the failure comes from rsyslogd itself, not from the
> namespace setup...
>
>>
>> The journal doesn't contain anything useful.
>
>
> I'm guessing rsyslog will log its own errors to /var/log/syslog rather than
> stderr.

I don't have anyting in /var/log/syslog

>>
>> Any hints how I can further debug this why rsyslog fails to start?
>
>
> rsyslogd -d -d -d

Already tried that, doesn't produce any useful logs.


> strace

Already tried
ExecStart=
ExecStart=/usr/bin/strace -f -o /var/log/strace /usr/sbin/rsyslogd -n

but this didn't produce any /var/log/strace log file.


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [Xen-devel] [PATCH v1] core: mount xenfs, ignore proc-xen.mount (#6442, #6662)

[Jan Beulich]

>>> On 30.11.17 at 09:23,  wrote:
> On Wed, Nov 29, Jan Beulich wrote:
> 
>> Ah, I see. But then still I don't see why at least on half way
>> recent Xen /sys/hypervisor/properties/features wouldn't have
>> the information you're after (and even more precise, because
>> down the road control domain and hardware domain may be
>> separate entities).
> 
> Per discussion in https://github.com/systemd/systemd/pull/6662, the
> feature bits should not be used for dom0 detection.

I can't seem to interpret that discussion the way you do. In fact
(as I've said before) using the feature flag is more reliable, as it
being set implies this is the hardware domain (rather than the
more fuzzy "control domain" implied by "control_d").

Wei, your comments there from Oct 27 and 30 are what I think
Olaf refers to. Could you clarify this?

Jan

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [Xen-devel] [PATCH v1] core: mount xenfs, ignore proc-xen.mount (#6442, #6662)

[Olaf Hering]

On Wed, Nov 29, Jan Beulich wrote:

> Ah, I see. But then still I don't see why at least on half way
> recent Xen /sys/hypervisor/properties/features wouldn't have
> the information you're after (and even more precise, because
> down the road control domain and hardware domain may be
> separate entities).

Per discussion in https://github.com/systemd/systemd/pull/6662, the
feature bits should not be used for dom0 detection.

Olaf


signature.asc
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel