Re: systmd-analyze security as a release goal

[Timo Röhling]

Hi,

* Colin Watson  [2023-07-14 11:50]:

On Fri, Jul 14, 2023 at 08:08:39AM +0100, Matthew Garrett wrote:

On Thu, Jul 13, 2023 at 08:03:39PM +0200, Timo Röhling wrote:
> qemu is basically an interpreter for foreign machine code. If your
> threat model allows access to qemu-user-static for an attacker, they
> can run pretty much any binary is if it were native, and the whole
> SystemCallArchitectures hardening becomes meaningless.

My understanding of the threat is that compatibility syscalls (eg, x32
on amd64) are less well-tested than the local architecture syscalls, and
so allowing apps to call them increases the risk - a compromised app
that can make compatibility syscalls stands a higher probability of
being able to elevate privileges, either in userland or to the kernel
itself. Allowing qemu to translate syscalls from other architectures to
the local syscall ABI doesn't increase that risk, so isn't a concern.
The goal isn't to prevent code form other architectures from running,
it's to reduce the attack surface by preventing calls to the
compatbility syscalls.


Not just that, but also using compatibility syscalls allows
circumventing other systemd hardening that filters syscalls
(RestrictAddressFamilies=, MemoryDenyWriteExecute=, SystemCallFilter=).
Allowing qemu also doesn't make much difference either way to that
though - if your process can't make a syscall directly, it can't make it
via qemu either.


Hm, I didn't think of it that way. Thank you both for your insights!


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: systmd-analyze security as a release goal

[Colin Watson]

On Fri, Jul 14, 2023 at 08:08:39AM +0100, Matthew Garrett wrote:
> On Thu, Jul 13, 2023 at 08:03:39PM +0200, Timo Röhling wrote:
> > qemu is basically an interpreter for foreign machine code. If your
> > threat model allows access to qemu-user-static for an attacker, they
> > can run pretty much any binary is if it were native, and the whole
> > SystemCallArchitectures hardening becomes meaningless.
> 
> My understanding of the threat is that compatibility syscalls (eg, x32 
> on amd64) are less well-tested than the local architecture syscalls, and 
> so allowing apps to call them increases the risk - a compromised app 
> that can make compatibility syscalls stands a higher probability of 
> being able to elevate privileges, either in userland or to the kernel 
> itself. Allowing qemu to translate syscalls from other architectures to 
> the local syscall ABI doesn't increase that risk, so isn't a concern. 
> The goal isn't to prevent code form other architectures from running, 
> it's to reduce the attack surface by preventing calls to the 
> compatbility syscalls.

Not just that, but also using compatibility syscalls allows
circumventing other systemd hardening that filters syscalls
(RestrictAddressFamilies=, MemoryDenyWriteExecute=, SystemCallFilter=).
Allowing qemu also doesn't make much difference either way to that
though - if your process can't make a syscall directly, it can't make it
via qemu either.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: systmd-analyze security as a release goal

[Matthew Garrett]

On Thu, Jul 13, 2023 at 08:03:39PM +0200, Timo Röhling wrote:

> qemu is basically an interpreter for foreign machine code. If your
> threat model allows access to qemu-user-static for an attacker, they
> can run pretty much any binary is if it were native, and the whole
> SystemCallArchitectures hardening becomes meaningless.

My understanding of the threat is that compatibility syscalls (eg, x32 
on amd64) are less well-tested than the local architecture syscalls, and 
so allowing apps to call them increases the risk - a compromised app 
that can make compatibility syscalls stands a higher probability of 
being able to elevate privileges, either in userland or to the kernel 
itself. Allowing qemu to translate syscalls from other architectures to 
the local syscall ABI doesn't increase that risk, so isn't a concern. 
The goal isn't to prevent code form other architectures from running, 
it's to reduce the attack surface by preventing calls to the 
compatbility syscalls.

Re: systmd-analyze security as a release goal

[Timo Röhling]

* Guillem Jover  [2023-07-13 19:36]:

The same would apply to any other interpreted program, as long as the
interpreter matches the systemd native architecture.


This, by the way, includes the following scenario:

* Trent W. Buck  [2023-07-06 10:41]:

   dpkg --add-architecture arm64
   apt update
   apt install mg:arm64 qemu-user-static
   systemctl edit dpkg-db-backup


qemu is basically an interpreter for foreign machine code. If your
threat model allows access to qemu-user-static for an attacker, they
can run pretty much any binary is if it were native, and the whole
SystemCallArchitectures hardening becomes meaningless.


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: systmd-analyze security as a release goal

[Guillem Jover]

Hi!

On Thu, 2023-07-06 at 18:41:38 +1000, Trent W. Buck wrote:
> "Trent W. Buck"  writes:
> > e.g. I expect "SystemCallArchitectures=native" to break for a lot of
> > people (anyone doing dpkg --add-architecture)

Yes, see #982456.

> Short version:
> 
>   • SystemCallArchitectures=native + debianutils:i386 doesn't break 
> dpkg-db-backup.service.
>   • Probably savelog simply never calls an ARCHITECTURE-SPECIFIC syscall.

That's because the only thing used from debianutils:i386 is savelog
which is a shell script, and as long as you have dash:amd64 then that
would work.

The same would apply to any other interpreted program, as long as the
interpreter matches the systemd native architecture.

>   • SystemCallArchitectures=native + nginx:i386 DOES break nginx.service.
>   • Neither journalctl nor coredumpctl makes it obvious this is WHY nginx 
> crashed.

Right.

> So I guess a program like savelog doesn't trigger it, because
> it's so simple it never hits an architecture-specific syscall?

Any binary program, will always at least call 1 syscall (either the
implicit exit() from the libc CRT function calling main(), or an
explicit exit() or raise() or similar to be able to terminate
itself. So there's no escape from this.

So SystemCallArchitectures=* seems fine as a local admin tool when they
control their deployment or for very controlled distribution builders.
Otherwise for a general purpose distribution where you can install
foreign packages, it has always seemed like a bad idea to include them
as part of the .deb, because it's depending on external implementation
details, and this is not helped with the error failure mode.

As I mentioned now on that report above, this could be made safe and
be enabled for I guess all installed services (for free!) by
"something" generating overrides based on the current list of dpkg
architectures (could perhaps be added as a hook on dpkg for the add
and remove architecture actions, or maybe as a generator, dunno).

Thanks,
Guillem

Re: Re: systmd-analyze security as a release goal

[Paul Wise]

On Sun, 2023-07-09 at 18:02 +0100, Luca Boccassi wrote:

> Note that we already have such a package in the archive: dbus-broker.
> It has been the default in Fedora for a long time, and it will be the
> default in Ubuntu in the future. It has been available in Debian since
> Bullseye - please help out testing it by installing it. No
> configuration is required, just installing dbus-broker and rebooting.
> It comes with some sandboxing by default (ProtectSystem=full), and I'm
> sure it could use more.

I've been using this for a while, it seems to just work. It also has
the advantage that moving both system and user dbus services into
systemd means that you can restart them with systemctl etc.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: systmd-analyze security as a release goal

[Luca Boccassi]

On Thu, 6 Jul 2023 at 09:42, Trent W. Buck  wrote:
>
> "Trent W. Buck"  writes:
> > e.g. I expect "SystemCallArchitectures=native" to break for a lot of
> > people (anyone doing dpkg --add-architecture)
>
> Short version:
>
>   • SystemCallArchitectures=native + debianutils:i386 doesn't break 
> dpkg-db-backup.service.
>   • Probably savelog simply never calls an ARCHITECTURE-SPECIFIC syscall.
>
>   • SystemCallArchitectures=native + nginx:i386 DOES break nginx.service.
>   • Neither journalctl nor coredumpctl makes it obvious this is WHY nginx 
> crashed.
>
>
> Boring detailed version follows.
>
> I tried to trigger this (SystemCallArchitectures=native vs. dpkg 
> --add-architecture) just now, and I can't!
>
> On an amd64 Debian 12 VM, I tried
>
> dpkg --add-architecture i386
> apt update
> apt install --allow-remove-essential debianutils:i386 debianutils:amd64-
> systemctl edit dpkg-db-backup
>
># Adding these:
>[Service]
>ReadWritePaths=/var/backups
>CapabilityBoundingSet=
>NoNewPrivileges=yes
>PrivateDevices=yes
>ProtectClock=yes
>ProtectKernelLogs=yes
>ProtectControlGroups=yes
>ProtectKernelModules=yes
>SystemCallArchitectures=native
>
> systemctl start dpkg-db-backup
> systemctl status dpkg-db-backup
>
> It seems to be running savelog:i386 happily.
>
> Then I tried a completely alien architecture,
> in case i386-on-amd64 was somehow special:
>
> dpkg --add-architecture arm64
> apt update
> apt install mg:arm64 qemu-user-static
> systemctl edit dpkg-db-backup
>
># Adding these:
>[Service]
>ExecStart=
>ExecStart=mg tmp.txt
>[Service]
>ReadWritePaths=/var/backups
>CapabilityBoundingSet=
>NoNewPrivileges=yes
>PrivateDevices=yes
>ProtectClock=yes
>ProtectKernelLogs=yes
>ProtectControlGroups=yes
>ProtectKernelModules=yes
>SystemCallArchitectures=native
>
> systemctl start dpkg-db-backup
> systemctl status dpkg-db-backup
>
>mg[1552]: panic: standard input and output must be a terminal
>
> And that worked (in the sense that systemd ran mg enough for it to call 
> printf).
>
> I also thought that it might not work in linux-image-cloud-amd64, so
> I switched to linux-image-amd64, but
> it didn't seem to help -- systemd wasn't blocking things.
>
> The main "user story" for SystemCallArchitectures=native is
> if an attacker replaces (say) /bin/sh with a compromised binary.
> Usually they use i386, so it works on both i386 and amd64 systems.
> So if you do SystemCallArchitectures=native on amd64, it SHOULD just go
> "haha no, this is i386, piss off".
>
> Ah OK, on rereading the manpage,
> https://manpages.debian.org/bookworm/systemd/systemd.exec.5.en.html#SystemCallArchitectures=
> it seems like this just blocks non-amd64 syscalls.
> So I guess a program like savelog doesn't trigger it, because
> it's so simple it never hits an architecture-specific syscall?
>
> Also (probably) when mg:arm64 transits through qemu-user-static,
> by the time the enforcing layer sees it, the syscalls are native amd64 
> syscalls.
>
> Let's test a more complicated program, like nginx:i386...
> OK, I can make that fail.  Phew!  I thought I was going mad.
>
> root@main:~# systemctl show -p SystemCallArchitectures nginx
> SystemCallArchitectures=native
>
> root@main:~# systemctl start nginx
> Job for nginx.service failed because a fatal signal was delivered causing 
> the control process to dump core.
> See "systemctl status nginx.service" and "journalctl -xeu nginx.service" 
> for details.
>
> root@main:~# systemctl status nginx
> × nginx.service - A high performance web server and a reverse proxy server
>  Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: 
> enabled)
> Drop-In: /etc/systemd/system/nginx.service.d
>  └─hardening.conf
>  Active: failed (Result: core-dump) since Thu 2023-07-06 18:32:40 
> AEST; 3s ago
>Duration: 2min 32.918s
>Docs: man:nginx(8)
> Process: 2919 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; 
> master_process on; (code=dumped, signal=SYS)
> CPU: 2ms
>
> Jul 06 18:32:40 main.lan systemd[1]: Starting nginx.service - A high 
> performance web server and a reverse proxy server...
> Jul 06 18:32:40 main.lan systemd[1]: nginx.service: Control process 
> exited, code=dumped, status=31/SYS
> Jul 06 18:32:40 main.lan systemd[1]: nginx.service: Failed with result 
> 'core-dump'.
> Jul 06 18:32:40 main.lan systemd[1]: Failed to start nginx.service - A 
> high performance web server and a reverse proxy server.
>
> root@main:~# coredumpctl
> TIME  PID UID GID SIGCOREFILE EXE 
>  SIZE
> Thu 2023-07-06 18:32:40 AEST 2919   0   0 SIGSYS present  /usr/sbin/nginx 
> 27.1K
>
> 

Re: Re: systmd-analyze security as a release goal

[Luca Boccassi]

On Tue, 4 Jul 2023 at 09:28, Josh Triplett  wrote:
>
> Simon McVittie wrote:
> > For example, dbus-daemon can only usefully have hardening applied if it
> > was built with traditional (non-systemd) service activation disabled,
> > which we cannot usefully do in Debian for two reasons: because we support
> > non-systemd init systems, and because we don't (currently) require
> > every D-Bus system service to have a corresponding systemd system unit.
> > Because of the way traditional activation works, a child process of a
> > setuid-root helper that is run by dbus-daemon must be allowed to exercise
> > any privilege that might legitimately be needed by any D-Bus-activated
> > system service, which rules out otherwise useful things like ProtectSystem.
>
> If we do want to further lock down D-Bus, we could have the D-Bus
> package build a variant that doesn't support traditional activation (for
> use on systemd-only systems), and a variant that does (for use on other
> systems). Then, we could work towards ensuring every D-Bus service
> supports service-based activation rather than only traditional
> activation. Over the course of a release cycle or so, we *could* get to
> the point of being able to lock down D-Bus on systemd systems.

Note that we already have such a package in the archive: dbus-broker.
It has been the default in Fedora for a long time, and it will be the
default in Ubuntu in the future. It has been available in Debian since
Bullseye - please help out testing it by installing it. No
configuration is required, just installing dbus-broker and rebooting.
It comes with some sandboxing by default (ProtectSystem=full), and I'm
sure it could use more.

Kind regards,
Luca Boccassi

Re: systmd-analyze security as a release goal

[Trent W. Buck]

"Trent W. Buck"  writes:
> e.g. I expect "SystemCallArchitectures=native" to break for a lot of
> people (anyone doing dpkg --add-architecture)

Short version:

  • SystemCallArchitectures=native + debianutils:i386 doesn't break 
dpkg-db-backup.service.
  • Probably savelog simply never calls an ARCHITECTURE-SPECIFIC syscall.

  • SystemCallArchitectures=native + nginx:i386 DOES break nginx.service.
  • Neither journalctl nor coredumpctl makes it obvious this is WHY nginx 
crashed.


Boring detailed version follows.

I tried to trigger this (SystemCallArchitectures=native vs. dpkg 
--add-architecture) just now, and I can't!

On an amd64 Debian 12 VM, I tried

dpkg --add-architecture i386
apt update
apt install --allow-remove-essential debianutils:i386 debianutils:amd64-
systemctl edit dpkg-db-backup

   # Adding these:
   [Service]
   ReadWritePaths=/var/backups
   CapabilityBoundingSet=
   NoNewPrivileges=yes
   PrivateDevices=yes
   ProtectClock=yes
   ProtectKernelLogs=yes
   ProtectControlGroups=yes
   ProtectKernelModules=yes
   SystemCallArchitectures=native

systemctl start dpkg-db-backup
systemctl status dpkg-db-backup

It seems to be running savelog:i386 happily.

Then I tried a completely alien architecture,
in case i386-on-amd64 was somehow special:

dpkg --add-architecture arm64
apt update
apt install mg:arm64 qemu-user-static
systemctl edit dpkg-db-backup

   # Adding these:
   [Service]
   ExecStart=
   ExecStart=mg tmp.txt
   [Service]
   ReadWritePaths=/var/backups
   CapabilityBoundingSet=
   NoNewPrivileges=yes
   PrivateDevices=yes
   ProtectClock=yes
   ProtectKernelLogs=yes
   ProtectControlGroups=yes
   ProtectKernelModules=yes
   SystemCallArchitectures=native

systemctl start dpkg-db-backup
systemctl status dpkg-db-backup

   mg[1552]: panic: standard input and output must be a terminal

And that worked (in the sense that systemd ran mg enough for it to call printf).

I also thought that it might not work in linux-image-cloud-amd64, so
I switched to linux-image-amd64, but
it didn't seem to help -- systemd wasn't blocking things.

The main "user story" for SystemCallArchitectures=native is
if an attacker replaces (say) /bin/sh with a compromised binary.
Usually they use i386, so it works on both i386 and amd64 systems.
So if you do SystemCallArchitectures=native on amd64, it SHOULD just go
"haha no, this is i386, piss off".

Ah OK, on rereading the manpage,
https://manpages.debian.org/bookworm/systemd/systemd.exec.5.en.html#SystemCallArchitectures=
it seems like this just blocks non-amd64 syscalls.
So I guess a program like savelog doesn't trigger it, because
it's so simple it never hits an architecture-specific syscall?

Also (probably) when mg:arm64 transits through qemu-user-static,
by the time the enforcing layer sees it, the syscalls are native amd64 syscalls.

Let's test a more complicated program, like nginx:i386...
OK, I can make that fail.  Phew!  I thought I was going mad.

root@main:~# systemctl show -p SystemCallArchitectures nginx
SystemCallArchitectures=native

root@main:~# systemctl start nginx
Job for nginx.service failed because a fatal signal was delivered causing 
the control process to dump core.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" 
for details.

root@main:~# systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
 Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: 
enabled)
Drop-In: /etc/systemd/system/nginx.service.d
 └─hardening.conf
 Active: failed (Result: core-dump) since Thu 2023-07-06 18:32:40 AEST; 
3s ago
   Duration: 2min 32.918s
   Docs: man:nginx(8)
Process: 2919 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; 
master_process on; (code=dumped, signal=SYS)
CPU: 2ms

Jul 06 18:32:40 main.lan systemd[1]: Starting nginx.service - A high 
performance web server and a reverse proxy server...
Jul 06 18:32:40 main.lan systemd[1]: nginx.service: Control process exited, 
code=dumped, status=31/SYS
Jul 06 18:32:40 main.lan systemd[1]: nginx.service: Failed with result 
'core-dump'.
Jul 06 18:32:40 main.lan systemd[1]: Failed to start nginx.service - A high 
performance web server and a reverse proxy server.

root@main:~# coredumpctl
TIME  PID UID GID SIGCOREFILE EXE  
SIZE
Thu 2023-07-06 18:32:40 AEST 2919   0   0 SIGSYS present  /usr/sbin/nginx 
27.1K

root@main:~# coredumpctl info
   PID: 2919 (nginx)
   UID: 0 (root)
   GID: 0 (root)
Signal: 31 (SYS)
 Timestamp: Thu 2023-07-06 18:32:40 AEST (13s ago)
  Command Line: /usr/sbin/nginx -t -q -g $'daemon on; master_process on;'

Re: systmd-analyze security as a release goal

[Trent W. Buck]

Russ Allbery  writes:

> "Trent W. Buck"  writes:
>
>> As someone who does that kind of thing a lot, I'd rather have
>> the increased annoyance of opt-out hardening than
>> the reduced security of opt-in hardening.
>> Even if it means I occasionally need to patch site-local rules into
>> /etc/apparmor.d/local/usr.bin.msmtp or
>> /etc/systemd/system/libvirtd.service.d/override.conf.
>
> I also feel this way but there are a bunch of people who really, really
> don't, and also it's not entirely obvious when hardening is failing or
> what overrides you need to add.  So making this the default is hard,
> because it fundamentally breaks the "it has to work out of the box"
> property that people expect.  Making it be semi-normal for daemons to not
> work out of the box depending on what configuration options or other
> packages you have installed is a hard sell.

I agree; that is reasonable.

> That makes me want some way to opt in to "hardening that might break
> something," but I'm not sure the best way to do that.

I think for me the important point is whether "might break something"
means a chance of in 1 in one hundred, or 1 in one crore.

e.g. I expect "SystemCallArchitectures=native" to break for a lot of
people (anyone doing dpkg --add-architecture)

e.g. I expect "ProtectClock=yes" to break for very few people, because
it's really unlikely that any libnss-foo.so or libpam-foo.so is
going to try to change the hardware clock.

(Although ProtectClock= requires NoNewPrivileges which is a whole
separate kettle of fish, as discussed elsewhere in this thread.)

Re: systmd-analyze security as a release goal

[Russ Allbery]

"Trent W. Buck"  writes:

> As someone who does that kind of thing a lot, I'd rather have
> the increased annoyance of opt-out hardening than
> the reduced security of opt-in hardening.
> Even if it means I occasionally need to patch site-local rules into
> /etc/apparmor.d/local/usr.bin.msmtp or
> /etc/systemd/system/libvirtd.service.d/override.conf.

I also feel this way but there are a bunch of people who really, really
don't, and also it's not entirely obvious when hardening is failing or
what overrides you need to add.  So making this the default is hard,
because it fundamentally breaks the "it has to work out of the box"
property that people expect.  Making it be semi-normal for daemons to not
work out of the box depending on what configuration options or other
packages you have installed is a hard sell.

That makes me want some way to opt in to "hardening that might break
something," but I'm not sure the best way to do that.

-- 
Russ Allbery (r...@debian.org)  

Re: systmd-analyze security as a release goal

[Trent W. Buck]

Russ Allbery  writes:

> [⋯]
> We know which PAM modules are installed and
> can analyze the PAM configuration files to know which ones are configured.
> We know which daemons use PAM.
> We similarly know which NSS modules are enabled.
> We can figure out what facilities they require, and could
> automate adding additional permitted facilities to a locked-down base
> configuration, so when you enable a PAM module that requires some
> functionality, it automatically updates the systemd configuration of all
> of the daemons that use PAM.

I broadly agree.
The fine details will be a bit annoying, though
that might be fixable by pushing some helper tools into systemd.

> (And of course additional narrowing to just
> the daemons that are configured to use that PAM module is possible if we
> know which PAM name the daemon uses.)

This last bit would be nontrivial to get 100% perfectly correct, but
I think we could get it 95% correct, and have debian/NEWS inform users
"if you're edge case X, do workaround Y".

An edge case I can think of is daemons that fork+exec other daemons.
For example, libvirtd may run qemu-system-x86_64, which may run smbd.
You could trigger that if you did manually added manual CLI options to "virsh 
edit my-cool-vm":


  
  


If all of that happens in a single systemd .service (which, admittedly, it 
probably DOESN'T),
and qemu's minimal smbd instance uses /etc/pam.d/samba (which, admittedly, it 
probably DOESN'T),
then libvirtd.service's hardening will apply to libpam_foo.so called by 
/etc/pam.d/samba.

I can't think of a detector that could notice in advance the libvirtd.service 
was gonna use samba pam rules.

As someone who does that kind of thing a lot, I'd rather have
the increased annoyance of opt-out hardening than
the reduced security of opt-in hardening.
Even if it means I occasionally need to patch site-local rules into
/etc/apparmor.d/local/usr.bin.msmtp or
/etc/systemd/system/libvirtd.service.d/override.conf.

Re: systmd-analyze security as a release goal

[Trent W. Buck]

Philipp Kern  writes:

> On 2023-07-05 09:36, Russell Coker wrote:
>> On Monday, 3 July 2023 22:37:35 AEST Russell Coker wrote:
>>> https://wiki.debian.org/ReleaseGoals/SystemdAnalyzeSecurity

> My fear here would be that you are not in control of what your
> dependencies are doing. This is especially true if you think of NIS
> and PAM, where libraries are dlopen()ed and can spawn arbitrary helper
> binaries. I remember openssh installing a syscall filter for its auth
> binary and then it failed with certain PAM modules (see also your
> allow_ypbind example). So we should also not be too limiting when
> sandboxing daemons.

I had this problem with PADL libpam-ldap, which
loads an entire buggy LDAP client (and all of openssl!) into EVERY process.
But that was fixed many releases ago with de Jong's nss-pam-ldapd, which
merely has each process make an AF_UNIX connection to a separate process 
(nslcd).
I think libnss-sss also works this way (AF_UNIX to sssd).

For the specific NIS+ example my reaction was:
Is anyone even still using NIS+?
And if so, do they care about security AT ALL?
They'll just disable systemd hardening on day 1, like
they already do for selinux and apparmor.

I agree that (almost) every frobozzd will use nss or pam, and
the frobozzd maintainer can't stop the end user doing
"apt install libnss-insane" and crashing frobozzd,
because frobozzd.service hardening isn't compatible with libnss-insane.
Because libnss-insane could do ANYTHING, and
it'll do so within the Linux namespace/cgroup context of frobozzd.service.

I think there's a reasonable middle ground, though.
I think the number of libnss-insane packages that do TRULY wacky things are 
limited, and
the number of people using them are small, and
the inconvenience of them having to opt out (with "systemctl edit") is worth the
improved security for everybody else.

I think for most pam and nss packages, allowing (not blocking/hardening)
AF_UNIX (e.g. sss, ldapd) and
AF_NETLINK (e.g. systemd, mymachines, resolve) is probably going to cover it.

Unfortunately I don't (yet) have any hard data to back up these feelings.


PS: PADL is gone in Debian 12: 
https://tracker.debian.org/news/1424430/libpam-ldap-removed-from-testing/
PS: NIS is off(ish) by default in Debian 11+: 
https://wiki.debian.org/BullseyeNis
PS: NIS+ seems to be in Debian 12, but deprecated 
https://salsa.debian.org/glibc-team/libnss-nisplus 
https://tracker.debian.org/pkg/libnss-nisplus


PS: in theory if libnss-insane is known to break frobozzd,
libnss-insane can pre-ship the errata for each daemon:

libnss-insane-common_1-1_all.deb:
/lib/systemd/system/frobozzd.service.d/libnss-insane.conf:
# frobozzd.service ships with
#   SystemCallFilter=@system-service
#   SystemCallFilter=~@resource @resource
# but libnss-insane needs access to nice(2) from @resource.
# Add that to the allowlist.
SystemCallFilter=nice

but it would have to do that for EVERY affected .service.
There isn't a way for libnss-insane to add a "global" allow rule.

Re: systmd-analyze security as a release goal

[Russ Allbery]

Philipp Kern  writes:

> My fear here would be that you are not in control of what your
> dependencies are doing. This is especially true if you think of NIS and
> PAM, where libraries are dlopen()ed and can spawn arbitrary helper
> binaries. I remember openssh installing a syscall filter for its auth
> binary and then it failed with certain PAM modules (see also your
> allow_ypbind example). So we should also not be too limiting when
> sandboxing daemons.

I am hesitant to propose work that I personally will not have time to work
on, but that feels like a place where Debian, as a distribution, could add
a great deal of value.  We know which PAM modules are installed and can
analyze the PAM configuration files to know which ones are configured.  We
know which daemons use PAM.  We similarly know which NSS modules are
enabled.  We can figure out what facilities they require, and could
automate adding additional permitted facilities to a locked-down base
configuration, so when you enable a PAM module that requires some
functionality, it automatically updates the systemd configuration of all
of the daemons that use PAM.  (And of course additional narrowing to just
the daemons that are configured to use that PAM module is possible if we
know which PAM name the daemon uses.)

-- 
Russ Allbery (r...@debian.org)  

Re: systmd-analyze security as a release goal

[Philipp Kern]

On 2023-07-05 09:36, Russell Coker wrote:

On Monday, 3 July 2023 22:37:35 AEST Russell Coker wrote:

https://wiki.debian.org/ReleaseGoals/SystemdAnalyzeSecurity


People have asked how hard it is to create policy for daemons.  For an
individual to create them it's a moderate amount of work, 1-2 hours per 
daemon
which is a lot considering the dozens of daemons that people use.  But 
for a
group of people it's not a big deal, it's almost nothing compared to 
the scale
of Debian development work.  The work that I've done writing SE Linux 
policy
for daemons is significantly greater than what I'd like the collective 
of DDs

to do in this regard.


My fear here would be that you are not in control of what your 
dependencies are doing. This is especially true if you think of NIS and 
PAM, where libraries are dlopen()ed and can spawn arbitrary helper 
binaries. I remember openssh installing a syscall filter for its auth 
binary and then it failed with certain PAM modules (see also your 
allow_ypbind example). So we should also not be too limiting when 
sandboxing daemons.


Kind regards
Philipp Kern

Re: systmd-analyze security as a release goal

[Russell Coker]

On Monday, 3 July 2023 22:37:35 AEST Russell Coker wrote:
> https://wiki.debian.org/ReleaseGoals/SystemdAnalyzeSecurity

People have asked how hard it is to create policy for daemons.  For an 
individual to create them it's a moderate amount of work, 1-2 hours per daemon 
which is a lot considering the dozens of daemons that people use.  But for a 
group of people it's not a big deal, it's almost nothing compared to the scale 
of Debian development work.  The work that I've done writing SE Linux policy 
for daemons is significantly greater than what I'd like the collective of DDs 
to do in this regard.

One thing that can be done to assist in this process is looking at SE Linux or 
AppArmor policy.  For example the following is an example of how to get the 
capabilities needed for crond which shows that when you don't need to have 
net_bind_service for NIS+ or { fsetid sys_admin } for polyinstantiation you 
only need { audit_control audit_write chown dac_override dac_read_search 
fowner setgid setuid sys_nice sys_resource }.

# sesearch -A -s crond_t -c capability
allow crond_t crond_t:capability net_bind_service; [ allow_ypbind ]:True
allow crond_t crond_t:capability net_bind_service; [ allow_ypbind ]:True
allow crond_t crond_t:capability net_bind_service; [ allow_ypbind ]:True
allow crond_t crond_t:capability net_bind_service; [ allow_ypbind ]:True
allow crond_t crond_t:capability { audit_control audit_write chown 
dac_override dac_read_search fowner setgid setuid sys_nice sys_resource };
allow crond_t crond_t:capability { chown fowner fsetid sys_admin }; [ 
allow_polyinstantiation ]:True

Now this is NOT the sum of all access needed by everything that crond might 
run, just everything that runs without running a setuid program etc.  Also 
there's nothing stopping regular users from using something like bwrap to wrap 
their own cron jobs, and this would be a good practice that we should 
encourage.

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/

Re: systmd-analyze security as a release goal

[Trent W. Buck]

Marco d'Itri  writes:
> This is a good example of what an almost fully sandboxed service looks like:
> https://salsa.debian.org/md/rpki-client/-/blob/master/debian/rpki-client.service

My best score is a little better :-)

On Debian 11 (systemd v247):
→ Overall exposure level for collection4.service: 0.2 SAFE 
→ Overall exposure level for rpki-client.service: 1.7 OK 

On Sid (systemd v253):
→ Overall exposure level for collection4.service: 0.4 SAFE 
→ Overall exposure level for rpki-client.service: 1.8 OK 

https://github.com/trentbuck/collection4/blob/main/debian/service


PS: I worked out to invoke it in offline mode (for lintian) you do this:

  systemd-analyze --offline=yes ./path/to/foo.service

I didn't understand (from the manpage) that I could pass a file instead of 
a unit name, so
I wasted a lot of time trying to make a minimal --root=tmpdir work.
Also it won't accept "./debian/service", nor a symlink to same.

Re: systmd-analyze security as a release goal

[Trent W. Buck]

Marco d'Itri  writes:

> On Jul 04, Andrey Rakhmatullin  wrote:
>
>> Cool but looks like a lot of work.

[...]

>> start with applying all of them and then looking what needs to be
>> disabled?
> This is what I do.

FYI below is my basic workflow.
Once you've done 2-5 daemons, you get a "feel" for the trouble spots.
Total time to harden a unit from EXPOSURE=10 to EXPOSURE=3 usually takes me 1-4 
hours.
If I've used the daemon before & know its config format & source code, usually 
1 hour.

I typically start with a "deny all" ruleset.
Either I copy-paste from another daemon I did earlier, or
I copy-paste from "systemd-analyze security".
A slightly out-of-date one is
https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/blob/main/systemd/system/0-EXAMPLES/20-default-deny.conf

Usually the daemon segfaults immediately.
In "coredumpctl" I see what the last syscall was.
Typically it is setuid so per I know to allowlist these:

SystemCallFilter=@setuid
CapabilityBoundingSet=CAP_SETUID CAP_SETGID

This is because the daemon does a no-op setuid(123) even if it's ALREADY 123 
(due to User=%p in frobozzd.service).
This could be patched away, but so far my policy has been
"focus on stuff that doesn't require patching", so
instead I just allow that syscall.

It is very common to need both AF_UNIX and AF_NETLINK, so I don't even try to 
block those.
Things that need network (e.g. postfix, nginx) would also need AF_INET, 
AF_INET6, IPAddressAllow=all, 

The next most common failure is being unable to write to somewhere due to 
ProtectSystem=strict,
so I look for things like /run/frobozzd.pid or /var/lib/frobozzd/state.db in 
the error logs (journalctl -u frobozzd).
If systemd's existing things like RuntimeDirectory=%p aren't enough to cover 
it, I add ReadWritePaths=, or
downgrade ProtectSystem=strict to ProtectSystem=yes.

If it's still crashing, I remove "SystemCallFilter=~@privileged @resources" and 
"CapabilityBoundingSet=" entirely.
If that works, I strace or bisect to find which syscalls must be allowlisted.

If it's *STILL* crashing, I bisect over the entire hardening denylist.
(Comment out half.  Does it work now?  If so, it's mad about the commented-out 
half.  Repeat.)


The hardest part is the rare case where a daemon will automatically detect that 
an action failed, then
*silently* switch to a less-secure mode.
It is very hard to spot this is happening until after the hardened unit has 
been in production for a month or two.


PS: I typically have a dev loop like:

  journalctl -fu frobozzd &

  while ! systemctl restart frobozzd;
  do systemctl edit frobozzd; done

Or if it's on another host,

  M-! /etc/systemd/system/frobozzd.service.d/hardening.conf;
  systemctl daemon-reload;
  systemctl restart frobozzd;
  systemctl status frobozzd'

PPS: so far I've been talking about system units, but
 user units can also have hardening!

 For example, I bet this only needs write access to /sys/blah/rfkill, and
 could have it's TCP privileges revoked:

org.gnome.SettingsDaemon.Rfkill.service 9.8 UNSAFE 

 Also by default "systemd-analyze security" doesn't mention 
timer/path-fired units like e2scrub or fsck.
 If you want to see those you have to do something like "systemctl 
list-units --all --type=service".

Re: systmd-analyze security as a release goal

[Marco d'Itri]

On Jul 04, Andrey Rakhmatullin  wrote:

> Cool but looks like a lot of work.
I do not think that this is really a lot of work.

> Is it possible to do this without
> applying the flags one by one and testing the result? Is it easier to
You may intimately know what the daemon needs to do and how the 
sandboxing features work, and be able to mentally create a correct and 
complete configuration.

> start with applying all of them and then looking what needs to be
> disabled?
This is what I do.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: systmd-analyze security as a release goal

[Andrey Rakhmatullin]

On Mon, Jul 03, 2023 at 11:40:18PM +0200, Marco d'Itri wrote:
> This is a good example of what an almost fully sandboxed service looks 
> like:
> 
> https://salsa.debian.org/md/rpki-client/-/blob/master/debian/rpki-client.service
Cool but looks like a lot of work. Is it possible to do this without
applying the flags one by one and testing the result? Is it easier to
start with applying all of them and then looking what needs to be
disabled?

Re: Re: systmd-analyze security as a release goal

[Josh Triplett]

Simon McVittie wrote:
> For example, dbus-daemon can only usefully have hardening applied if it
> was built with traditional (non-systemd) service activation disabled,
> which we cannot usefully do in Debian for two reasons: because we support
> non-systemd init systems, and because we don't (currently) require
> every D-Bus system service to have a corresponding systemd system unit.
> Because of the way traditional activation works, a child process of a
> setuid-root helper that is run by dbus-daemon must be allowed to exercise
> any privilege that might legitimately be needed by any D-Bus-activated
> system service, which rules out otherwise useful things like ProtectSystem.

If we do want to further lock down D-Bus, we could have the D-Bus
package build a variant that doesn't support traditional activation (for
use on systemd-only systems), and a variant that does (for use on other
systems). Then, we could work towards ensuring every D-Bus service
supports service-based activation rather than only traditional
activation. Over the course of a release cycle or so, we *could* get to
the point of being able to lock down D-Bus on systemd systems.

Re: systmd-analyze security as a release goal

[Cyril Brulebois]

Hi,

Jonathan Carter  (2023-07-03):
> One shouldn't trust everything that is read on Matrix. I suggest asking
> the release team for clarification, because my last understanding is
> that release goals still exist, it's just that the release team doesn't
> want to be the entity pushing it.

FWIW, just a few months ago:
  https://lists.debian.org/debian-release/2023/02/msg00015.html


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: systmd-analyze security as a release goal

[Marco d'Itri]

On Jul 03, RL  wrote:

> (One of the issues for services that send email is that it is very
> easy to break exim)
NoNewPrivileges breaks by design anything which depends on suid, so Exim 
and (in the default configuration) Postfix.

I agree that we should do much better in terms on sandboxing, 
considering that it is so easy with systemd.
I am not sure if it can be a release goal, but it definitely should be 
a best practice.

This is a good example of what an almost fully sandboxed service looks 
like:

https://salsa.debian.org/md/rpki-client/-/blob/master/debian/rpki-client.service

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: systmd-analyze security as a release goal

[Simon McVittie]

On Mon, 03 Jul 2023 at 20:21:20 +0100, RL wrote:
> (One of the issues for services that send email is that it is very
> easy to break exim)

It's also very easy to break anything else that relies on running a
setuid/setgid/setcap executable (including many mail delivery agents,
not just Exim), as the maintainers of systemd-cron already discovered.

More generally, systemd's hardening features are a lot like AppArmor:
great for services that have no business executing arbitrary code and
would be simple to put in a minimal container (for example most game
servers), but difficult or even impossible to use successfully for
services that are expected to be allowed to execute arbitrary code via
plugins, hooks or configuration.

For example, dbus-daemon can only usefully have hardening applied if it
was built with traditional (non-systemd) service activation disabled,
which we cannot usefully do in Debian for two reasons: because we support
non-systemd init systems, and because we don't (currently) require
every D-Bus system service to have a corresponding systemd system unit.
Because of the way traditional activation works, a child process of a
setuid-root helper that is run by dbus-daemon must be allowed to exercise
any privilege that might legitimately be needed by any D-Bus-activated
system service, which rules out otherwise useful things like ProtectSystem.

smcv

Re: systmd-analyze security as a release goal

[Richard Laager]

On 2023-07-03 14:21, RL wrote:

Russell Coker  writes:


https://wiki.debian.org/ReleaseGoals/SystemdAnalyzeSecurity

I think we should make it a release goal to have as many daemons as
possible running with systemd security features to aim for a low score
from "systmd- analyze security".


It would be great if we could get a lintian check for this.

The wiki page says, "systemd-analyze now supports working offline" (i.e. 
it can operate on files in the filesystem rather than talking to systemd 
about only installed services). Lack of that was previously a blocker 
for such a lintian check.



This repos from Trent Buck has a lot of research -
https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/tree/main/systemd/system/0-EXAMPLES


Indeed.

--
Richard

Re: systmd-analyze security as a release goal

[RL]

Russell Coker  writes:

> https://wiki.debian.org/ReleaseGoals/SystemdAnalyzeSecurity
>
> I think we should make it a release goal to have as many daemons as
> possible running with systemd security features to aim for a low score
> from "systmd- analyze security".


This repos from Trent Buck has a lot of research -
https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/tree/main/systemd/system/0-EXAMPLES

(One of the issues for services that send email is that it is very
easy to break exim)

Re: systmd-analyze security as a release goal

[Jonathan Carter]

Hi Russell

On 2023/07/03 14:37, Russell Coker wrote:

Someone said on Matrix that we aren't going to have official release goals in
future.


One shouldn't trust everything that is read on Matrix. I suggest asking 
the release team for clarification, because my last understanding is 
that release goals still exist, it's just that the release team doesn't 
want to be the entity pushing it.


-Jonathan

systmd-analyze security as a release goal

[Russell Coker]

Someone said on Matrix that we aren't going to have official release goals in 
future.  If so that doesn't stop people from doing the work just makes it less 
of an issue to release with some of the bugs unsolved.

https://wiki.debian.org/ReleaseGoals/SystemdAnalyzeSecurity

I think we should make it a release goal to have as many daemons as possible 
running with systemd security features to aim for a low score from "systmd-
analyze security".  If it can't be an official release goal we can still use 
the above wiki page to coordinate work.

Anyone who's interested in this can edit that page to track their work.

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/