Bug#1020328: Acknowledgement (Native systemd units)
On Wed 07 Dec 2022 23:46:43 +, Richard Lewis wrote: > I have been studying and experimenting - and learning a lot. > For exim4, i found ⋯ I slurped your exim notes into my repo. I probably won't do any actual testing with exim myself :-) https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/commits/main/systemd/system/0-EXAMPLES/30-allow-mail-exim.conf I was indeed originally plannign to push it into Debian as a kind of "apt install increased-hardening" package. But I ran into enough nitpicking and static, my current approach is to instead try to work on individual packages, and try to push upstreams to be more hardened by default. e.g. SyscallFilters=@foo isn't backwards-compatible with old systemd, so you can't use it if you don't know a minimum systemd version. e.g. Architecture=native works fine until someone does something like "apt install curl:armhf". e.g. the whole "if you call /usr/sbin/sendmail, everything becomes messy" > - whether the unit is 'oneshot' - if so the unit needs to ensure exim has > delivered the mail before the script exits - adding a small 'sleep' is > enough - i think otherwise systemd gets confused about what process to > monitor if the script causes exim to launch. That doesn't sound right, but I suppose it's possible. KillMode=process might trigger that. > - whether or not the unit runs as root or a different user. With User=root > you can get away with more hardening directives, but i think better to > continue running as a non-root user I think you'll find this is just because User=root implicitly disables a bunch of the settings. If you set User=root, then "systemctl daemon-reload" then "systemd-analyze security foo.service", you will see a bunch of stuff like: Service runs as root, option does not matter Service runs as root, option does not apply PS: "systemd-analyze syscall-filter" is a good thing to look at when dealing with chown/seteuid.
Bug#1020328: Acknowledgement (Native systemd units)
On Wed, 9 Nov 2022, 08:37 Trent W. Buck, wrote: > My old (Debian 9) notes about different techniques are here: > > https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/tree/main/systemd/system/0-EXAMPLES > > This is an amazing resource! (did you consider trying to introduce it into a debian package somehow?) I have been studying and experimenting - and learning a lot. For exim4, i found that it depends on - whether the unit is 'oneshot' - if so the unit needs to ensure exim has delivered the mail before the script exits - adding a small 'sleep' is enough - i think otherwise systemd gets confused about what process to monitor if the script causes exim to launch. - whether or not the unit runs as root or a different user. With User=root you can get away with more hardening directives, but i think better to continue running as a non-root user This all makes me want to abandon exim for postfix but exim is still debian's default. I think this is consistent with what you found for other mtas, but for future reference and to help people searching for how to use exim in a systemd unit, ive been using the following with exim: ExecStart=/usr/sbin/logcheck ExecStart=sleep 0.1s User=logcheck UMask=0066 ProtectSystem=strict ReadWritePaths=/var/lib/logcheck # for exim - probably debian should allow all of /var/spool and /var/log here ReadWritePaths=-/var/spool/exim4 -/var/mail -/var/log/exim4 # ProtectHome=true is possible, but the message will be # frozen as exim wants to cd to $HOME (for .forward) ProtectHome=read-only PrivateTmp=true PrivateMounts=true DevicePolicy=strict # *cannot set: PrivateDevices=true DeviceAllow=/dev/stdout w DeviceAllow=/dev/stdin r DeviceAllow=/dev/stderr w DeviceAllow=/dev/null rw ProtectProc=invisible ProcSubset=pid RemoveIPC=true ProtectControlGroups=true AmbientCapabilities= # exim needs to change ownership of mail - both when it # receives the mail and when it delivers it to the local user # see capabilities(7) CapabilityBoundingSet=CAP_SETGID CapabilityBoundingSet=CAP_SETUID CapabilityBoundingSet=CAP_FSETID CapabilityBoundingSet=CAP_CHOWN CapabilityBoundingSet=CAP_DAC_OVERRIDE CapabilityBoundingSet=CAP_FOWNER # Anything that implies NoNewPrivileges cannot be set # cannot set: NoNewPrivileges=yes # cannot set: DynamicUser=yes # cannot set: PrivateUsers=true # *cannot set: RestrictNamespaces=true # *cannot set: LockPersonality=true # *cannot set: ProtectKernelModules=true # *cannot set: ProtectKernelLogs=true # *cannot set: ProtectHostname=true # *cannot set: ProtectClock=true # *cannot set: RestrictRealtime=true # *cannot set: MemoryDenyWriteExecute=true # *cannot set - and would break remote delivery: RestrictAddressFamilies=AF_INET AF_INET6 # *cannot set: RestrictSUIDSGID=true # *cannot set: SystemCallArchitectures=native # *cannot set: SystemCallFilter=@system-service # cannot set (due to chown): SystemCallFilter=~@privileged # directives with a * can be set if we use User=root instead of User=logcheck - This still needs the tmpfiles.d dropin from your earlier email. However, i wonder if it is better to make logcheck use a single dir for both the lockfile (currently /run/lock/logcheck) and the scratch dir ( currently created in /tmp each time) and then we could use a single RuntimeDirectory=logcheck with RuntimeDirectoryPreserve=yes (to ensure it is not deleted and no clashes if logcheck dies it would retain the current code in logcheck to use a random subdir and delete it on success)
Bug#1020328: Acknowledgement (Native systemd units)
On Wed 09 Nov 2022 19:29:56 +1100, Trent W. Buck wrote: > In short, what I'm saying is: > > 1. you can't harden a script/daemon that uses the "fork+exec > /usr/sbin/sendmail" API, because > different /usr/sbin/sendmail implementations (e.g. postfix) require > different privileges. > > In particular, "requires setgid" prevents ALL of the following hardening > options: > > DynamicUser LockPersonality MemoryDenyWriteExecute > NoNewPrivileges PrivateDevices ProtectClock > ProtectHostname ProtectKernelLogs ProtectKernelModules > ProtectKernelTunables RestrictAddressFamilies RestrictNamespaces > RestrictRealtimeRestrictSUIDSGID > SystemCallArchitectures > SystemCallFilterSystemCallLog > > 2. the smtp://localhost:25 API is usually available. > > It prevents fewer hardening options: > > PrivateNetwork=yes > IPAddressDeny=any > RestrictAddressFamilies=~AF_TCP > > Basically you have to leave TCP/IP unblocked, but that's all. I made a minor braino here, it should be AF_INET AF_INET6 not AF_TCP. My old (Debian 9) notes about different techniques are here: https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/tree/main/systemd/system/0-EXAMPLES 30-allow-mail-msmtp.conf: # → Overall exposure level: 1.0 OK 30-allow-mail-postfix-via-msmtp.conf: # → Overall exposure level: 0.9 SAFE 30-allow-mail-postfix-root-dac-override.conf: # → Overall exposure level: 1.1 OK 30-allow-mail-postfix-root-sys-admin.conf:# → Overall exposure level: 1.4 OK 30-allow-mail-postfix-non-root-addgroup.conf: # → Overall exposure level: 0.5 SAFE 30-allow-mail-postfix-non-root-dac-override.conf: # → Overall exposure level: 0.9 SAFE 30-allow-mail-postfix-non-root-setgid.conf: # → Overall exposure level: 2.4 OK
Bug#1020328: Acknowledgement (Native systemd units)
On Fri 04 Nov 2022 00:45:52 +, Richard Lewis wrote: > Hi trent - i am interested in this approach: > > i see you are binding msmtp over /usr/sbin/sendmail - i dont > understand how this would lead to a different outcome: how else does > msmtp know where to send the mail? is there some implicit assumption > about local delivery here? i tried testing but msmtp does not work at > all out of the box for me - complains that it has no configuration > file) If you install msmtp you need to configure it. Just as if you installed postfix, you would need to configure it. "dpkg-reconfigure msmtp" ought to prompt you, but here are some basic examples: 1. $ sudo apt install msmtp postfix $ cat >/etc/msmtprc <<'EOF' # Send everything to postfix (smtp://localhost:25) account default syslog on auto_from on host localhost EOF 2. $ sudo apt install msmtp-mta $ cat >/etc/msmtprc <<'EOF' # Send everything to gmail (no "real" MTA on localhost) account default syslog on auto_from on host smtp.gmail.com port 587 tls on auth on user alice passwordeval /usr/bin/cat /etc/secret.gmail.password EOF $ printf swordfish >/etc/secret.gmail.password $ chmod 640 /etc/secret.gmail.password $ chown -h root:logcheck /etc/secret.gmail.password > As far as i can tell, the issue isn't with the "send mail" part, but > the part where the mta (exim/postfix) tries to deliver it I don't know about exim. When /usr/sbin/sendmail is implemented by postfix (i.e. "apt install postfix), 1. sendmail calls postdrop 2. postdrop is sgid postdrop, so now you run with elevated privileges 3. postdrop writes to /var/spool/postfix/maildrop, which normal users can't write to Note the sgid bit: -rwxr-xr-x 1 rootroot /usr/sbin/sendmail -r-xr-sr-x 1 rootpostdrop /usr/sbin/postdrop drwx-wx--T 2 postfix postdrop /var/spool/postfix/maildrop If you use systemd hardening NoNewPrivileges=yes, that DISABLES SETGID -- by design. So logcheck.service run /usr/bin/logcheck which runs /usr/bin/mail which runs /usr/sbin/sendmail which runs /usr/sbin/postdrop which DOESN'T get escalated privileges (group postdrop) which FAILS to write to /var/spool/postfix/maildrop/. By telling logcheck.service "actually just use msmtp", the path instead becomes logcheck.service runs /usr/bin/logcheck which runs /usr/bin/mail which runs /usr/sbin/sendmail (actually /usr/bin/msmtp) which connects to (say) smtp://localhost:25 or smtp://smtp.gmail.com:587 Thereafter, the rest of the flow (whatever is listening on localhost:25) is not running inside the logcheck.service hardened namespace/cgroup. So it can do whatever it wants. In short, what I'm saying is: 1. you can't harden a script/daemon that uses the "fork+exec /usr/sbin/sendmail" API, because different /usr/sbin/sendmail implementations (e.g. postfix) require different privileges. In particular, "requires setgid" prevents ALL of the following hardening options: DynamicUser LockPersonality MemoryDenyWriteExecute NoNewPrivileges PrivateDevices ProtectClock ProtectHostname ProtectKernelLogs ProtectKernelModules ProtectKernelTunables RestrictAddressFamilies RestrictNamespaces RestrictRealtimeRestrictSUIDSGIDSystemCallArchitectures SystemCallFilterSystemCallLog 2. the smtp://localhost:25 API is usually available. It prevents fewer hardening options: PrivateNetwork=yes IPAddressDeny=any RestrictAddressFamilies=~AF_TCP Basically you have to leave TCP/IP unblocked, but that's all. 3. msmtp is a quick and easy way to convert (1) to (2). 4. "apt install msmtp-mta" does (3) easily, but won't work if a "real" MTA is already installed. 5. BindReadOnlyPaths=/usr/bin/msmtp:/usr/sbin/sendmail does (3), and works even if a "real" MTA is installed.
Bug#1020328: Acknowledgement (Native systemd units)
Hi trent - i am interested in this approach: i see you are binding msmtp over /usr/sbin/sendmail - i dont understand how this would lead to a different outcome: how else does msmtp know where to send the mail? is there some implicit assumption about local delivery here? i tried testing but msmtp does not work at all out of the box for me - complains that it has no configuration file) As far as i can tell, the issue isn't with the "send mail" part, but the part where the mta (exim/postfix) tries to deliver it (returning to logcheck - id agree with losing the 'reboot' mail - it's pretty clear from the log message that the system has been rebooted, so not sure what value this is even under cron. I'd suggest adding Persistent=true in the .timer's [Timer] so that it runs after a resuming from suspend. Perhaps set it to skip if on battery too )
Bug#1020328: Acknowledgement (Native systemd units)
UPDATE: a debian/logcheck.tmpfiles (/etc/tmpfiles.d/logcheck.conf) is also
needed.
The security hardening I added prevents logcheck from creating it.
See attached.
# Hardened logcheck.service started complaining after a reboot:
#
# systemd[1]: Starting logcheck — email sysadmin about anomalous log
events...
# logcheck[807044]: mkdir: cannot create directory ‘/run/lock/logcheck’:
Read-only file system
# msmtp[808400]: host=localhost tls=off auth=off from=logch...@cyber.com.au
recipients=logcheck mailsize=1368 smtpstatus=250 smtpmsg='250 2.0.0 Ok: queued
as CB96A283D8' exitcode=EX_OK
# systemd[1]: logcheck.service: Succeeded.
#
# It didn't complain BEFORE the reboot, presumably because that dir already
existed.
#
https://salsa.debian.org/debian/logcheck/-/blob/debian/1.3.24/src/logcheck#L653-657
#
# Move that "make sure the dir exists" into the separate systemd daemon whose
entire job is to do that.
#
# I have an existing batch job that I don't want to patch.
# It is doing
# LOCKDIR=/run/lock/logcheck
# if [ ! -d "$LOCKDIR" ]; then mkdir -m 0755 "$LOCKDIR" fi
# This fails when the .service is hardened.
# Is there a way to make systemd manage that dir?
# I initially thought RuntimeDirectory=%p, but that's /run/logcheck not
/run/lock/logcheck.
# I don't see anything relevant in "git grep -Fw lock -- man".
# I guess I can write a tmpfiles.d, and then
ReadWritePaths=/run/lock/logcheck...
# Oh the latter is probably implicitly there already.
d /run/lock/logcheck 0755 logcheck logcheck - -
# If we just use Debian 11 default /etc/cron.d/logcheck, with systemd-cron,
# then after an unattended-upgrade, I regularly see this:
#
# $ systemctl status
# ● heavy
# State: degraded
# Jobs: 0 queued
#Failed: 1 units
#
# $ systemctl –state=failed
# UNIT LOAD ACTIVE SUBDESCRIPTION
# ● cron-logcheck-logcheck-1.timer not-found failed failed
cron-logcheck-logcheck-1.timer
#
# This is a minor nuisance.
# If we create a native unit with the same name as the /etc/cron.d job,
# systemd-cron will automatically skip the /etc/cron.d job.
# This will make the annoying state=degraded go away.
[Unit]
Description=logcheck — email sysadmin about anomalous log events
Documentation=https://salsa.debian.org/debian/logcheck/-/blob/master/debian/logcheck.cron.d
# NOTE: if using systemd-cron, "logcheck.timer exists" will automatically
disable /etc/cron.d/logcheck.
# if using vixie cron, you will need to change /etc/cron.d/logcheck to
either
# 1) not exist; or
# 2) have something like like "[ -d /run/systemd ] ||"
#
# FIXME: Can I express this cron job as *ONE* .timer/.service pair?
#
https://salsa.debian.org/debian/logcheck/-/blob/master/debian/logcheck.cron.d
#The hard part is to supply -R ("reboot mode") iff the unit was started
due to OnBootSec=, and not OnCalendar=.
#For now, I will just accept that reboot jobs lack a "Reboot:" in the
subject line.
# twb: if the command is different, then no. you'll have to
write two services and two timers
#I don't think the -R is very important, so I say: "sorry, you can't
have that feature anymore".
# logcheck if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R;
fi
[Unit]
ConditionPathExists=/usr/sbin/logcheck
[Service]
Type=oneshot
ExecStart=logcheck
Nice=10
User=logcheck
# logcheck sends an "on reboot" email.
# That ought to wait until the mail can reach a remote smarthost.
# As we aren't gated by vixie crond anymore,
# copy the "after network is up" from cron.service?
# TIL cron.service will cheerfully start before the network is up, even
though
# cron mails might just flop around on the floor without a remote
smarthost.
# (Specifically I'm thinking of @reboot jobs and msmtp-mta)
# OK so let's just add a wild-ass sloppy guess.
[Unit]
After=remote-fs.target nss-user-lookup.target network-online.target
# Security hardening.
[Service]
CapabilityBoundingSet=
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=yes
DevicePolicy=closed
IPAddressDeny=any
NoNewPrivileges=yes
PrivateDevices=yes
PrivateMounts=yes
PrivateTmp=yes
PrivateUsers=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=noaccess
ProtectSystem=strict
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
RestrictRealtime=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RemoveIPC=yes
UMask=0077
ProtectHostname=yes
ProcSubset=pid
# Implicitly grants read-write access to /var/lib/logcheck, needed for the
inode+offset stamp files.
StateDirectory=%p
# Use msmtp (not postfix) for sendmail.
# Trick logcheck(8)/mail(1) into using msmtp instead of postfix.
# This is because sendmail(1postfix) requires sgid maildrop.
#
