On Wed, Dec 13, 2023 at 10:36 AM Christopher Wong <christopher.w...@axis.com> wrote:
> Hi Mantas, > > > > I tried with StopWhenUnneeded=no in user-runtime-dir@.service, then when > user@1001.service fails the status of user-runtime-dir@.service is > active. At this state the directory /run/user/1001 is created, it is empty, > owned by root. Running the mount command doesn’t show /run/user/1001. > Run the "/usr/lib/systemd/systemd-user-runtime-dir start 1001" manually and check whether the mounted filesystem is there afterwards. If it's still not there, then run "mount -t tmpfs -o uid=1001,mode=0700 none /run/user/1001" and then check whether it stays mounted. > > > I have mentioned it before, but I want to point out that if I put > “ExecStartPre=+chown > %i /run/user/%i” in user@.service then the user@1001.service can be > started manually. The mount command doesn’t show /run/user/1001 either, but > since the service is started the path contains bus socket and systemd > directory with content, which are the things I am after. > > > > The main issue here is that /run/user/1001 is owned by root after > user-runtime-dir@.service has been exited successfully. > No, that's only a symptom of the main issue. The current design that systemd implements is to have a user-specific tmpfs mounted at that location (for quota purposes), and so the underlying mountpoint is deliberately created as owned by root – its ownership is not changed because it's supposed to have a new filesystem mounted on top (which would make the mountpoint hidden and its ownership moot). If you specifically want to *not* have an additional tmpfs there, then you can continue using the manual "ExecStartPre=chown" (or in fact you could replace the entire user-runtime-dir@ with a simpler one that only mkdirs and chowns), but in that case you shouldn't be saying that it's a systemd issue that it doesn't chown something that it was never meant to chown to begin with. > > > Best regards, > > Christopher Wong > > > > > > *From: *Mantas Mikulėnas <graw...@gmail.com> > *Date: *Wednesday, 13 December 2023 at 08:08 > *To: *Christopher Wong <christopher.w...@axis.com> > *Cc: *Systemd <systemd-devel@lists.freedesktop.org> > *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed > with permission denied > > On Tue, Dec 12, 2023 at 6:15 PM Christopher Wong < > christopher.w...@axis.com> wrote: > > Hi Mantas, > > > > After user@1001.service failed, it trigger the stopping process and > become inactive. > > > > Ah yeah, that makes sense, user-runtime-dir@ has StopWhenUnneeded=yes – > so of course after user@1001 crashes you're not going to see anything > mounted anymore. > > > > Could you try temporarily removing that option / setting it to 'no', just > to see what changes? > > > > > > ○ user-runtime-dir@1001.service - User Runtime Directory /run/user/1001 > > Loaded: loaded (/etc/systemd/system/user-runtime-dir@.service; > static) > > Drop-In: /usr/lib/systemd/system/service.d > > └─10-axis.conf, 20-axis-sandbox.conf > > Active: inactive (dead) since Tue 2023-12-12 16:33:35 CET; 36min ago > > Duration: 315ms > > Docs: man:user@.service(5) > > Process: 16325 ExecStartPre=ls -la /run/user (code=exited, > status=0/SUCCESS) > > Process: 16327 ExecStartPre=mount (code=exited, status=0/SUCCESS) > > Process: 16329 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir > start 1001 (code=exited, status=0/SUCCESS) > > Process: 16334 ExecStartPost=sleep 5 (code=exited, status=0/SUCCESS) > > Process: 16347 ExecStartPost=ls -la /run/user/1001 (code=exited, > status=0/SUCCESS) > > Process: 16351 ExecStartPost=mount (code=exited, status=0/SUCCESS) > > Process: 16361 ExecStop=/usr/lib/systemd/systemd-user-runtime-dir > stop 1001 (code=exited, status=0/SUCCESS) > > Main PID: 16329 (code=exited, status=0/SUCCESS) > > CPU: 48ms > > > > /etc/fstab don’t include anything on /run/user/1001 and there is no mount > unit for run-user-1001.mount either. > > > > Best regards, > > Christopher Wong > > > > > > *From: *Mantas Mikulėnas <graw...@gmail.com> > *Date: *Tuesday, 12 December 2023 at 17:05 > *To: *Christopher Wong <christopher.w...@axis.com> > *Cc: *Systemd <systemd-devel@lists.freedesktop.org> > *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed > with permission denied > > That sounds like it's getting immediately unmounted (or maybe not being > mounted at all despite the program doing so). > > > > Does the user-runtime-dir service continue to show as "active" after this, > or does it return to "inactive"? > > > > Does your /etc/fstab have any mentions of /run/user/1001? Or more > generally, are there any run-user-1001.mount units? (If you 'systemctl > status' this unit, does the status include a source path?) > > > > On Tue, Dec 12, 2023, 17:34 Christopher Wong <christopher.w...@axis.com> > wrote: > > Hi Mantas, > > > > I currently have the following flow: > > > > 1. No /run/user/1001 directory > 2. systemctl start user@1001.service > 3. systemd start user-runtime-dir@1001.service which ends successfully. > 4. The directory /run/user/1001 exists now, but is empty, owned by > root with mode 0700 > 5. I don’t have findmnt on my system, so I used mount, but > /run/user/1001 is not listed. > 6. systemd start user@1001.service which fails due to permission > denied. > > > > I can’t explain why the /run/user/1001 is owned by root after > user-runtime-dir@1001.service successfully exited. I added some personal > print in systemd code to ensure that the mount command returned success > (r=0). Although, the mount was successful the command “mount” didn’t list > it. In the list of mounts starting with /run I could only find these > entries: > > > > Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs > (rw,nosuid,nodev,mode=755) > > Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs > (ro,nosuid,nodev,noexec,mode=755) > > Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type > tmpfs (ro,nosuid,nodev,mode=755) > > > > If I do a chown of the directory in user@1001.service then it works > > > > root@host:/run/user# ls -la 1001 > > drwx------ 3 ida root 80 Dec 12 16:19 . > > drwxr-xr-x 3 root root 60 Dec 12 16:19 .. > > srw-rw-rw- 1 ida ssh-user 0 Dec 12 16:19 bus > > drwxr-xr-x 5 ida ssh-user 140 Dec 12 16:19 systemd > > > > The ”mount” command don’t list /run/user/1001 for the successful case > either. > > > > Best regards, > > Christopher Wong > > > > > > *From: *Mantas Mikulėnas <graw...@gmail.com> > *Date: *Monday, 11 December 2023 at 17:56 > *To: *Christopher Wong <christopher.w...@axis.com> > *Cc: *Systemd <systemd-devel@lists.freedesktop.org> > *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed > with permission denied > > On Mon, Dec 11, 2023, 17:28 Christopher Wong <christopher.w...@axis.com> > wrote: > > Hi Mantas, > > > > I have added ExecStartPre to user@.service to run “id” and “ls -la”: > > > > Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount > /run/user/1001 owned by 1001:118 > > Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs > (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV > "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")... > > Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory > /run/user/1001. > > Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001... > > Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) > groups=118(ssh-users),236(systemd-journal) > > Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 3 root root > 60 Dec 11 15:50 . > > Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 98 root root > 2120 Dec 11 15:30 .. > > Dec 11 15:50:34 host ls[40293]: drwx------ 2 root root > 40 Dec 11 15:50 1001 > > Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in > user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK > +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 > -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT > -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP > -SYSVINIT default-hierarchy=unified) > > > > The /run/user/1001 belongs to root with mode 0700. Should this belong to > root? > > No, it should be owned by UID 1001 (though still mode 0700). > > Is it because I manually start user@1001.service as root? > > Which user started the .service is usually not important, all services get > a "fresh" environment that's fully described by the unit file. > > > > So even if you did 'systemctl start' as root, the unit has User=%i so the > instance parameter tells it which UID to run as, so will be running as UID > 1001. Likewise user-runtime-dir@1001 will get the UID for the mount from > its instance name (you can see that the "Mounting tmpfs" message has the > correct information). > > However, after user-runtime-dir@1001.service has finished it startup, > the user@1001.service is started as uid=1001 and therefore can’t create > any directories under /run/user/1001. Resulting in user@1001.service > failed to start. > > > > If I add “ExecStartPre=+chown %i /run/user/%i” to user@.service then it > works! But I am unsure if this is really the way fix this. > > > > So far, it sounds like the directory is being created *by something else* > before user-runtime-dir@ is even invoked. > > > > Try adding the same "-/bin/ls -lad /run/user/%i" as both ExecStartPre and > ExecStartPost of user-runtime-dir@ (and maybe even a findmnt). If the > directory already exists during ExecStartPre, start looking for other > services or cronjobs, or tmpfiles.d configs, or 'su' invocations, which may > cause it to be created. > > > > There might also be something that chowns it to root *after* it was > created correctly. If you actually see the tmpfs mount in 'findmnt' or in > 'mount', but it's owned by root despite having uid=1001 in its mount > options, something has chowned it...or your tmpfs feature is broken. > > > > If you don't see it in findmnt at all, even after user-runtime-dir has > succeeded – either the mount failed quietly, or... something (like systemd > itself) has quietly unmounted it. > > > > > > Regarding the testing, I have done both restart of everything and manual, > but the result is the same. Now that I have the > “Environment=XDG_RUNTIME_DIR=/run/user/%i” I no longer need to do > “systemctl set-environment …” > > > > Thank you for taking your time! > > > > Best regards, > > Christopher Wong > > > > > > *From: *Mantas Mikulėnas <graw...@gmail.com> > *Date: *Friday, 8 December 2023 at 21:53 > *To: *Christopher Wong <christopher.w...@axis.com> > *Cc: *Systemd <systemd-devel@lists.freedesktop.org> > *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed > with permission denied > > On Fri, Dec 8, 2023 at 6:53 PM Christopher Wong <christopher.w...@axis.com> > wrote: > > Hi Mantas, > > > > I have from your suggestion done the following: > > > > Putting the below in user@.service > > > > [Service] > > ... > > Environment=XDG_RUNTIME_DIR=/run/user/%i > > Environment=SYSTEMD_LOG_LEVEL=debug > > > > Putting the below in user-runtime-dir@.service > > > > [Service] > > ... > > Environment=SYSTEMD_LOG_LEVEL=debug > > > > Then I have disabled the global set-log-level debug (if this is also > required, please let me know). > > > > Unlike set-environment that's not global, it only affects pid1. > > > > > > What I can see from the logs is that user-runtime-dir@1001.service seems > to be started and mount /run/user/1001, but addition creation of directory > under this mount is getting permission denied. > > > > Dec 08 17:33:29 host systemd[1]: Created slice User Slice of UID 1001. > > Dec 08 17:33:29 host systemd[1]: Starting User Runtime Directory > /run/user/1001... > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing > state UNSET -> OPENING > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: sd-bus: starting bus > by connecting to /run/dbus/system_bus_socket... > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing > state OPENING -> AUTHENTICATING > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing > state AUTHENTICATING -> HELLO > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message > type=method_call sender=n/a destination=org.freedesktop.DBus > path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello > cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message > type=method_return sender=org.freedesktop.DBus destination=:1.2536 path=n/a > interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s > error-name=n/a error-message=n/a > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing > state HELLO -> RUNNING > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message > type=method_call sender=n/a destination=org.freedesktop.login1 > path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties > member=Get cookie=2 reply_cookie=0 signature=ss error-name=n/a > error-message=n/a > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message > type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a > member=n/a cookie=15 reply_cookie=2 signature=v error-name=n/a > error-message=n/a > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message > type=method_call sender=n/a destination=org.freedesktop.login1 > path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties > member=Get cookie=3 reply_cookie=0 signature=ss error-name=n/a > error-message=n/a > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message > type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a > member=n/a cookie=16 reply_cookie=3 signature=v error-name=n/a > error-message=n/a > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing > state RUNNING -> CLOSED > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Will mount > /run/user/1001 owned by 1001:118 > > Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Mounting tmpfs > (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV > "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")... > > Dec 08 17:33:29 host systemd[1]: Finished User Runtime Directory > /run/user/1001. > > Dec 08 17:33:29 host systemd[1]: Starting User Manager for UID 1001... > > Dec 08 17:33:29 host systemd[36280]: systemd 254.7-2-g9edc143 running in > user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK > +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 > -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT > -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP > -SYSVINIT default-hierarchy=unified) > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible', ignoring: Permission denied > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible/reg', ignoring: Permission denied > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible/dir', ignoring: Permission denied > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible/fifo', ignoring: Permission denied > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible/sock', ignoring: Permission denied > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible/chr', ignoring: Permission denied > > Dec 08 17:33:29 host systemd[36280]: Failed to create > '/run/user/1001/systemd/inaccessible/blk', ignoring: Permission denied > > > > What's the ownership of /run/user/1001 and /run/user/1001/systemd after > all of this? > > > > Are you rebooting between tests or just manually starting it? > > > > My current guess is that due to the earlier `systemctl set-environment`, > some *other* thing that's running as root inherited the /run/user/1001 path > and created root-owned directories there? That's the issue with setting > global environment, it needs to be unset afterwards... > > > -- > > Mantas Mikulėnas > > > > -- > > Mantas Mikulėnas > -- Mantas Mikulėnas