[systemd-devel] Permission/updating problems; different behaviour of two identical nspawn containers
[Sorry for not answering to Lennart's mail directly - it somehow got lost on my side so I have to copy/paste it from the archive.] >> I set up two (hopefully) identical debian containers in nspawn for a single >> service (DNS) on a debian host. Today's "apt upgrade" now throws >> permissions problem on _one_ of the containers (ns4 fails, all others still >> work - ns3 should be identical but some service data): > Most likely something went wrong with the userns UID mapping... Not > sure what though... >> As you could see the few lines above, the groups in ns4 aren't correct for >> certain files/directories. But correcting them in the guest as well as the >> host fails: > Are you suggesting that doing this on the host has no effect at all? > That's seriously strange... Yes, that's the case - at least for the group ownership. And yes, I agree it's strange... > When you ran this, was the container running? Yes, it is running: root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l total 0 -rw-r- 1 vu-ns4-0 vg-ns4-00 Apr 28 22:04 lock drwx-- 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# chgrp vg-ns4-0 _ partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# echo $? 0 root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l total 0 -rw-r- 1 vu-ns4-0 vg-ns4-00 Apr 28 22:04 lock drwx-- 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# machinectl list MACHINE CLASS SERVICEOS VERSION ADDRESSES ns3 container systemd-nspawn debian 9 10.225.32.1... ns4 container systemd-nspawn debian 9 10.225.64.1... nsrec2 container systemd-nspawn debian 9 10.225.1.1... 3 machines listed. root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# Thanks for having a look! ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Permission/updating problems; different behaviour of two identical nspawn containers
[Sorry for not answering to Lennart's answer directly - it somehow got lost so I have to copy/paste it from the archive.] >> I set up two (hopefully) identical debian containers in nspawn for a single >> service (DNS) on a debian host. Today's "apt upgrade" now throws >> permissions problem on _one_ of the containers (ns4 fails, all others still >> work - ns3 should be identical but some service data): > Most likely something went wrong with the userns UID mapping... Not > sure what though... >> As you could see the few lines above, the groups in ns4 aren't correct for >> certain files/directories. But correcting them in the guest as well as the >> host fails: > Are you suggesting that doing this on the host has no effect at all? > That's seriously strange... Yes, that's the case - at least for the group ownership. And yes, I agree it's strange ;-) > When you ran this, was the container running? Yes, it is running: root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l total 0 -rw-r- 1 vu-ns4-0 vg-ns4-00 Apr 28 22:04 lock drwx-- 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# chgrp vg-ns4-0 _ partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# echo $? 0 root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l total 0 -rw-r- 1 vu-ns4-0 vg-ns4-00 Apr 28 22:04 lock drwx-- 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# machinectl list MACHINE CLASS SERVICEOS VERSION ADDRESSES ns3 container systemd-nspawn debian 9 10.225.32.1... ns4 container systemd-nspawn debian 9 10.225.64.1... nsrec2 container systemd-nspawn debian 9 10.225.1.1... 3 machines listed. root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# Thanks for having a look! ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Permission/updating problems; different behaviour of two identical nspawn containers
Hi ML, currently I am seeing differences between two, what I consider identical, nspawn-containers which prevents me to update one of them. (Lots of) details are at the end of the mail. I set up two (hopefully) identical debian containers in nspawn for a single service (DNS) on a debian host. Today's "apt upgrade" now throws permissions problem on _one_ of the containers (ns4 fails, all others still work - ns3 should be identical but some service data): root@ns4:~# apt upgrade ... 75 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 50.0 MB of archives. After this operation, 313 kB of additional disk space will be used. W: chown to _apt:root of directory /var/cache/apt/archives/partial failed - _ SetupAPTPartialDirectory (1: Operation not permitted) Do you want to continue? [Y/n] Downloading works, but then moving the archives fails: ... E: Failed to fetch http://security.debian.org/pool/updates/main/p/ _ postgresql-9.6/postgresql-9.6_9.6.4-0+deb9u1_amd64.deb rename failed, _ Permission denied (/var/cache/apt/archives/partial/ _ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb -> /var/cache/apt/archives/ _ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb). E: Unable to fetch some archives, maybe run apt-get update or try with -- _ fix-missing? root@ns4:~# I also cannot set the correct container group on the host! (Please see an example at the very end of the mail.) Neither in the HOST, nor in the ns4 journal anything is shown. Following I try to give as much information I consider as relevant as I can. Please do not hesitate to ask for more details. The system is not critical and can be rebooted (which I already did) or whatever. Thanks a lot! == Host root@HOST:~# cat /etc/debian_version 9.1 root@HOST:~# systemd --v systemd 232 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP _ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN root@HOST:~# machinectl list MACHINE CLASS SERVICEOS VERSION ADDRESSES ns3 container systemd-nspawn debian 9 10.225.32.1... ns4 container systemd-nspawn debian 9 10.225.64.1... nsrec2 container systemd-nspawn debian 9 10.225.1.1... 3 machines listed. root@HOST:~# == nspawn container 1 (ns3) == root@ns3:~# cat /etc/debian_version 9.1 root@ns3:~# systemd --v systemd 232 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP _ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN == nspawn container 2 (ns4) == root@ns4:~# cat /etc/debian_version 9.1 root@ns4:~# systemd --v systemd 232 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP _ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN The configuration of both containers look the same to me: == nspawn config root@HOST:~# cat /etc/systemd/nspawn/ns3.nspawn [Exec] # -> guid parse bug in the kernel #PrivateUsers=false [Files] # -> dynamic uid mounts apt w/o root access #Bind=/var/cache/apt/ #Bind=/var/lib/apt/ root@HOST:~# diff /etc/systemd/nspawn/ns3.nspawn /etc/systemd/nspawn/ ns4.nspawn root@HOST:~# == mount config root@HOST:~# cat /etc/systemd/system/var-lib-machines-ns3.mount [Unit] Before=local-fs.target [Install] WantedBy=local-fs.target [Mount] What=/dev/disk/by-label/virt Where=/var/lib/machines/ns3/ Type=btrfs Options=noatime,nodiratime,subvol=vm-ns3_rootfs@active root@HOST:~# cat /etc/systemd/system/var-lib-machines-ns3-var-cache.mount [Unit] Before=local-fs.target [Install] WantedBy=local-fs.target [Mount] What=/dev/disk/by-label/virt Where=/var/lib/machines/ns3//var/cache Type=btrfs Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var- _ cache@active root@HOST:~# root@HOST:~# diff /etc/systemd/system/var-lib-machines-ns3.mount /etc/ _ systemd/system/var-lib-machines-ns4.mount 9c9 < Where=/var/lib/machines/ns3/ --- > Where=/var/lib/machines/ns4/ 11c11 < Options=noatime,nodiratime,subvol=vm-ns3_rootfs@active --- > Options=noatime,nodiratime,subvol=vm-ns4_rootfs@active root@HOST:~# diff /etc/systemd/system/var-lib-machines-ns3-var-cache.mount / _ etc/systemd/system/var-lib-machines-ns4-var-cache.mount 9c9 < Where=/var/lib/machines/ns3//var/cache --- > Where=/var/lib/machines/ns4//var/cache 11c11 < Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var- _ cache@active --- > Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns4_var- _ cache@active root@HOST:~# root@HOST:~# mount | grep 'ns[34].*/cache' /dev/mapper/volg-virt on /var/lib/machines/ns4/var/cache type btrfs _ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=331,subvol=/ _ vm-ns4_var-cache@active) /dev/mapper/volg-virt on /var/lib/machines/ns3/var/cache type btrfs _ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=350,subvol=/
Re: [systemd-devel] nspawn: devpts not mounted with PrivateUsers
Dear Lennart & list, thank you for taking your time to answer (and for systemd :-) )! (TL;DR -> SOLVED) On Mittwoch, 19. April 2017 18:37:58 CEST Lennart Poettering wrote: > > == On the host: > > $ groupadd -g3777036288 MY_GROUP > > Don't do this. If you register the group like this, nspawn will > normally abstain from using this group. Use "nss-mymachines" instead > (consider lobbying your distro to turn it on automatically when > nspawn/machined are installed) which will make all private UIDs used > by nspawn (or any other machined registered containers) appear in the > user database (as shown by getent passwd) without any modification of > /etc/passwd or any other file therein. I'm glad that I don't have to do that! It was one of the experiments to get things to to work. Letting systemd do that automatically is much, much better! The nss-mymachines/myhostname/resolve/systemd aren't installed in a Debian minbase - you are right! I figured that out, too, and installed them manually. But I guess this is fair as I explicitly asked for a _minimal_ installation. > > $ echo MY_GROUP:3777036288:65536 >> /etc/subgid > > "subuid" and "subgid" is not used by nspawn. Quite frankly, I am > pretty sure these files are pretty broken concepts... That's very relieving to hear - here the same experiments as above were the culprit. > > systemd-nspawn[6345]: Failed to mount devpts on > > /var/lib/machines/MY_CONTAINER/dev/pts (MS_NOSUID|MS_NOEXEC > > "newinstance,ptmxmode=0666,mode=620,gid=3777036293"): Invalid > > argument > > Because Linux is really really broken, devpts' gid= parameter is > parsed as signed integer, and then cast to an unsigned one. This > effectively prohibits using any UIDs > 2^31-1 for containers, if you > want to use devpts... If you try anyway, you get EINVAL back... In > other words, pick a lower UID base (or let nspawn pick it for you), > and it should work. Wow - how unfortunate. Thank you for pointing that out - even though my naming scheme (first byte host, second byte container) goes down the drain now ;). A short summary for people going through the archive: I meanwhile deleted all sub?id/passwd/group entries, installed all the systemd provided nss packages and activated them in /etc/nsswitch.conf as described in the systemd-manual: passwd: compat mymachines systemd group: compat mymachines systemd hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname PrivateUsers is commented out in the nspawn config file. machined did choose a random UID for a newly started machine: # ls -ld /var/lib/machines/nsstest1 drwxr-xr-x 1 vu-nsstest1-0 vg-nsstest1-0 154 Apr 19 17:14 /var/lib/machines/ nsstest1 Actually, I wasted some time and even rebooted the whole server to be safe because the container user was still named vu-container/vg-container even though I thought I deleted all traces... until I found out machined is giving out the same names as I did before manually :D So, in the end, Lennart's solution (install the systemd provided nss libs and let PrivateUsers alone) seems to be the best (and working ;-) ) one! Thanks again for the help! ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] nspawn: devpts not mounted with PrivateUsers
Hi! On the bug tracker guideline page it said that the systemd-devel-list is also meant for support, so I hope it's okay to ask here this beginnger's question: == Environment: - systemd-232 (systemd-232-22_amd64) - Debian Stretch (minbase + systemd + systemd-container + ...) == Goal: - Run each nspawn-container with a dedicated user id. == Unexpected behaviour: Setting up and running nspawn based containers without any PrivateUsers- setting works. The containers run using a random user-id. (Here I seem to misunderstand the manual as it says "false" is the default setting and therefore no mapping at all should happen.) But as soon as I add a PrivateUsers=true or PrivateUsers=ID setting into the corresponding .nspawn-file, systemd fails while mounting devpts in the container. Since I like the idea of having a dedicated user for each container (and therefore seeing his uid in ps & co on the host), I did the following: == On the host: $ groupadd -g3777036288 MY_GROUP $ echo MY_GROUP:3777036288:65536 >> /etc/subgid $ useradd -d/var/lib/machines/MY_CONTAINER -M -g3777036288 -u3777036288 MY_USER $ echo MY_USER:3777036288:65536 >> /etc/subguid $ chown MY_USER:MY_GROUP /var/lib/machines/MY_CONTAINER $ echo -e "[Exec]\nPrivateUsers=true\n" > /etc/systemd/nspawn/MY_CONTAINER.nspawn -OR- $ echo -e "[Exec]\nPrivateUsers=3777036288\n" > /etc/systemd/nspawn/MY_CONTAINER.nspawn $ machinectl start MY_CONTAINER (The strangely looking ID represents the container in the upper 16 bits so that nspawn can use the lower 16 bits for the local uids. And not putting anything in /etc/sub?id doesn't change anything. But putting the IDs there is the correct way, right?) Journalctl shows the following: == On the host: systemd[1]: Starting Container MY_CONTAINER... systemd-nspawn[6345]: Selected user namespace base 3777036288 and range 65536. systemd-nspawn[6345]: Failed to mount n/a on /var/lib/machines/MY_CONTAINER/sys/fs/selinux (MS_BIND ""): No such file or directory systemd-nspawn[6345]: Failed to mount n/a on /var/lib/machines/MY_CONTAINER/sys/fs/selinux (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND ""): Invalid argument systemd-nspawn[6345]: Failed to mount devpts on /var/lib/machines/MY_CONTAINER/dev/pts (MS_NOSUID|MS_NOEXEC "newinstance,ptmxmode=0666,mode=620,gid=3777036293"): Invalid argument systemd[1]: systemd-nspawn@MY_CONTAINER.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: Failed to start Container MY_CONTAINER. systemd[1]: systemd-nspawn@MY_CONTAINER.service: Unit entered failed state. systemd[1]: systemd-nspawn@MY_CONTAINER.service: Failed with result 'exit-code'. The first two failed mounts (selinux) happen always - it's a minbase installation after all - including successful starts of containers (when not using PrivateUsers settings). But the second one seems to lead to the failed start. Systemd creates the gid 3777036293 for the pts mount, which is +5 from my given uid. And 5 is the group tty (which should be the owner of pts). == On the host: $ grep pts /proc/mounts devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 == On a successfully started container (no PrivateUsers setting, random uid): $ grep pts /proc/mounts devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=32702469,mode=620,ptmxmode=666 0 0 devpts /dev/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 I hope all needed information is included in this mail! Thanks PS: I wonder if this could be connected to https://github.com/systemd/systemd/issues/337 ? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [nspawn] devpts not mounted with PrivateUsers
Hi! On the bug tracker guideline page it said that the systemd-devel-list is also meant for support, so I hope it's okay to ask here this beginnger's question: == Environment: - systemd-232 (systemd-232-22_amd64) - Debian Stretch (minbase + systemd + systemd-container + ...) == Goal: - Run each nspawn-container with a dedicated user id. == Unexpected behaviour: Setting up and running nspawn based containers without any PrivateUsers- setting works. The containers run using a random user-id. (Here I seem to misunderstand the manual as it says "false" is the default setting and therefore no mapping at all should happen.) But as soon as I add a PrivateUsers=true or PrivateUsers=ID setting into the corresponding .nspawn-file, systemd fails while mounting devpts in the container. Since I like the idea of having a dedicated user for each container (and therefore seeing his uid in ps & co on the host), I did the following: == On the host: $ groupadd -g3777036288 MY_GROUP $ echo MY_GROUP:3777036288:65536 >> /etc/subgid $ useradd -d/var/lib/machines/MY_CONTAINER -M -g3777036288 -u3777036288 MY_USER $ echo MY_USER:3777036288:65536 >> /etc/subguid $ chown MY_USER:MY_GROUP /var/lib/machines/MY_CONTAINER $ echo -e "[Exec]\nPrivateUsers=true\n" > /etc/systemd/nspawn/MY_CONTAINER.nspawn -OR- $ echo -e "[Exec]\nPrivateUsers=3777036288\n" > /etc/systemd/nspawn/MY_CONTAINER.nspawn $ machinectl start MY_CONTAINER (The strangely looking ID represents the container in the upper 16 bits so that nspawn can use the lower 16 bits for the local uids. And not putting anything in /etc/sub?id doesn't change anything. But putting the IDs there is the correct way, right?) Journalctl shows the following: == On the host: systemd[1]: Starting Container MY_CONTAINER... systemd-nspawn[6345]: Selected user namespace base 3777036288 and range 65536. systemd-nspawn[6345]: Failed to mount n/a on /var/lib/machines/MY_CONTAINER/sys/fs/selinux (MS_BIND ""): No such file or directory systemd-nspawn[6345]: Failed to mount n/a on /var/lib/machines/MY_CONTAINER/sys/fs/selinux (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND ""): Invalid argument ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
