Hi ML,
currently I am seeing differences between two, what I consider identical, nspawn-containers which prevents me to update one of them. (Lots of) details are at the end of the mail. I set up two (hopefully) identical debian containers in nspawn for a single service (DNS) on a debian host. Today's "apt upgrade" now throws permissions problem on _one_ of the containers (ns4 fails, all others still work - ns3 should be identical but some service data): root@ns4:~# apt upgrade ... 75 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 50.0 MB of archives. After this operation, 313 kB of additional disk space will be used. W: chown to _apt:root of directory /var/cache/apt/archives/partial failed - _ SetupAPTPartialDirectory (1: Operation not permitted) Do you want to continue? [Y/n] Downloading works, but then moving the archives fails: ... E: Failed to fetch http://security.debian.org/pool/updates/main/p/ _ postgresql-9.6/postgresql-9.6_9.6.4-0+deb9u1_amd64.deb rename failed, _ Permission denied (/var/cache/apt/archives/partial/ _ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb -> /var/cache/apt/archives/ _ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb). E: Unable to fetch some archives, maybe run apt-get update or try with -- _ fix-missing? root@ns4:~# I also cannot set the correct container group on the host! (Please see an example at the very end of the mail.) Neither in the HOST, nor in the ns4 journal anything is shown. Following I try to give as much information I consider as relevant as I can. Please do not hesitate to ask for more details. The system is not critical and can be rebooted (which I already did) or whatever. Thanks a lot! == Host root@HOST:~# cat /etc/debian_version 9.1 root@HOST:~# systemd --v systemd 232 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP _ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN root@HOST:~# machinectl list MACHINE CLASS SERVICE OS VERSION ADDRESSES ns3 container systemd-nspawn debian 9 10.225.32.1... ns4 container systemd-nspawn debian 9 10.225.64.1... nsrec2 container systemd-nspawn debian 9 10.225.1.1... 3 machines listed. root@HOST:~# == nspawn container 1 (ns3) == root@ns3:~# cat /etc/debian_version 9.1 root@ns3:~# systemd --v systemd 232 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP _ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN == nspawn container 2 (ns4) == root@ns4:~# cat /etc/debian_version 9.1 root@ns4:~# systemd --v systemd 232 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP _ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN The configuration of both containers look the same to me: == nspawn config root@HOST:~# cat /etc/systemd/nspawn/ns3.nspawn [Exec] # -> guid parse bug in the kernel #PrivateUsers=false [Files] # -> dynamic uid mounts apt w/o root access #Bind=/var/cache/apt/ #Bind=/var/lib/apt/ root@HOST:~# diff /etc/systemd/nspawn/ns3.nspawn /etc/systemd/nspawn/ ns4.nspawn root@HOST:~# == mount config root@HOST:~# cat /etc/systemd/system/var-lib-machines-ns3.mount [Unit] Before=local-fs.target [Install] WantedBy=local-fs.target [Mount] What=/dev/disk/by-label/virt Where=/var/lib/machines/ns3/ Type=btrfs Options=noatime,nodiratime,subvol=vm-ns3_rootfs@active root@HOST:~# cat /etc/systemd/system/var-lib-machines-ns3-var-cache.mount [Unit] Before=local-fs.target [Install] WantedBy=local-fs.target [Mount] What=/dev/disk/by-label/virt Where=/var/lib/machines/ns3//var/cache Type=btrfs Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var- _ cache@active root@HOST:~# root@HOST:~# diff /etc/systemd/system/var-lib-machines-ns3.mount /etc/ _ systemd/system/var-lib-machines-ns4.mount 9c9 < Where=/var/lib/machines/ns3/ --- > Where=/var/lib/machines/ns4/ 11c11 < Options=noatime,nodiratime,subvol=vm-ns3_rootfs@active --- > Options=noatime,nodiratime,subvol=vm-ns4_rootfs@active root@HOST:~# diff /etc/systemd/system/var-lib-machines-ns3-var-cache.mount / _ etc/systemd/system/var-lib-machines-ns4-var-cache.mount 9c9 < Where=/var/lib/machines/ns3//var/cache --- > Where=/var/lib/machines/ns4//var/cache 11c11 < Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var- _ cache@active --- > Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns4_var- _ cache@active root@HOST:~# root@HOST:~# mount | grep 'ns[34].*/cache' /dev/mapper/volg-virt on /var/lib/machines/ns4/var/cache type btrfs _ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=331,subvol=/ _ vm-ns4_var-cache@active) /dev/mapper/volg-virt on /var/lib/machines/ns3/var/cache type btrfs _ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=350,subvol=/ _ vm-ns3_var-cache@active) root@HOST:~# root@HOST:~# btrfs subvolume list /var/lib/btrfs/ | grep _ 'ns[34].*cache' ID 331 gen 68872 top level 5 path vm-ns4_var-cache@active ID 350 gen 67791 top level 5 path vm-ns3_var-cache@active root@HOST:~# == file permissions root@HOST:~# ls -l /var/lib/machines/ns3/ total 4 ... drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 100 Apr 26 12:33 var root@HOST:~# ls -l /var/lib/machines/ns3/var/ total 8 ... drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 120 May 1 20:48 cache ... root@HOST:~# ls -l /var/lib/machines/ns3/var/cache/ total 4 drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 70 May 1 20:47 apt ... root@HOST:~# ls -l /var/lib/machines/ns3/var/cache/apt/ total 1044 drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 22 Aug 30 14:48 archives -rw-r--r-- 1 vu-ns3-0 vg-ns3-0 641725 Apr 29 12:31 pkgcache.bin -rw-r--r-- 1 vu-ns3-0 vg-ns3-0 425316 Apr 29 12:31 srcpkgcache.bin root@HOST:~# ls -l /var/lib/machines/ns3/var/cache/apt/archives/ total 0 -rw-r----- 1 vu-ns3-0 vg-ns3-0 0 May 1 20:47 lock drwx------ 1 vu-ns3-104 vg-ns3-0 0 Aug 30 14:41 partial root@HOST:~# root@HOST:~# ls -l /var/lib/machines/ns4 total 4 ... drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 100 Apr 26 12:33 var root@HOST:~# ls -l /var/lib/machines/ns4/var/ total 8 ... drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 120 Apr 28 22:07 cache ... root@HOST:~# ls -l /var/lib/machines/ns4/var/cache/ total 4 drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 70 Apr 29 12:31 apt ... root@HOST:~# ls -l /var/lib/machines/ns4/var/cache/apt/ total 51920 drwxr-xr-x 1 vu-ns4-0 root 22 Aug 30 14:49 archives -rw-r--r-- 1 vu-ns4-0 root 26581616 Apr 29 12:31 pkgcache.bin -rw-r--r-- 1 vu-ns4-0 root 26581534 Apr 29 12:31 srcpkgcache.bin root@HOST:~# ls -l /var/lib/machines/ns4/var/cache/apt/archives/ total 0 -rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:~# == Problems As you could see the few lines above, the groups in ns4 aren't correct for certain files/directories. But correcting them in the guest as well as the host fails: root@ns4:/var/cache/apt/archives# ls -l total 0 -rw-r----- 1 root root 0 Apr 28 22:04 lock drwx------ 1 _apt nogroup 5000 Aug 30 17:01 partial root@ns4:/var/cache/apt/archives# chgrp root partial/ chgrp: changing group of 'partial/': Operation not permitted root@ns4:/var/cache/apt/archives# root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l total 0 -rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# chgrp vg-ns4-0 _ partial/ root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l total 0 -rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial root@HOST:/var/lib/machines/ns4/var/cache/apt/archives# _______________________________________________ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
