Re: [lxc-users] unable to start containers ("Permission denied - Failed to mount")
It just broke for me on two servers again, more or less at the same time: root@backup01 ~ # ls -l /data/lxd total 0 drwx-- 1 root root 198 Jan 24 03:34 containers (...) Both servers are running Ubuntu 18.04 with LXD from snap: lxd 3.99919 stablecanonical✓ - And storage on a btrfs device: root@lxd05 ~ # lxc storage list +-+-++---+-+ | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY | +-+-++---+-+ | default | | btrfs | /data/lxd | 16 | +-+-++---+-+ root@backup01 ~ # lxc storage list +-+-++---+-+ | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY | +-+-++---+-+ | default | | btrfs | /data/lxd | 44 | +-+-++---+-+ Not sure what's causing, but it's yet another time I'm seeing it. Tomasz On 2018-09-24 22:43, Christian Brauner wrote: On Mon, Sep 24, 2018 at 03:40:57PM +0200, Tomasz Chmielewski wrote: Turns out something changed the permissions on "containers" directory: Odd, the new storage snapshot api performs an on-disk upgrade but it shouldn't touch the containers directory... //cc Stéphane Christian # lxc storage list +-+-++---+-+ | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY | +-+-++---+-+ | default | | btrfs | /data/lxd | 12 | +-+-++---+-+ # ls -l /data/lxd total 0 drwxr-xr-x 1 root root 90 Sep 24 13:05 archives drwx-- 1 root root 518 Sep 24 13:12 containers <- here drwx--x--x 1 root root 0 Mar 28 16:14 custom drwx-- 1 root root 0 Sep 21 06:05 images drwx-- 1 root root 0 Sep 24 05:48 snapshots This fixed it: chmod 711 /data/lxd/containers/ I'm 99% sure we did not change the permissions on that directory... Tomasz On 2018-09-24 15:32, Tomasz Chmielewski wrote: > I'm not able to start any container today. > > # lxc start preprod-app > Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app > /var/snap/lxd/common/lxd/containers > /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf: > Try `lxc info --show-log preprod-app` for more info > > > # lxc info --show-log preprod-app > Name: preprod-app > Remote: unix:// > Architecture: x86_64 > Created: 2018/09/05 15:01 UTC > Status: Stopped > Type: persistent > Profiles: default > > Log: > > lxc preprod-app 20180924132438.883 WARN conf - > conf.c:lxc_map_ids:2917 - newuidmap binary is missing > lxc preprod-app 20180924132438.883 WARN conf - > conf.c:lxc_map_ids:2923 - newgidmap binary is missing > lxc preprod-app 20180924132438.887 WARN conf - > conf.c:lxc_map_ids:2917 - newuidmap binary is missing > lxc preprod-app 20180924132438.887 WARN conf - > conf.c:lxc_map_ids:2923 - newgidmap binary is missing > lxc preprod-app 20180924132438.917 ERRORdir - > storage/dir.c:dir_mount:195 - Permission denied - Failed to mount > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on > "/var/snap/lxd/common/lxc/" > lxc preprod-app 20180924132438.917 ERRORconf - > conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto > "/var/snap/lxd/common/lxc/" with options "(null)" > lxc preprod-app 20180924132438.917 ERRORconf - > conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for > lxc preprod-app 20180924132438.917 ERRORconf - > conf.c:lxc_setup:3510 - Failed to setup rootfs > lxc preprod-app 20180924132438.917 ERRORstart - > start.c:do_start:1234 - Failed to setup container "preprod-app" > lxc preprod-app 20180924132438.918 ERRORsync - > sync.c:__sync_wait:59 - An error occurred in another process (expected > sequence number 5) > lxc preprod-app 20180924132439.235 ERRORstart - > start.c:__lxc_start:1910 - Failed to spawn container "preprod-app" > lxc preprod-app 20180924132439.235 ERRORlxccontainer - > lxccontainer.c:wait_on_daemonized_start:840 - Received container state > "ABORTING" instead of "RUNNING" > lxc preprod-app 20180924132439.963 WARN conf - > conf.c:lxc_map_ids:2917 - newuidmap binary is missing > lxc preprod-app 20180924132439.101 WARN conf - > conf.c:lxc_map_ids:2923 - newgidmap binary is missing > lxc 20180924132439.380 WARN commands - > commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to > receive response for command "get_state" > > > # snap list > Name Version Rev Tracking Publisher Notes > core 16-2.35 5328 stablecanonical✓ core > lxd 3.5 8774 stablecanonical✓ - > > > This is on Ubuntu 18.04. ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org
Re: [lxc-users] unable to start containers ("Permission denied - Failed to mount")
On Mon, Sep 24, 2018 at 03:40:57PM +0200, Tomasz Chmielewski wrote: > Turns out something changed the permissions on "containers" directory: Odd, the new storage snapshot api performs an on-disk upgrade but it shouldn't touch the containers directory... //cc Stéphane Christian > > > # lxc storage list > +-+-++---+-+ > | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY | > +-+-++---+-+ > | default | | btrfs | /data/lxd | 12 | > +-+-++---+-+ > > > # ls -l /data/lxd > total 0 > drwxr-xr-x 1 root root 90 Sep 24 13:05 archives > drwx-- 1 root root 518 Sep 24 13:12 containers <- here > drwx--x--x 1 root root 0 Mar 28 16:14 custom > drwx-- 1 root root 0 Sep 21 06:05 images > drwx-- 1 root root 0 Sep 24 05:48 snapshots > > > This fixed it: > > chmod 711 /data/lxd/containers/ > > > I'm 99% sure we did not change the permissions on that directory... > > > Tomasz > > > > On 2018-09-24 15:32, Tomasz Chmielewski wrote: > > I'm not able to start any container today. > > > > # lxc start preprod-app > > Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app > > /var/snap/lxd/common/lxd/containers > > /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf: > > Try `lxc info --show-log preprod-app` for more info > > > > > > # lxc info --show-log preprod-app > > Name: preprod-app > > Remote: unix:// > > Architecture: x86_64 > > Created: 2018/09/05 15:01 UTC > > Status: Stopped > > Type: persistent > > Profiles: default > > > > Log: > > > > lxc preprod-app 20180924132438.883 WARN conf - > > conf.c:lxc_map_ids:2917 - newuidmap binary is missing > > lxc preprod-app 20180924132438.883 WARN conf - > > conf.c:lxc_map_ids:2923 - newgidmap binary is missing > > lxc preprod-app 20180924132438.887 WARN conf - > > conf.c:lxc_map_ids:2917 - newuidmap binary is missing > > lxc preprod-app 20180924132438.887 WARN conf - > > conf.c:lxc_map_ids:2923 - newgidmap binary is missing > > lxc preprod-app 20180924132438.917 ERRORdir - > > storage/dir.c:dir_mount:195 - Permission denied - Failed to mount > > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on > > "/var/snap/lxd/common/lxc/" > > lxc preprod-app 20180924132438.917 ERRORconf - > > conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs > > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto > > "/var/snap/lxd/common/lxc/" with options "(null)" > > lxc preprod-app 20180924132438.917 ERRORconf - > > conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for > > lxc preprod-app 20180924132438.917 ERRORconf - > > conf.c:lxc_setup:3510 - Failed to setup rootfs > > lxc preprod-app 20180924132438.917 ERRORstart - > > start.c:do_start:1234 - Failed to setup container "preprod-app" > > lxc preprod-app 20180924132438.918 ERRORsync - > > sync.c:__sync_wait:59 - An error occurred in another process (expected > > sequence number 5) > > lxc preprod-app 20180924132439.235 ERRORstart - > > start.c:__lxc_start:1910 - Failed to spawn container "preprod-app" > > lxc preprod-app 20180924132439.235 ERRORlxccontainer - > > lxccontainer.c:wait_on_daemonized_start:840 - Received container state > > "ABORTING" instead of "RUNNING" > > lxc preprod-app 20180924132439.963 WARN conf - > > conf.c:lxc_map_ids:2917 - newuidmap binary is missing > > lxc preprod-app 20180924132439.101 WARN conf - > > conf.c:lxc_map_ids:2923 - newgidmap binary is missing > > lxc 20180924132439.380 WARN commands - > > commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to > > receive response for command "get_state" > > > > > > # snap list > > Name Version Rev Tracking Publisher Notes > > core 16-2.35 5328 stablecanonical✓ core > > lxd 3.5 8774 stablecanonical✓ - > > > > > > This is on Ubuntu 18.04. > > > > > > Tomasz Chmielewski > > https://lxadm.com > > ___ > > lxc-users mailing list > > lxc-users@lists.linuxcontainers.org > > http://lists.linuxcontainers.org/listinfo/lxc-users > ___ > lxc-users mailing list > lxc-users@lists.linuxcontainers.org > http://lists.linuxcontainers.org/listinfo/lxc-users ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] unable to start containers ("Permission denied - Failed to mount")
Turns out something changed the permissions on "containers" directory: # lxc storage list +-+-++---+-+ | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY | +-+-++---+-+ | default | | btrfs | /data/lxd | 12 | +-+-++---+-+ # ls -l /data/lxd total 0 drwxr-xr-x 1 root root 90 Sep 24 13:05 archives drwx-- 1 root root 518 Sep 24 13:12 containers <- here drwx--x--x 1 root root 0 Mar 28 16:14 custom drwx-- 1 root root 0 Sep 21 06:05 images drwx-- 1 root root 0 Sep 24 05:48 snapshots This fixed it: chmod 711 /data/lxd/containers/ I'm 99% sure we did not change the permissions on that directory... Tomasz On 2018-09-24 15:32, Tomasz Chmielewski wrote: I'm not able to start any container today. # lxc start preprod-app Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf: Try `lxc info --show-log preprod-app` for more info # lxc info --show-log preprod-app Name: preprod-app Remote: unix:// Architecture: x86_64 Created: 2018/09/05 15:01 UTC Status: Stopped Type: persistent Profiles: default Log: lxc preprod-app 20180924132438.883 WARN conf - conf.c:lxc_map_ids:2917 - newuidmap binary is missing lxc preprod-app 20180924132438.883 WARN conf - conf.c:lxc_map_ids:2923 - newgidmap binary is missing lxc preprod-app 20180924132438.887 WARN conf - conf.c:lxc_map_ids:2917 - newuidmap binary is missing lxc preprod-app 20180924132438.887 WARN conf - conf.c:lxc_map_ids:2923 - newgidmap binary is missing lxc preprod-app 20180924132438.917 ERRORdir - storage/dir.c:dir_mount:195 - Permission denied - Failed to mount "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on "/var/snap/lxd/common/lxc/" lxc preprod-app 20180924132438.917 ERRORconf - conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)" lxc preprod-app 20180924132438.917 ERRORconf - conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for lxc preprod-app 20180924132438.917 ERRORconf - conf.c:lxc_setup:3510 - Failed to setup rootfs lxc preprod-app 20180924132438.917 ERRORstart - start.c:do_start:1234 - Failed to setup container "preprod-app" lxc preprod-app 20180924132438.918 ERRORsync - sync.c:__sync_wait:59 - An error occurred in another process (expected sequence number 5) lxc preprod-app 20180924132439.235 ERRORstart - start.c:__lxc_start:1910 - Failed to spawn container "preprod-app" lxc preprod-app 20180924132439.235 ERRORlxccontainer - lxccontainer.c:wait_on_daemonized_start:840 - Received container state "ABORTING" instead of "RUNNING" lxc preprod-app 20180924132439.963 WARN conf - conf.c:lxc_map_ids:2917 - newuidmap binary is missing lxc preprod-app 20180924132439.101 WARN conf - conf.c:lxc_map_ids:2923 - newgidmap binary is missing lxc 20180924132439.380 WARN commands - commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to receive response for command "get_state" # snap list Name Version Rev Tracking Publisher Notes core 16-2.35 5328 stablecanonical✓ core lxd 3.5 8774 stablecanonical✓ - This is on Ubuntu 18.04. Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
[lxc-users] unable to start containers ("Permission denied - Failed to mount")
I'm not able to start any container today. # lxc start preprod-app Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf: Try `lxc info --show-log preprod-app` for more info # lxc info --show-log preprod-app Name: preprod-app Remote: unix:// Architecture: x86_64 Created: 2018/09/05 15:01 UTC Status: Stopped Type: persistent Profiles: default Log: lxc preprod-app 20180924132438.883 WARN conf - conf.c:lxc_map_ids:2917 - newuidmap binary is missing lxc preprod-app 20180924132438.883 WARN conf - conf.c:lxc_map_ids:2923 - newgidmap binary is missing lxc preprod-app 20180924132438.887 WARN conf - conf.c:lxc_map_ids:2917 - newuidmap binary is missing lxc preprod-app 20180924132438.887 WARN conf - conf.c:lxc_map_ids:2923 - newgidmap binary is missing lxc preprod-app 20180924132438.917 ERRORdir - storage/dir.c:dir_mount:195 - Permission denied - Failed to mount "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on "/var/snap/lxd/common/lxc/" lxc preprod-app 20180924132438.917 ERRORconf - conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)" lxc preprod-app 20180924132438.917 ERRORconf - conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for lxc preprod-app 20180924132438.917 ERRORconf - conf.c:lxc_setup:3510 - Failed to setup rootfs lxc preprod-app 20180924132438.917 ERRORstart - start.c:do_start:1234 - Failed to setup container "preprod-app" lxc preprod-app 20180924132438.918 ERRORsync - sync.c:__sync_wait:59 - An error occurred in another process (expected sequence number 5) lxc preprod-app 20180924132439.235 ERRORstart - start.c:__lxc_start:1910 - Failed to spawn container "preprod-app" lxc preprod-app 20180924132439.235 ERRORlxccontainer - lxccontainer.c:wait_on_daemonized_start:840 - Received container state "ABORTING" instead of "RUNNING" lxc preprod-app 20180924132439.963 WARN conf - conf.c:lxc_map_ids:2917 - newuidmap binary is missing lxc preprod-app 20180924132439.101 WARN conf - conf.c:lxc_map_ids:2923 - newgidmap binary is missing lxc 20180924132439.380 WARN commands - commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to receive response for command "get_state" # snap list Name Version Rev Tracking Publisher Notes core 16-2.35 5328 stablecanonical✓ core lxd 3.5 8774 stablecanonical✓ - This is on Ubuntu 18.04. Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
