[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
** Changed in: snapd Milestone: None => 2.45 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in snapd: Fix Released Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: Invalid Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
This was fixed in snapd: https://github.com/snapcore/snapd/pull/8426 ** Changed in: snapd Status: Triaged => Fix Released ** Changed in: linux-raspi2 (Ubuntu Bionic) Status: New => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in snapd: Fix Released Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: Invalid Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
I noticed that this same denial happens on UC20 on amd64 VM with kernel 5.4.0-20-generic, so perhaps this is just a new kernel behavior, and the linux-raspi2 kernel has newer patches than all the other UC18 kernels we have available. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in snapd: Triaged Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: New Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
Denial on UC20 VM: Apr 06 13:11:35 ubuntu kernel: audit: type=1400 audit(1586178695.710:59): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common /var-lib- docker/overlay2/ba5b8287c3c94aade8e3254eadbee5d4a4d279b4645e4337af9dd877c97ca1f2-init/diff/etc/resolv.conf" pid=1197 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 kernel info: Linux ubuntu 5.4.0-20-generic #24-Ubuntu SMP Mon Mar 23 20:55:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in snapd: Triaged Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: New Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
Ian proposed a PR with the change suggested by Jamie: https://github.com/snapcore/snapd/pull/8426 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in snapd: Triaged Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: New Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
** Changed in: snapd Importance: Undecided => Medium -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in snapd: Triaged Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: New Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
I can't comment on the interaction of AppArmor and overlay with the
available information. I can say that we already have these rules:
const dockerSupportConnectedPlugAppArmorCore = `
# These accesses are necessary for Ubuntu Core 16 and 18, likely due to the
# version of apparmor or the kernel which doesn't resolve the upper layer of an
# overlayfs mount correctly the accesses show up as runc trying to read from
# /system-data/var/snap/docker/common/var-lib-docker/overlay2/$SHA/diff/
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/{,**/} rwl,
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/{,**/}
rwl,
`
The denial of 'apparmor="DENIED" operation="open"
profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common
/var-lib-
docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv"
pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0
ouid=0' doesn't match this though, because '.dockerenv' is a file, not a
directory. If I were to guess, I'd guess that perhaps the snap is
overlaying a file rather than a dir, but again, I don't know for sure.
It would be fine to adjust the policy to use this instead:
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/{,**} rwl,
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/{,**}
rwl,
since the snap already has read/write access to these directories when
/system-data is not prepended. I've taken a todo to send up a PR for
this.
** Also affects: snapd
Importance: Undecided
Status: New
** Changed in: snapd
Status: New => Triaged
** Changed in: snapd
Assignee: (unassigned) => Jamie Strandboge (jdstrand)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-raspi2 in Ubuntu.
https://bugs.launchpad.net/bugs/1868894
Title:
[uc18] docker overlayfs* seems broken
Status in snapd:
Triaged
Status in linux-raspi2 package in Ubuntu:
Confirmed
Status in linux-raspi2 source package in Bionic:
New
Bug description:
A customer recently reported that 'sudo docker run hello-world' fails
on a pi3 or pi4 running UC18. Looking at the journal, the failure
appears to be caused by an apparmor denial related docker's overlay2
storage driver. I've tried both the unified and the Pi3 specific UC18
images and both fail with the same error. The same command works fine
on other devices running UC18 (I've tested multipass+macOS, and
dragonboard), and also works on a Pi3b running our standard UC16
image.
Here are the details from the UC18 image.
$ snap list
core 16-2.43.3 8691stablecanonical✓
core
core18202001241673stablecanonical✓
base
docker18.09.9 427 stablecanonical✓ -
pi18-127 18-pi canonical✓
gadget
pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓
kernel
snapd 2.43.3 6438stablecanonical✓
snapd
And here's the apparmor denial:
Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ;
USER=root ; COMMAND=/snap/bin/docker run hello-world
Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open"
profile="snap.docker.dockerd"
name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv"
pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
I've been told this may end up being something that gets worked around
in snapd, however as this looks like a regression, I'm erring on the
side of caution and filing this bug anyways.
Please let me know if there's anything else I can provide.
To manage notifications about this bug go to:
https://bugs.launchpad.net/snapd/+bug/1868894/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
** Also affects: linux-raspi2 (Ubuntu Bionic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in linux-raspi2 package in Ubuntu: Confirmed Status in linux-raspi2 source package in Bionic: New Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1868894] Re: [uc18] docker overlayfs* seems broken
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-raspi2 (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-raspi2 in Ubuntu. https://bugs.launchpad.net/bugs/1868894 Title: [uc18] docker overlayfs* seems broken Status in linux-raspi2 package in Ubuntu: Confirmed Bug description: A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image. Here are the details from the UC18 image. $ snap list core 16-2.43.3 8691stablecanonical✓ core core18202001241673stablecanonical✓ base docker18.09.9 427 stablecanonical✓ - pi18-127 18-pi canonical✓ gadget pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel snapd 2.43.3 6438stablecanonical✓ snapd And here's the apparmor denial: Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways. Please let me know if there's anything else I can provide. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1868894/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
