[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Changed in: cloud-init (Ubuntu) Status: Incomplete => Confirmed ** Changed in: cloud-init Assignee: (unassigned) => Arthit Kosachunhanun (richshops) ** Changed in: apparmor (Ubuntu) Assignee: (unassigned) => Arthit Kosachunhanun (richshops) ** Changed in: cloud-init (Ubuntu) Assignee: (unassigned) => Arthit Kosachunhanun (richshops) ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Arthit Kosachunhanun (richshops) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Changed in: linux (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Changed in: apparmor (Ubuntu) Status: Confirmed => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
Hi all, sorry to necro a thread, but I'm having this issue on MAAS 2.3.1. Is anyone able to provide some insight into this issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
An addition to workaround b.) from #32. We changed the code in question and restarted the maas rackd service which fixed the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
Hi Marzog, What commit has been committed to Linux? I cannot find it. Regards, Daniel -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Changed in: linux (Ubuntu) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Changed in: cloud-init Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
If you are affected by this bug, then you have the following options: a.) upgrade to maas 2.2 MAAS 2.2 sends 'apparmor=0' to the installation/commissionging kernel command line. 2.2 is in -proposed for 16.04, 16.10, 17.04 repositories and is already available in artful. Alternatively you can use the maas ppa (add-apt-repository ppa:maas/stable). Note, however, that updates to that repository are not managed by the same Stable Release Updates policy that is applied to the ubuntu release. b.) cherry pick the commit to your installation. The MAAS commit that put 'a' into place is: https://git.launchpad.net/maas/commit/?id=df9a79b9dba9 It is a one line change. You will need to restart the maas region controller after applying the change. c.) add a global 'kernel_parameter' in maas 2.1.5 with 'apparmor=0'. **WARNING**: this will copy over the kernel parameter to the installed system, thus without further change, installed systems would run without apparmor. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
** Description changed: After update [1] of cloud-init in Ubuntu (which landed in xenial-updates on 2017-06-27), it is causing NTP reload failures. https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f- 0ubuntu1~16.04.1 In MAAS scenarios, this is causing the machine to fail to deploy. + + Related bugs: + * bug 1645644: cloud-init ntp not using expected servers -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
There is a xenial test kernel at http://people.canonical.com/~jj/lp1701297/ I have not had a chance to try it yet. I'll try to get to it in a few hours after some sleep. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
I am also getting the same issue with ga kernel Where we hav to set the hwe kernel? We have to change the hwe kernel to maas node or to the deploying nodes? And apparmor and hwe kernel is mandatory to set to deployment work? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
Well that explains it. So we would have seen this issues from release except for the cloud-init bug. Now we need to isolate the fix and backport it to the ga kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
@Tyler, The reason why this wasn't seen before is that previously in Xenial, cloud-init did not restart 'ntp' with a new config file. Since cloud- init recently SRU'd a fixed cloud-init that does restart 'ntp' on overlay, the issue started to show up. In other words, after a cloud-init bugfix , these issues started surfacing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
To elaborate a bit more, the apparmor and overlayfs incompatibility has been a known kernel issue from before 16.04's release and, at this time, isn't something that is likely to be fixed in 16.04. I'd like to better understand if something changed in userspace that started tickling the incompatibility issue. Did MAAS change how it was using AppArmor in the recent SRU that took it to version 2.1.5? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
@Andres One thing that I'm struggling with is why this bug hasn't been seen before. IIUC, it should be present in the very first ga-16.04 kernel that Ubuntu 16.04 LTS was released with (in addition to earlier kernels while Xenial was a development release). Has MAAS 2.1.x and ga-16.04 kernels just now been used together or could there be some other change (in the kernel, MAAS, or maybe even something else) since the time that Ubuntu 16.04 LTS was released that we're not considering? Would it be possible for someone on your team to test with the ga-16.04 kernel version 4.4.0-21.37 from the xenial-release pocket with both maas 2.0.0~beta3+bzr4941-0ubuntu1 from xenial-release and maas 2.1.5+bzr5596-0ubuntu1~16.04.1 from xenial-updates? I don't know how much effort that is but it would be helpful in understanding what changed in 16.04 to start triggering this bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
John is going to build a test kernel, based on the ga-16.04 kernel, with the binfmt_elf commit cherry-picked from the hwe-16.04. That will let someone from the MAAS team attempt to reproduce the issue with the test kernel and, if the deployment succeeds, it'll tell us that the binfmt_elf commit is causing the change in behavior. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
>From an apparmor pov those 2 kernels are almost identical, with the 4.4 kernel picking up a couple of backport patches, that just do some simple remapping and should not affect behavior. There are however some external changes that could affect apparmor mediation binfmt_elf change (9f834ec18defc369d73ccf9e87a2790bfa05bf46) - that could very well be related to the name="overlay/etc/ld.so.cache" failure there are several changes in overlayfs that may result in permission changes. We can build som test kernels, a 4.4 kernel with the binfmt_elf commit cherry-picked and another one with the newer overlayfs. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
Tyler - thanks for that. John - this is coming up in some internal support team escalations so I'm going to have a look at the kernel changes myself and will let you know if I find anything. I'd be keen to sync up if you have any leads. Regards, Daniel -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
On 07/05/2017 08:14 PM, Daniel Axtens wrote: > Hi Tyler, > > Do you know what the changes between the ga-16.04 and hwe-16.04 kernel > are that make apparmor+overlayfs work? No, we're not currently aware of any code changes that would cause the behavioral change that is reported in the bug. Now that we have the specific kernel version of the HWE kernel, John Johansen can look into possible causes for the change. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
When using the following kernel (the default Xenial kernel, aka ga-16.04 in MAAS), we see this issue: 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux When using the HWE kernel (aka hwe-16.04 in MAAS), we do NOT see this issue: 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
Hi Tyler, Do you know what the changes between the ga-16.04 and hwe-16.04 kernel are that make apparmor+overlayfs work? I'm worried we might hit this problem elsewhere... Regards, Daniel -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
Andres, can you be more specific about the kernel version of the hwe kernel you are seeing this on? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1701297] Re: NTP reload failure (unable to read library) on overlayfs
@Tyler, That's is correct, MAAS 2.2.0+ sends the apparmor=0 for the ephemeral environments. That said, however, this affects else who is not using 2.2 (which in fact, affects customers who are still in 2.1). Also, based on my testing, it seems that when using hwe-16.04 kernel this doesn't happen, but it does with the ga-16.04 kernel. ** Summary changed: - NTP reload failure (causing deployment failures with MAAS) + NTP reload failure (unable to read library) on overlayfs -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1701297 Title: NTP reload failure (unable to read library) on overlayfs To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1701297/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
