[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

This bug was fixed in the package lxd - 2.0.10-0ubuntu1~16.04.2 --- lxd (2.0.10-0ubuntu1~16.04.2) xenial; urgency=medium * Fix regression in image update logic (LP: #1712455): - 0005-Fix-regression-in-image-auto-update-logic.patch -

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

I confirmed that the sysctl.d file is now present and functional. ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Hello Christian, or anyone else affected, Accepted lxd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxd/2.0.10-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Also affects: lxd (Ubuntu Xenial) Importance: Undecided Status: New ** Description changed: - Reported by Uros Jovanovic here: https://bugs.launchpad.net/juju- - core/+bug/1593828/comments/18 + == SRU + === Rationale + LXD containers using systemd will use a very large amount of

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** No longer affects: juju -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many open files" To manage

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Milestone: 2.2-beta4 => 2.2-rc1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Milestone: 2.2-beta3 => 2.2-beta4 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

This bug was fixed in the package lxd - 2.12-0ubuntu2 --- lxd (2.12-0ubuntu2) zesty; urgency=medium * Increate default inotify limits to 1024 instances per uid. (LP: #1602192) -- Stéphane Graber Mon, 03 Apr 2017 18:51:34 -0400 ** Changed in: lxd

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Milestone: 2.2-beta2 => 2.2-beta3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Milestone: 2.2-beta1 => 2.2-beta2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

>From internal discussion, for Juju, in clouds, there is no guarantee that the images can support particular kernel parameters. For us, it is best not to try and guess and better surface the error in the status. ** Changed in: juju Importance: Critical => High ** Changed in: juju

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Milestone: 2.1.0 => 2.1-rc1 ** Changed in: juju Assignee: Richard Harding (rharding) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Status: Confirmed => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

billy-olsen: You're right - because we don't install juju from the package on the deployed machines the config to enable more lxd containers doesn't get done there. We should apply it in that case. ** Changed in: juju Status: Fix Committed => Confirmed -- You received this bug

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

This bug is fixed for the local LXD provider scenario for 2.0 on Xenial with this commit - http://bazaar.launchpad.net/~juju- qa/ubuntu/xenial/juju/2.0.0/revision/214 - in which the juju-2.0.conf file has the settings identified and dropped into /usr/lib/sysctl.d. Unfortunately, this only

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Can we see the commit related to the bug? Also, is there a plan to backport it to 2.0 series as I'm not sure when we get 2.1 GA? https://github.com/juju/juju/wiki/Juju-Release-Schedule -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Tested w Juju 2.1 beta3 and verified I can get over 20 containers and while it's slow does not get stuck in pending. The changes to the profiles on the server that Michael has done appear to be working. ** Changed in: juju Status: In Progress => Fix Committed -- You received this bug

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Not really. The problem is that bumping those sysctls to the value in that document will work on most normal systems but will fail dramatically on very low memory systems like some cloud instances and ARM systems. As for having LXD know when things will fail, it simply has not idea. All the

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Would it be reasonable to apply the settings that https://github.com/lxc/lxd/blob/master/doc/production-setup.md suggests when we add the first LXD to a machine? What about interrogating the kernel about available resources and refusing to add a container when it won't work? I can try adding 20

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

$ juju bootstrap foo lxd Creating Juju controller "foo" on lxd/localhost Looking for packaged Juju agent version 2.0.0 for amd64 No packaged binary found, preparing local Juju agent binary To configure your system to better support LXD containers, please see:

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

I have filed bug #1631914 against lxd for better surfacing of the "too many files open" error. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

As a note, increasing fs.max-files had no effect for me. Using the following settings I was able to deploy about 23 containers with juju, which is higher than I could previously: fs.inotify.max_user_watches = 524288 fs.inotify.max_user_instances = 256 These are the proposed settings to

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

FYI a new version of the inotify patchset was sent today for review: https://lists.linuxfoundation.org/pipermail/containers/2016-October/037480.html This approach is the real fix for the inotify part of this problem. For open files, we've had a few reports of systemd occasionally misbehaving and

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

with fs.inotify.max_user_instances = 2048 it didn't fail until the 187th container. It did fail with a new message that I haven't seen: [3831] x-187:error: Error calling 'lxd forkstart x-187 /var/lib/lxd/containers /var/log/lxd/x-187/lxc.conf': err='exit status 1' lxc 20161007100331.798

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

I added a "cat /proc/meminfo | grep Slab" to go-lxc-run.sh and found this: $ sysctl fs.inotify fs.inotify.max_queued_events = 65536 fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 524288 $ ulimit -a ... open files (-n) 1048576 ... $ go-lxc-run.sh [0]

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

After rebooting with the new values, I did manage to get to launch a lot more containers, failing at only the 232nd one. I wonder if the issue is not the User number of open files, but the Root number of open files, which requires a reboot to get updated. I'll try to play around more to really

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Interestingly, my baseline Kernel memory with no containers (and not much other software) was about the same (~400MB). I'm not entirely sure why it grew faster with the new settings, but didn't effect the baseline. -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

I did try setting all of the items that are mentioned in production- setup.md. To start with, a few of them are not reasonable. max_user_instances defaults to 128, and we were able to see a difference at 256, but not at 1024. Setting it to 1M seems silly. I'll also note that my Kernel memory

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Created bug #1631038 for the addition of an /etc/sysctl.d/10-juju.conf file upping max_user_watches and max_user_instances which will increase the number of containers we can run out of the box, but not by enough. -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Note that there should be support for /etc/system/limits.d/10-juju.conf I'm testing it now, but it may be that we can drop something in there as well. I'll test it a bit, but if we have some tasteful defaults, maybe we can make it work. I think we can change from their default so instead of "*

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Chatting with tych0 we were linked to: https://github.com/lxc/lxd/blob/master/doc/production-setup.md That notes a bunch of tweaks to run the system at scale. We need to evaluate what items make sense to enable ootb, and what things are solid for a prod lxd system but not a great idea for all

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Some settings to tweak from tych0 on IRC: https://github.com/lxc/lxd/blob/master/doc/production-setup.md -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Note from jam after some experimentation: So having played with it for a bit, I'm more comfortable with an /etc/sysctl.d/10-juju.conf that sets max_user_watches=512k and max_user_instances=256 but if we want to get to 50 instances we need to dig harder. I can just barely get to 10 instances of

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

With Juju in the loop, I run into whatever limit a bit faster. I was successful at doing: juju bootstrap test-lxd lxd juju deploy ubuntu juju add-unit -n 5 ubuntu # wait for status to say everything is running juju add-unit -n 5 ubuntu # wait for status to be happy but then after doing one

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

5) Another data point, with $ sysctl fs.inotify fs.inotify.max_queued_events = 131072 fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 524288 (so max_queued_events 8x greater, and max_user_instances well above previously established useful level), I still only get 19 containers.

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Michael and I played around with some different settings, and here are my notes. 1) Package kde-runtime seems to install /etc/sysctl.d/30-baloo-inotify-limits.conf which sets max_user_watches to 512*1024 'slabtop' says that my baseline kernel memory is 380-420MB with no containers

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

We can include this in cloud-init for machines that will host containers. However, for the lxd provider this would need to be set on the machine running juju (the host machine). Should this be done at juju install time? -- You received this bug notification because you are a member of Ubuntu

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju Milestone: 2.0.0 => 2.1.0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many open

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Per email thread, Juju needs to ship with expanded details: > Stephane, I share your concerns around selecting the right knobs and > testing accordingly but my main concern now is that I feel many people will > hit this limitation when they deploy bigger charms in containers. Is there > any way

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

** Changed in: juju-core Assignee: Christian Muirhead (2-xtian) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

@stgrabber, what's the status on this? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1602192 Title: when starting many LXD containers, they start failing to boot with "Too many open files" To

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

So there is a patchset that's been sent for review a couple of times upstream which would tie a bunch of the inotify limits to a user namespace. This would almost certainly fix this issue. Until then, we can workaround it by bumping the default values for those knobs. This bump could be done in

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

It sounds like the "fix" for this lies outside of LXD. Can we add the affected packages and involve the right upstreams for this? Are we going to pursue this purely upstream with systemd / kernel or are we going to attempt a distro-patch of some sort? -- You received this bug notification

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

It does look like max_user_instances is global and not namespaced. So unless there's a resource leak somewhere that needs to be fixed, bumping that limit may be the only option. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

I don't know that much about /proc/sys/fs/inotify/max_user_instances, but apparently this isn't properly namespaced, so it is a global limit for all containers on the host? As a workaround, lxd could ship a /usr/lib/sysctl.d/ snippet to bump it, but I'm not sure if that has other downsides

[Bug 1602192] Re: when starting many LXD containers, they start failing to boot with "Too many open files"

Apparently the relevant limit is /proc/sys/fs/inotify/max_user_instances. This is "128" by default. When increasing it with sudo sysctl fs.inotify.max_user_instances=256 then the failed container reboots fine (and udev starts). -- You received this bug notification because you are a member