The following pull request was submitted through Github. It can be accessed and reviewed at: https://github.com/lxc/lxd/pull/4801
This e-mail was sent by the LXC bot, direct replies will not reach the author unless they happen to be subscribed to this list. === Description (from pull-request) === The default ARP table size in the kernel has a limit of 1024 entries. If you launch more than a thousand containers, you will fill up the time and no new entries get accepted because each container takes up an entry in the ARP table.
From fd6bf81d98f65625c8406bd816ae88baa46b9455 Mon Sep 17 00:00:00 2001 From: Simos Xenitellis <[email protected]> Date: Tue, 17 Jul 2018 18:24:26 +0300 Subject: [PATCH] "neighbour: ndisc_cache: neighbor table overflow" The default ARP table size in the kernel has a limit of 1024 entries. If you launch more than a thousand containers, you will fill up the time and no new entries get accepted because each container takes up an entry in the ARP table. --- doc/production-setup.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/production-setup.md b/doc/production-setup.md index cfd75ce06..2a15e4fd3 100644 --- a/doc/production-setup.md +++ b/doc/production-setup.md @@ -16,6 +16,7 @@ using containers that require tens of thousands of file operations. `failed to open stream: Too many open files in...` +`neighbour: ndisc_cache: neighbor table overflow!` # Server Changes ## /etc/security/limits.conf @@ -39,12 +40,14 @@ fs.inotify.max\_user\_instances | 1048576 | 128 | This specifies an upper fs.inotify.max\_user\_watches | 1048576 | 8192 | This specifies an upper limit on the number of watches that can be created per real user ID. [1] vm.max\_map\_count | 262144 | 65530 | This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries. kernel.dmesg\_restrict | 1 | 0 | This denies container access to the messages in the kernel ring buffer. Please note that this also will deny access to non-root users on the host system. - +net.ipv4.neigh.default.gc_thresh3| 8192 | 1024 | This is the maximum number of entries in ARP table (IPv4). You should increase this if you create over 1024 containers. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and those containers will not be able to get a network configuration. [2] +net.ipv6.neigh.default.gc_thresh3| 8192 | 1024 | This is the maximum number of entries in ARP table (IPv6). You should increase this if you plan to create over 1024 containers. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and those containers will not be able to get a network configuration. [2] Then, reboot the server. [1]: http://man7.org/linux/man-pages/man7/inotify.7.html +[2]: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt ## Network Bandwidth Tweaking If you have at least 1GbE NIC on your lxd host with a lot of local activity (container - container connections, or host - container connections), or you have 1GbE or better internet connection on your lxd host it worth play with txqueuelen. These settings work even better with 10GbE NIC.
_______________________________________________ lxc-devel mailing list [email protected] http://lists.linuxcontainers.org/listinfo/lxc-devel
