ulimits are applied to processes not users. That process has a 4096 file limit.
How are you running the API server? Systemd? Docker container? Can you provide a gist of what you are doing so I can suggest a fix? On Wed, Nov 30, 2016 at 1:22 PM <kvcha...@gmail.com> wrote: > On Tuesday, November 29, 2016 at 5:30:53 PM UTC-8, Brandon Philips wrote: > > How are you running the API server? If you are running it in a container > your shell ulimits won't match. Find the API server PID and do: > > > > > > cat /proc/$PID/limits to check > > > > > > On Tue, Nov 29, 2016 at 5:11 PM <kvch...@gmail.com> wrote: > > On Tuesday, November 29, 2016 at 4:34:19 PM UTC-8, Daniel Smith wrote: > > > > > You need to increase the number of file descriptors. ulimit is the > relevant search term. > > > > > > > > > > > > > > > On Tue, Nov 29, 2016 at 3:26 PM, <kvch...@gmail.com> wrote: > > > > > with <= 120 nodes there is no issue, once it reaches to node 121 then > I got an error: /var/log/upstart/kube-apiserver.log > > > > > > > > > > > > > > > > > > > > I1129 06:35:40.163511 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:41.163668 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:42.163800 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:43.163919 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > W1129 06:35:43.912814 7655 storage_extensions.go:83] third party > resource sync failed: client: etcd cluster is unavailable or misconfigured > > > > > > > > > > I1129 06:35:44.164048 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:45.164165 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:46.164277 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:47.164399 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:48.164524 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > I1129 06:35:49.164664 7655 logs.go:41] http: Accept error: accept > tcp [::]:8080: accept4: too many open files; retrying in 1s > > > > > > > > > > -------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > root@ovrhxxxx:/var/log/upstart$ ulimit -a > > > > > > > > > > core file size (blocks, -c) 0 > > > > > > > > > > data seg size (kbytes, -d) unlimited > > > > > > > > > > scheduling priority (-e) 0 > > > > > > > > > > file size (blocks, -f) unlimited > > > > > > > > > > pending signals (-i) 772117 > > > > > > > > > > max locked memory (kbytes, -l) 64 > > > > > > > > > > max memory size (kbytes, -m) unlimited > > > > > > > > > > open files (-n) 100000 > > > > > > > > > > pipe size (512 bytes, -p) 8 > > > > > > > > > > POSIX message queues (bytes, -q) 819200 > > > > > > > > > > real-time priority (-r) 0 > > > > > > > > > > stack size (kbytes, -s) 8192 > > > > > > > > > > cpu time (seconds, -t) unlimited > > > > > > > > > > max user processes (-u) 772117 > > > > > > > > > > virtual memory (kbytes, -v) unlimited > > > > > > > > > > file locks (-x) unlimited > > > > > > > > > > root@ovrhxxxx:/var/log/upstart$ > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the Google > Groups "Kubernetes user discussion and Q&A" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send > an email to kubernetes-use...@googlegroups.com. > > > > > > > > > > To post to this group, send email to kubernet...@googlegroups.com. > > > > > > > > > > Visit this group at https://groups.google.com/group/kubernetes-users. > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > ----------------------- > > > > > > > > I increated the "open file" value from 1024 to 100000 and still had the > same issue. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups "Kubernetes user discussion and Q&A" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > an email to kubernetes-use...@googlegroups.com. > > > > To post to this group, send email to kubernet...@googlegroups.com. > > > > Visit this group at https://groups.google.com/group/kubernetes-users. > > > > For more options, visit https://groups.google.com/d/optout. > > --------------- > Brandon, Thanks for the info... > > Since kube-apiserver is running under "root" privileges. > Is there a way to increase the "open file" value for "root"? > I changed on /etc/security/limits.conf for 'root' and still show with > default value with 1024 for soft. > > root@ovrhxx0301:/var/log/upstart# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 772117 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 100000 > pipe size (512 bytes, -p) 8 > ------------------------------- > # cat /proc/3539/limits > Limit Soft Limit Hard Limit Units > Max cpu time unlimited unlimited seconds > Max file size unlimited unlimited bytes > Max data size unlimited unlimited bytes > Max stack size 8388608 unlimited bytes > Max core file size 0 unlimited bytes > Max resident set unlimited unlimited bytes > Max processes 772117 772117 > processes > Max open files 1024 4096 files > ---------------------- > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
