Found this in kube-proxy logs, not sure how useful this is: Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: W0224 > 23:55:15.159640 3039 server.go:468] Failed to retrieve node info: nodes " > kube-worker01.mydomain.com" not found
Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: W0224 > 23:55:15.160308 3039 proxier.go:249] invalid nodeIP, initialize > kube-proxy with 127.0.0.1 as nodeIP Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: W0224 > 23:55:15.160596 3039 proxier.go:254] clusterCIDR not specified, unable > to distinguish between internal and external traffic Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.160832 3039 server.go:227] Tearing down userspace rules. Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.161206 3039 healthcheck.go:119] Initializing kube-proxy health checker Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.170297 3039 conntrack.go:81] Set sysctl > 'net/netfilter/nf_conntrack_max' to 131072 Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.170731 3039 conntrack.go:66] Setting conntrack hashsize to 32768 Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.170920 3039 conntrack.go:81] Set sysctl > 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.170950 3039 conntrack.go:81] Set sysctl > 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.172295 3039 proxier.go:802] Not syncing iptables until Services > and Endpoints have been received from master Feb 24 23:55:15 kube-worker01.mydomain.com kube-proxy[3039]: I0224 > 23:55:15.173529 3039 proxier.go:472] Adding new service > "default/kubernetes:https" at 172.32.0.1:443/TCP Abejide Ayodele It always seems impossible until it's done. --Nelson Mandela On Fri, Feb 24, 2017 at 5:51 PM, 'Vishnu Kannan' via Kubernetes user discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > +sig-cluster-lifecycle > > On Fri, Feb 24, 2017 at 3:45 PM, Àbéjídé Àyodélé <abejideayod...@gmail.com > > wrote: > >> Like I said I have a similar cluster that I setup by hand that works, >> trying to automate the cluster I setup by hand is where I am running into >> problems, also I am interested in knowing why I am running into this >> problem, so I can learn for the future >> >> Abejide Ayodele >> It always seems impossible until it's done. --Nelson Mandela >> >> On Fri, Feb 24, 2017 at 5:42 PM, 'Vishnu Kannan' via Kubernetes user >> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >> >>> It depends on how you are trying to setup the cluster. I'd recommend >>> starting with a known working solution like "kubeadm >>> <https://kubernetes.io/docs/getting-started-guides/kubeadm/>" to >>> bootstrap your cluster before customizing custer bringup. >>> >>> On Fri, Feb 24, 2017 at 3:39 PM, ayodele abejide < >>> abejideayod...@gmail.com> wrote: >>> >>>> Your node bootstrapping is failing. >>>> >>>> >>>> How do I debug this? >>>> >>>> >>>> >>>> On Friday, 24 February 2017 17:28:53 UTC-6, Vishnu Kannan wrote: >>>>> >>>>> Your node bootstrapping is failing. I'd recommend focussing on getting >>>>> your node registered and ignore the eviction manager error. The latter is >>>>> a >>>>> red herring. >>>>> >>>>> On Fri, Feb 24, 2017 at 2:45 PM, ayodele abejide <abejide...@gmail.com >>>>> > wrote: >>>>> >>>>>> kubeclt get nodes returns No resources found. I do not know how to >>>>>> verify: >>>>>> >>>>>> >>>>>> Has the node successfully registered itself with the apiserver? >>>>>> >>>>>> >>>>>> Relevant code that is generating the failure is here >>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Fblob%2F55bee3ad21f025b1416a4e1f10de753f484b66d3%2Fpkg%2Fkubelet%2Fserver%2Fstats%2Fsummary.go%23L80&sa=D&sntz=1&usg=AFQjCNEzlGPM6j6QAZEUHvbLqsKVNqo6CQ> >>>>>> . >>>>>> >>>>>> >>>>>> I sort of have browsed the code, but don't have enough context to get >>>>>> enough information out of it >>>>>> >>>>>> >>>>>> On Friday, 24 February 2017 16:25:49 UTC-6, Vishnu Kannan wrote: >>>>>>> >>>>>>> Kubelet is unable to retrieve the "Node" object that represents the >>>>>>> node its running on from the api server. Has the node successfully >>>>>>> registered itself with the apiserver? >>>>>>> >>>>>>> Relevant code that is generating the failure is here >>>>>>> <https://github.com/kubernetes/kubernetes/blob/55bee3ad21f025b1416a4e1f10de753f484b66d3/pkg/kubelet/server/stats/summary.go#L80> >>>>>>> . >>>>>>> >>>>>>> On Fri, Feb 24, 2017 at 2:19 PM, ayodele abejide < >>>>>>> abejide...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have searched the internet and found no answers that solve my >>>>>>>> problem. >>>>>>>> >>>>>>>> Background: >>>>>>>> >>>>>>>> I have a working cluster that I intended to automate it's creation >>>>>>>> via puppet and terraform, all seemed to have worked well except I find: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Feb 24 22:05:41 kube-worker02.mydomain.com kubelet[18956]: E0224 22 >>>>>>>> :05:41.775438 18956 eviction_manager.go:204] eviction manager: >>>>>>>> unexpected err: failed GetNode: node 'kube-worker02.mydomain.com' >>>>>>>> not found >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: I0224 22 >>>>>>>> :05:51.725950 18956 kubelet.go:1155] Image garbage collection >>>>>>>> succeeded >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: W0224 22 >>>>>>>> :05:51.733365 18956 container_manager_linux.go:728] CPUAccounting >>>>>>>> not enabled for pid: 6095 >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: W0224 22 >>>>>>>> :05:51.733381 18956 container_manager_linux.go:731] >>>>>>>> MemoryAccounting not enabled for pid: 6095 >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: I0224 22 >>>>>>>> :05:51.733391 18956 container_manager_linux.go:434] Discovered >>>>>>>> runtime cgroups name: /system.slice/docker.service >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: W0224 22 >>>>>>>> :05:51.733478 18956 container_manager_linux.go:728] CPUAccounting >>>>>>>> not enabled for pid: 18956 >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: W0224 22 >>>>>>>> :05:51.733489 18956 container_manager_linux.go:731] >>>>>>>> MemoryAccounting not enabled for pid: 18956 >>>>>>>> Feb 24 22:05:51 kube-worker02.mydomain.com kubelet[18956]: E0224 22 >>>>>>>> :05:51.776184 18956 eviction_manager.go:204] eviction manager: >>>>>>>> unexpected err: failed GetNode: node 'kube-worker02.mydomain.com' >>>>>>>> not found >>>>>>>> >>>>>>>> in the kubelet logs. >>>>>>>> >>>>>>>> What have I tried? >>>>>>>> >>>>>>>> I have disabled ssl, set authorization-mode=AlwaysAllow, confirmed >>>>>>>> that kube-proxy running on the same worker can reach the api-server, >>>>>>>> also >>>>>>>> confirmed that cadvisor is running: >>>>>>>> >>>>>>>> curl localhost:4194/api/v2.0/ >>>>>>>> >>>>>>>> >>>>>>>> Supported request types: "appmetrics,attributes,events, >>>>>>>> machine,ps,spec,stats,storage,summary,version" >>>>>>>> >>>>>>>> >>>>>>>> I have also rebooted the worker a couple of times and restarted >>>>>>>> kubelet and kube-proxy so many times. >>>>>>>> >>>>>>>> I am at loss here and don't know what to try, any help will be >>>>>>>> appreciated. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Kubernetes user discussion and Q&A" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to kubernetes-use...@googlegroups.com. >>>>>>>> To post to this group, send email to kubernet...@googlegroups.com. >>>>>>>> Visit this group at https://groups.google.com/grou >>>>>>>> p/kubernetes-users. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Kubernetes user discussion and Q&A" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to kubernetes-use...@googlegroups.com. >>>>>> To post to this group, send email to kubernet...@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Kubernetes user discussion and Q&A" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to kubernetes-users+unsubscr...@googlegroups.com. >>>> To post to this group, send email to kubernetes-users@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Kubernetes user discussion and Q&A" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/kubernetes-users/bpER7WJX-Jc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> kubernetes-users+unsubscr...@googlegroups.com. >>> To post to this group, send email to kubernetes-users@googlegroups.com. >>> Visit this group at https://groups.google.com/group/kubernetes-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to kubernetes-users+unsubscr...@googlegroups.com. >> To post to this group, send email to kubernetes-users@googlegroups.com. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Kubernetes user discussion and Q&A" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/kubernetes-users/bpER7WJX-Jc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
