so ive built yocto with kubernetes k8s flavor, image created and booted,
ive done this before with overc which back then worked fine, yet vanilla
yocto master seems to have issue
swapoff -a
k8sinit
root@node-1:~# swapoff -a
root@node-1:~# k8s-init
/bin/k8s-init: line 11: /etc/sysconfig/iptables: No such file or directory
I0528 02:21:13.465437 479 version.go:255] remote version is much newer:
v1.24.1; falling back to: stable-1.23
[init] Using Kubernetes version: v1.23.7
[preflight] Running pre-flight checks
[WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your
internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm
config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes
kubernetes.default kubernetes.default.svc
kubernetes.default.svc.cluster.local node-1] and IPs [10.96.0.1
192.168.0.189]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node-1]
and IPs [192.168.0.189 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node-1]
and IPs [192.168.0.189 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file
"/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file
"/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in
"/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane
as static Pods from directory "/etc/kubernetes/manifests". This can take up
to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some
way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the
error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when
started by the container runtime.
To troubleshoot, list all containers using your preferred container
runtimes CLI.
Here is one example how you may list all Kubernetes containers running in
cri-o/containerd using crictl:
- 'crictl --runtime-endpoint /run/containerd/containerd.sock ps -a | grep
kube | grep -v pause'
Once you have found the failing container, you can inspect its
logs with:
- 'crictl --runtime-endpoint /run/containerd/containerd.sock logs
CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes
cluster
To see the stack trace of this error execute with --v=5 or higher
journalctl says
May 28 02:22:50 node-1 kubelet[825]: E0528 02:22:50.457191 825
remote_runtime.go:448] "StartContainer from runtime service failed"
err="rpc error: code = Unknown desc = failed to create containerd task:
failed to create shim task: OCI runtime create failed: runc create failed:
unable to start container process: error during container init: error
setting cgroup config for procHooks process: openat2
/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod2f35c35e9fe662a834b6d53268894e8a.slice:cri-containerd:a87e4c5256ef0c61cd2ccc453580936d6cc6f10c9febec5ddf190fd40753c148/cpu.cfs_period_us:
no such file or directory: unknown"
containerID="a87e4c5256ef0c61cd2ccc453580936d6cc6f10c9febec5ddf190fd40753c148"
May 28 02:22:50 node-1 kubelet[825]: E0528 02:22:50.457462 825
kuberuntime_manager.go:919] container
&Container{Name:etcd,Image:k8s.gcr.io/etcd:3.5.1-0,Command:[etcd
--advertise-client-urls=https://192.168.0.189:2379
--cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true
--data-dir=/var/lib/etcd
--initial-advertise-peer-urls=https://192.168.0.189:2380
--initial-cluster=node-1=https://192.168.0.189:2380
--key-file=/etc/kubernetes/pki/etcd/server.key
--listen-client-urls=https://127.0.0.1:2379,https://192.168.0.189:2379
--listen-metrics-urls=http://127.0.0.1:2381
--listen-peer-urls=https://192.168.0.189:2380 --name=node-1
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-client-cert-auth=true
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--snapshot-count=10000
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu:
{{100 -3} {<nil>} 100m DecimalSI},memory: {{104857600 0} {<nil>} 100Mi
BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:etcd-data,ReadOnly:false,MountPath:/var/lib/etcd,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:etcd-certs,ReadOnly:false,MountPath:/etc/kubernetes/pki/etcd,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:{0
2381
},Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,TerminationGracePeriodSeconds:nil,},ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:{0
2381
},Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:24,TerminationGracePeriodSeconds:nil,},}
start failed in pod
etcd-node-1_kube-system(2f35c35e9fe662a834b6d53268894e8a):
RunContainerError: failed to create containerd task: failed to create shim
task: OCI runtime create failed: runc create failed: unable to start
container process: error during container init: error setting cgroup config
for procHooks process: openat2
/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod2f35c35e9fe662a834b6d53268894e8a.slice:cri-containerd:a87e4c5256ef0c61cd2ccc453580936d6cc6f10c9febec5ddf190fd40753c148/cpu.cfs_period_us:
no such file or directory: unknown
May 28 02:22:50 node-1 kubelet[825]: E0528 02:22:50.457564 825
pod_workers.go:951] "Error syncing pod, skipping" err="failed to
\"StartContainer\" for \"etcd\" with RunContainerError: \"failed to create
containerd task: failed to create shim task: OCI runtime create failed:
runc create failed: unable to start container process: error during
container init: error setting cgroup config for procHooks process: openat2
/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod2f35c35e9fe662a834b6d53268894e8a.slice:cri-containerd:a87e4c5256ef0c61cd2ccc453580936d6cc6f10c9febec5ddf190fd40753c148/cpu.cfs_period_us:
no such file or directory: unknown\"" pod="kube-system/etcd-node-1"
podUID=2f35c35e9fe662a834b6d53268894e8a
May 28 02:22:50 node-1 containerd[350]:
time="2022-05-28T02:22:50.469596452Z" level=error msg="StartContainer for
\"d5f0203e814a4d14c24c17a2c2b3982b43e7427ab135b0c98956ee638c82aeaf\"
failed" error="failed to create containerd task: failed to create shim
task: OCI runtime create failed: runc create failed: unable to start
container process: error during container init: error setting cgroup config
for procHooks process: openat2
/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod90c43fd55dbe3c307fa34d50b43224e6.slice:cri-containerd:d5f0203e814a4d14c24c17a2c2b3982b43e7427ab135b0c98956ee638c82aeaf/cpu.cfs_period_us:
no such file or directory: unknown"
May 28 02:22:50 node-1 containerd[350]:
time="2022-05-28T02:22:50.469657943Z" level=error msg="StartContainer for
\"70b9504b4733741c4274884ad69f702818e071ecd6e62dca81382a68ca64c1b1\"
failed" error="failed to create containerd task: failed to create shim
task: OCI runtime create failed: runc create failed: unable to start
container process: error during container init: error setting cgroup config
for procHooks process: openat2
/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod4339400940323fa9543500bd60488683.slice:cri-containerd:70b9504b4733741c4274884ad69f702818e071ecd6e62dca81382a68ca64c1b1/cpu.cfs_period_us:
no such file or directory: unknown"
May 28 02:22:50 node-1 containerd[350]:
time="2022-05-28T02:22:50.469702603Z" level=error msg="StartContainer for
\"038b05d9f6f77c2c705da5f66f417b84697f0b66642f7d9b90693877e36c9b83\"
failed" error="failed to create containerd task: failed to create shim
task: OCI runtime create failed: runc create failed: unable to start
container process: error during container init: error setting cgroup config
for procHooks process: openat2
/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-podd94f28892ce55932f3909341450b4129.slice:cri-containerd:038b05d9f6f77c2c705da5f66f417b84697f0b66642f7d9b90693877e36c9b83/cpu.cfs_period_us:
no such file or directory: unknown"
root@node-1:~# crictl --runtime-endpoint /run/containerd/containerd.sock ps
-a
I0528 02:30:50.674997 2376 util_unix.go:104] "Using this endpoint is
deprecated, please consider using full URL format"
endpoint="/run/containerd/containerd.sock"
URL="unix:///run/containerd/containerd.sock"
I0528 02:30:50.675769 2376 util_unix.go:104] "Using this endpoint is
deprecated, please consider using full URL format"
endpoint="/run/containerd/containerd.sock"
URL="unix:///run/containerd/containerd.sock"
CONTAINER IMAGE CREATED STATE
NAME ATTEMPT POD ID POD
027a394f5bc72 ed0ccfa052ab4 2 minutes ago Exited
kube-scheduler 6 f3d96a4b9301d
kube-scheduler-node-1
4516ebb79f90e 03c169f383d97 2 minutes ago Exited
kube-apiserver 6 8937c47e4a03d
kube-apiserver-node-1
0d1497cffec50 25f8c7f3da61c 2 minutes ago Exited
etcd 6 1ee2992c605bc
etcd-node-1
881632903e7f3 e34d4a6252edd 2 minutes ago Exited
kube-controller-manager 6 25a48784ec11f
kube-controller-manager-node-1
--
Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#7326):
https://lists.yoctoproject.org/g/meta-virtualization/message/7326
Mute This Topic: https://lists.yoctoproject.org/mt/91389955/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
