On Wed, Nov 11, 2020 at 5:07 AM Lance Yang <[email protected]> wrote: > > Hi Bruce, > > Thanks for your reply. It took me a long time to address the $PATH issue. > > I found the cause after I cleaned my environment. it is because I set my own > $PATH before k3s startup which did not include "/bin" and some components in > k3s will use "os.Getenv("PATH") to search the related tool. > > Since some OS may not have systemd and the k3s-clean script seems not to > delete all interfaces created by cni and umount all mount points related to > k3s. I wrote a script for this situation and I will send it through a > separate email based on your previous scripts. >
Sounds good. You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now. I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server. As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error. I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration. Bruce > Best Regards, > Lance > > -----Original Message----- > > From: [email protected] > > <[email protected]> > > On Behalf Of Bruce Ashfield via lists.yoctoproject.org > > Sent: Tuesday, November 10, 2020 9:35 PM > > To: Bruce Ashfield <[email protected]> > > Cc: Lance Yang <[email protected]>; Joakim Roubert > > <[email protected]>; meta- > > [email protected]; Michael Zhao <[email protected]>; Kaly > > Xin > > <[email protected]> > > Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe > > > > On Tue, Nov 10, 2020 at 8:17 AM Bruce Ashfield via lists.yoctoproject.org > > <[email protected]> wrote: > > > > > > On Tue, Nov 10, 2020 at 7:46 AM Bruce Ashfield via > > > lists.yoctoproject.org > > > <[email protected]> wrote: > > > > > > > > On Tue, Nov 10, 2020 at 1:43 AM Lance Yang <[email protected]> wrote: > > > > > > > > > > Hi Bruce and Joakim, > > > > > > > > > > Thanks for sharing this branch: k3s-wip. I have tested against my > > > > > yocto build. > > > > > > > > The branch will be more functional shortly, I have quite a few > > > > changes to factor things for k8s and generally more usable :D > > > > > > > > modified: classes/cni_networking.bbclass > > > > modified: conf/layer.conf > > > > modified: > > > > recipes-containers/containerd/containerd-docker_git.bb > > > > modified: > > > > recipes-containers/containerd/containerd-opencontainers_git.bb > > > > modified: recipes-containers/k3s/README.md > > > > modified: recipes-containers/k3s/k3s_git.bb > > > > modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg > > > > modified: recipes-networking/cni/cni_git.bb > > > > container-deploy.txt > > > > recipes-core/packagegroups/ > > > > > > > > > > > > > > My Image: Linux qemuarm64 by yocto. > > > > > > > > > > The master node can be ready after I started the k3s server. However, > > > > > the pods in kube- > > system (which are essential components for k3s) cannot turn to ready state > > on qemuarm64. > > > > > > > > > > > > > That's interesting, since in my configuration, the master never comes > > > > ready: > > > > > > > > root@qemux86-64:~# kubectl get nodes > > > > NAME STATUS ROLES AGE VERSION > > > > qemux86-64 NotReady master 15h v1.18.9-k3s1 > > > > > > > > > > Hah. > > > > > > I finally got the node to show up as ready: > > > > > > root@qemux86-64:~# kubectl get nodes > > > NAME STATUS ROLES AGE VERSION > > > qemux86-64 Ready master 112s v1.18.9-k3s1 > > > > > > > Lance, > > > > What image type were you building ? I'm pulling in dependencies to > > packagegroups and the > > recipes themselves. > > > > I'm not seeing the mount issue on my master/server node: > > > > root@qemux86-64:~# kubectl get pods -n kube-system > > NAME READY STATUS RESTARTS AGE > > local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 > > 3m32s > > metrics-server-7566d596c8-mwntr 1/1 Running 0 > > 3m32s > > helm-install-traefik-229v7 0/1 Completed 0 > > 3m32s > > coredns-7944c66d8d-9rfj7 1/1 Running 0 > > 3m32s > > svclb-traefik-pb5j4 2/2 Running 0 > > 2m29s > > traefik-758cd5fc85-lxpr8 1/1 Running 0 > > 2m29s > > > > I'm going back to all-in-one node debugging, but can look into the mount > > issue more later. > > > > Bruce > > > > > I'm attempting to build an all-in-one node, and that is likely causing > > > me some issues. > > > > > > I'm revisiting those potential conflicts now. > > > > > > But if anyone else does have an all in one working and has some tips, > > > feel free to share :D > > > > > > Bruce > > > > > > > I've sorted out more of the dependencies, and have packagegroups to > > > > make them easier now. > > > > > > > > Hopefully, I can figure out what is now missing and keeping my > > > > master from moving into ready today. > > > > > > > > Bruce > > > > > > > > > After the master node itself turned to ready state, I check the pods > > > > > with kubectl: > > > > > > > > > > kubectl get nodes > > > > > NAME STATUS ROLES AGE VERSION > > > > > qemuarm64 Ready master 11m v1.18.9-k3s1 > > > > > root@qemuarm64:~# ls > > > > > root@qemuarm64:~# kubectl get pods -n kube-system > > > > > NAME READY STATUS > > > > > RESTARTS AGE > > > > > local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating > > > > > 0 12m > > > > > coredns-7944c66d8d-tlrm9 0/1 ContainerCreating > > > > > 0 12m > > > > > metrics-server-7566d596c8-svkff 0/1 ContainerCreating > > > > > 0 12m > > > > > helm-install-traefik-s8p5g 0/1 ContainerCreating > > > > > 0 12m > > > > > > > > > > Then I describe the pods with: > > > > > > > > > > Events: > > > > > Type Reason Age From > > > > > Message > > > > > ---- ------ ---- ---- > > > > > ------- > > > > > Normal Scheduled 16m default-scheduler > > > > > Successfully assigned kube- > > system/coredns-7944c66d8d-tlrm9 to qemuarm64 > > > > > Warning FailedMount 5m23s (x3 over 14m) kubelet > > > > > Unable to attach or mount > > volumes: unmounted volumes=[coredns-token-b7nlh], unattached > > volumes=[coredns-token- > > b7nlh config-volume]: timed out waiting for the condition > > > > > Warning FailedMount 50s (x4 over 12m) kubelet > > > > > Unable to attach or mount > > volumes: unmounted volumes=[coredns-token-b7nlh], unattached > > volumes=[config-volume > > coredns-token-b7nlh]: timed out waiting for the condition > > > > > Warning FailedMount 11s (x16 over 16m) kubelet > > > > > MountVolume.SetUp failed > > for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable > > file not found in > > $PATH > > > > > > > > > > I found the "mount" binary is not found in $PATH. However, I > > > > > confirmed the $PATH and > > mount binary on my qemuarm64 image: > > > > > > > > > > root@qemuarm64:~# echo $PATH > > > > > /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin > > > > > root@qemuarm64:~# which mount > > > > > /bin/mount > > > > > > > > > > When I type mount command, it worked fine: > > > > > > > > > > /dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type > > > > > devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) > > > > > proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs > > > > > (rw,relatime) debugfs on /sys/kernel/debug type debugfs > > > > > (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) > > > > > ... > > > > > ... (skipped the verbose output) > > > > > > > > > > I would like to know whether you have met this "mount" issue ever? > > > > > > > > > > Best Regards, > > > > > Lance > > > > > > > > > > > -----Original Message----- > > > > > > From: [email protected] > > > > > > <[email protected]> > > > > > > On Behalf Of Bruce Ashfield via lists.yoctoproject.org > > > > > > Sent: Monday, October 26, 2020 11:46 PM > > > > > > To: Joakim Roubert <[email protected]> > > > > > > Cc: [email protected] > > > > > > Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe > > > > > > > > > > > > On Wed, Oct 21, 2020 at 2:00 AM Joakim Roubert > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > On 2020-10-21 05:10, Bruce Ashfield wrote: > > > > > > > > Ha!!!! > > > > > > > > > > > > > > > > This applies. > > > > > > > > > > > > > > Wonderful, thank you! I guess this is what is called "five times > > > > > > > lucky"... > > > > > > > > > > > > > > > I'm now testing and completing some of my networking > > > > > > > > factoring, as well as importing / forking some recipes to > > > > > > > > avoid extra layer depends. > > > > > > > > > > > > > > Excellent! > > > > > > > > > > > > I've pushed some of my WIP to: > > > > > > https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/l > > > > > > og/?h=k3s-wip > > > > > > > > > > > > That includes the split of the networking, the import of some of > > > > > > the dependencies and some small tweaks I'm working on. > > > > > > > > > > > > I did have a couple of questions on the k3s packaging itself, I > > > > > > was getting the following > > > > > > error: > > > > > > > > > > > > ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s: > > > > > > Files/directories were installed but not shipped in any package: > > > > > > /usr/local/bin/k3s-clean > > > > > > /usr/local/bin/crictl > > > > > > /usr/local/bin/kubectl > > > > > > /usr/local/bin/k3s > > > > > > > > > > > > So I added them to the FILES of the k3s package itself (so both > > > > > > k3s-server and k3s-agent will get them), is that the split you were > > > > > > looking for ? > > > > > > > > > > > > Bruce > > > > > > > > > > > > > > > > > > > > BR, > > > > > > > > > > > > > > /Joakim > > > > > > > -- > > > > > > > Joakim Roubert > > > > > > > Senior Engineer > > > > > > > > > > > > > > Axis Communications AB > > > > > > > Emdalavägen 14, SE-223 69 Lund, Sweden > > > > > > > Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48 > > > > > > > Fax: +46 46 13 61 30, www.axis.com > > > > > > > > > > > > > -- > > > > > > - Thou shalt not follow the NULL pointer, for chaos and madness > > > > > > await thee at its end > > > > > > - "Use the force Harry" - Gandalf, Star Trek II > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > > > > > confidential and > > may also be privileged. If you are not the intended recipient, please > > notify the sender > > immediately and do not disclose the contents to any other person, use it > > for any purpose, or > > store or copy the information in any medium. Thank you. > > > > > > > > > > > > > > > > -- > > > > - Thou shalt not follow the NULL pointer, for chaos and madness > > > > await thee at its end > > > > - "Use the force Harry" - Gandalf, Star Trek II > > > > > > > > > > > > > > > > > > > > > -- > > > - Thou shalt not follow the NULL pointer, for chaos and madness await > > > thee at its end > > > - "Use the force Harry" - Gandalf, Star Trek II > > > > > > > > > > > > > > > -- > > - Thou shalt not follow the NULL pointer, for chaos and madness await thee > > at its end > > - "Use the force Harry" - Gandalf, Star Trek II > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#6059): https://lists.yoctoproject.org/g/meta-virtualization/message/6059 Mute This Topic: https://lists.yoctoproject.org/mt/77679236/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
