Hi Wei, upgrading from 4.16 to 4.19 will be a huge work for me I have to test vpc/shared network, vrrp vpc with public lb Netscaler and withou it on several VPCs, i added support for VPX 12 and VPX 13 on it with a nginx middleware rewriting rules from CS I am on xcp-ng 8.2.1 i kept during years with a CS 4.3, restarting from 4.16, i would like this time to keep LTS in production Is there a specific upgrade path to respect ? a specific process to observe ? what rollback mechanism is possible ?
For my K8S needs, I would like to upgrade it, but it was looking first for a preproduction K8s working on 1.28 in order integrating CI/CD pipelines So this would not too be a problem upgrading CS after installing a k8s cluster for CI/CD Rohit was talking about CAPC, i can use it independently of the version of ACS i use ? Thanks Le ven. 12 avr. 2024 à 14:30, Wei ZHOU <ustcweiz...@gmail.com> a écrit : > Hi, > > I would suggest you not to spend time on this... 4.16 is EOL > > Please upgrade to cloudstack 4.18/4.19, or use old kubernetes versions .. > > > -Wei > > > On Fri, Apr 12, 2024 at 2:25 PM benoit lair <kurushi4...@gmail.com> wrote: > > > > i succeded to install systemvm template 4.19 as serving template for > > control and worker nodes with cks community iso 1.28.4 > > when cluster saying "starting", i connect to node controller via p2222, > saw > > that docker was not installed, but containerd.io yes > > > > I've done the following : > > > > apt install docker-ce > > cp /etc/containerd/config.toml /etc/containerd/config.toml.bck > > containerd config default | tee /etc/containerd/config.toml > > /opt/bin/setup-kube-system > > /opt/bin/deploy-kube-system > > > > on to control node and same after on worker node > > CS UI show kubernetes yaml > > a "kubectl.exe --kubeconfig kube.conf.conf get nodes" > > > > NAME STATUS ROLES AGE > VERSION > > k8s-cks-cl19-control-18ed18311c9 Ready control-plane 3h1m > v1.28.4 > > k8s-cks-cl19-node-18ed1850433 Ready <none> 170m > v1.28.4 > > > > However CS says the cluster is in alert state > > And dashboad is not working > > > > Any advice ? > > > > when executing this on my laptop : > > kubectl.exe --kubeconfig cl19_k8s_1.28.4.conf proxy > > > > and opening this : > > > http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ > > > > I have this result : > > { > > "kind": "Status", > > "apiVersion": "v1", > > "metadata": {}, > > "status": "Failure", > > "message": "no endpoints available for service > \"kubernetes-dashboard\"", > > "reason": "ServiceUnavailable", > > "code": 503 > > } > > > > Somebody got this problem with dashboard ? > > > > Did i miss something installing the cluster manually with acs stuff ? > > > > Le jeu. 11 avr. 2024 à 17:40, benoit lair <kurushi4...@gmail.com> a > écrit : > > > > > Hi Wei, > > > > > > Thanks for the sharing, i also tried to install the systemvm 4.19 > > > I have control node and worker node under template systemvm 4.19 ( > > > > http://download.cloudstack.org/systemvm/4.19/systemvmtemplate-4.19.0-xen.vhd.bz2 > > > ) > > > I tried with community cks iso 1.25.0 and 1.28.9 > > > On systemvm 4.19 docker was not present by default, just containerd in > > > 1.6.x version > > > I installed docker-ce and docker-ce-cli : > > > > > > apt install -y apt-transport-https ca-certificates curl > > > software-properties-common > > > curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg > > > --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg > > > echo "deb [arch=$(dpkg --print-architecture) > > > signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] > > > https://download.docker.com/linux/debian $(lsb_release -cs) stable" | > > > sudo tee /etc/apt/sources.list.d/docker.list > /dev/null > > > remove line about docker in double in /etc/apt/sources.list > > > apt update > > > apt install -y docker-ce > > > apt install -y docker-cli > > > i mounted the 1.28.4 cks community iso in control vm > > > > > > and /opt/bin/setup-kube-system > > > i have these errors : > > > > > > Is it the good way to manually install kube system > > > > > > root@k8s-cks-cl17-control-18ecdc1e5bd:/home/core# > > > /opt/bin/setup-kube-system > > > Installing binaries from /mnt/k8sdisk/ > > > 5b1fa8e3e100: Loading layer > > > [==================================================>] 803.8kB/803.8kB > > > 39c831b1aa26: Loading layer > > > [==================================================>] 26.25MB/26.25MB > > > Loaded image: apache/cloudstack-kubernetes-autoscaler:latest > > > 417cb9b79ade: Loading layer > > > [==================================================>] 657.7kB/657.7kB > > > 8d323b160d65: Loading layer > > > [==================================================>] 24.95MB/24.95MB > > > Loaded image: apache/cloudstack-kubernetes-provider:v1.0.0 > > > 6a4a177e62f3: Loading layer > > > [==================================================>] 115.2kB/115.2kB > > > 398c9baff0ce: Loading layer > > > [==================================================>] 16.07MB/16.07MB > > > Loaded image: registry.k8s.io/coredns/coredns:v1.10.1 > > > bd8a70623766: Loading layer > > > [==================================================>] 75.78MB/75.78MB > > > c88361932af5: Loading layer > > > [==================================================>] 508B/508B > > > Loaded image: kubernetesui/dashboard:v2.7.0 > > > e023e0e48e6e: Loading layer > > > [==================================================>] 103.7kB/103.7kB > > > 6fbdf253bbc2: Loading layer > > > [==================================================>] 21.2kB/21.2kB > > > 7bea6b893187: Loading layer > > > [==================================================>] 716.5kB/716.5kB > > > ff5700ec5418: Loading layer > > > [==================================================>] 317B/317B > > > d52f02c6501c: Loading layer > > > [==================================================>] 198B/198B > > > e624a5370eca: Loading layer > > > [==================================================>] 113B/113B > > > 1a73b54f556b: Loading layer > > > [==================================================>] 385B/385B > > > d2d7ec0f6756: Loading layer > > > [==================================================>] 355B/355B > > > 4cb10dd2545b: Loading layer > > > [==================================================>] 130.6kB/130.6kB > > > ba9afb2b3e0c: Loading layer > > > [==================================================>] 1.076MB/1.076MB > > > 22bba3da6b0d: Loading layer > > > [==================================================>] 91.29MB/91.29MB > > > a4563151d59b: Loading layer > > > [==================================================>] 9.544MB/9.544MB > > > Loaded image: registry.k8s.io/etcd:3.5.9-0 > > > 54ad2ec71039: Loading layer > > > [==================================================>] 103.7kB/103.7kB > > > 6fbdf253bbc2: Loading layer > > > [==================================================>] 21.2kB/21.2kB > > > 7bea6b893187: Loading layer > > > [==================================================>] 716.5kB/716.5kB > > > ff5700ec5418: Loading layer > > > [==================================================>] 317B/317B > > > d52f02c6501c: Loading layer > > > [==================================================>] 198B/198B > > > e624a5370eca: Loading layer > > > [==================================================>] 113B/113B > > > 1a73b54f556b: Loading layer > > > [==================================================>] 385B/385B > > > d2d7ec0f6756: Loading layer > > > [==================================================>] 355B/355B > > > 4cb10dd2545b: Loading layer > > > [==================================================>] 130.6kB/130.6kB > > > 3f3c83f9a2fa: Loading layer > > > [==================================================>] 724.8kB/724.8kB > > > 6a2c4bedb8de: Loading layer > > > [==================================================>] 32.98MB/32.98MB > > > Loaded image: registry.k8s.io/kube-apiserver:v1.28.4 > > > e0b89915180d: Loading layer > > > [==================================================>] 31.72MB/31.72MB > > > Loaded image: registry.k8s.io/kube-controller-manager:v1.28.4 > > > d2a4b045f3d0: Loading layer > > > [==================================================>] 7.73MB/7.73MB > > > 3905b20a2df2: Loading layer > > > [==================================================>] 16.85MB/16.85MB > > > Loaded image: registry.k8s.io/kube-proxy:v1.28.4 > > > f35f4c1ae44f: Loading layer > > > [==================================================>] 17.13MB/17.13MB > > > Loaded image: registry.k8s.io/kube-scheduler:v1.28.4 > > > d01384fea991: Loading layer > > > [==================================================>] 19.74MB/19.74MB > > > bcec7eb9e567: Loading layer > > > [==================================================>] 530B/530B > > > Loaded image: kubernetesui/metrics-scraper:v1.0.8 > > > e3e5579ddd43: Loading layer > > > [==================================================>] 317.6kB/317.6kB > > > Loaded image: registry.k8s.io/pause:3.9 > > > 1b3ee35aacca: Loading layer > > > [==================================================>] 2.796MB/2.796MB > > > 910ce076f504: Loading layer > > > [==================================================>] 7.425MB/7.425MB > > > a8e8b7b8e08a: Loading layer > > > [==================================================>] 8.101MB/8.101MB > > > 084e56a9c24b: Loading layer > > > [==================================================>] 4.376MB/4.376MB > > > 334d70dc85ec: Loading layer > > > [==================================================>] 159B/159B > > > a13197d8bda5: Loading layer > > > [==================================================>] 8.216MB/8.216MB > > > Loaded image: weaveworks/weave-kube:2.8.1 > > > 998efb010df6: Loading layer > > > [==================================================>] 728.2kB/728.2kB > > > ed37391def99: Loading layer > > > [==================================================>] 9.28MB/9.28MB > > > 175a472c5f77: Loading layer > > > [==================================================>] 299B/299B > > > a8764e32e9fe: Loading layer > > > [==================================================>] 613B/613B > > > Loaded image: weaveworks/weave-npc:2.8.1 > > > net.bridge.bridge-nf-call-iptables = 1 > > > [init] Using Kubernetes version: v1.28.4 > > > [preflight] Running pre-flight checks > > > error execution phase preflight: [preflight] Some fatal errors > occurred: > > > [ERROR CRI]: container runtime is not running: output: > > > time="2024-04-11T15:33:26Z" level=fatal msg="validate service > connection: > > > validate CRI v1 runtime API for endpoint > > > \"unix:///var/run/containerd/containerd.sock\": rpc error: code = > > > Unimplemented desc = unknown service runtime.v1.RuntimeService" > > > , error: exit status 1 > > > [preflight] If you know what you are doing, you can make a check > non-fatal > > > with `--ignore-preflight-errors=...` > > > To see the stack trace of this error execute with --v=5 or higher > > > [init] Using Kubernetes version: v1.28.4 > > > [preflight] Running pre-flight checks > > > error execution phase preflight: [preflight] Some fatal errors > occurred: > > > [ERROR CRI]: container runtime is not running: output: > > > time="2024-04-11T15:33:26Z" level=fatal msg="validate service > connection: > > > validate CRI v1 runtime API for endpoint > > > \"unix:///var/run/containerd/containerd.sock\": rpc error: code = > > > Unimplemented desc = unknown service runtime.v1.RuntimeService" > > > , error: exit status 1 > > > [preflight] If you know what you are doing, you can make a check > non-fatal > > > with `--ignore-preflight-errors=...` > > > To see the stack trace of this error execute with --v=5 or higher > > > [init] Using Kubernetes version: v1.28.4 > > > [preflight] Running pre-flight checks > > > error execution phase preflight: [preflight] Some fatal errors > occurred: > > > [ERROR CRI]: container runtime is not running: output: > > > time="2024-04-11T15:33:27Z" level=fatal msg="validate service > connection: > > > validate CRI v1 runtime API for endpoint > > > \"unix:///var/run/containerd/containerd.sock\": rpc error: code = > > > Unimplemented desc = unknown service runtime.v1.RuntimeService" > > > , error: exit status 1 > > > [preflight] If you know what you are doing, you can make a check > non-fatal > > > with `--ignore-preflight-errors=...` > > > To see the stack trace of this error execute with --v=5 or higher > > > Error: kubeadm init failed! > > > > > > > > > > > > > > > > > > Le jeu. 11 avr. 2024 à 17:19, Wei ZHOU <ustcweiz...@gmail.com> a > écrit : > > > > > >> Hi, > > >> > > >> Please refer to > > >> > https://github.com/apache/cloudstack/issues/8681#issuecomment-1999083241 > > >> > > >> The containerd in 4.16/4.17 sysyemvm template is too old. > > >> > > >> -Wei > > >> > > >> > > >> On Thursday, April 11, 2024, benoit lair <kurushi4...@gmail.com> > wrote: > > >> > > >> > Hi Rohit, > > >> > > > >> > I already tested with different iso from cloudstack from CKS repos, > > >> > It is ok with ACS 4.16 with community iso 1.23.3 but it fails from > > >> 1.24.0 > > >> > and following others versions > > >> > > > >> > I tried to do an upgrade from 1.23.3 with 1.24.0, it fails, > > >> > I tried to bootstrap a k8s cluster with 1.24.0, also with 1.25 or > 1.27.3 > > >> > and 1.28.4 > > >> > > > >> > Last try, i tested k8s community iso 1.25.0 > > >> > in CS UI it saying "Create Kubernetes Cluster k8s-cks-cl16 in > > >> progress" > > >> > and it fails > > >> > If i try to ssh on control node with VR_Pub_IP port 2222 > > >> > i have nothing in execution > > >> > > > >> > If i try manual install : > > >> > core@k8s-cks-cl16-control-18eccd0c12c:~$ sudo > > >> /opt/bin/setup-kube-system > > >> > mount: /mnt/k8sdisk: /dev/sr0 already mounted on /mnt/k8sdisk. > > >> > Installing binaries from /mnt/k8sdisk/ > > >> > 5b1fa8e3e100: Loading layer > > >> > [==================================================>] > 803.8kB/803.8kB > > >> > 39c831b1aa26: Loading layer > > >> > [==================================================>] > 26.25MB/26.25MB > > >> > Loaded image: apache/cloudstack-kubernetes-autoscaler:latest > > >> > 417cb9b79ade: Loading layer > > >> > [==================================================>] > 657.7kB/657.7kB > > >> > 8d323b160d65: Loading layer > > >> > [==================================================>] > 24.95MB/24.95MB > > >> > Loaded image: apache/cloudstack-kubernetes-provider:v1.0.0 > > >> > 256bc5c338a6: Loading layer > > >> > [==================================================>] > 122.2kB/122.2kB > > >> > df1818f16337: Loading layer > > >> > [==================================================>] > 14.71MB/14.71MB > > >> > Loaded image: registry.k8s.io/coredns/coredns:v1.9.3 > > >> > bd8a70623766: Loading layer > > >> > [==================================================>] > 75.78MB/75.78MB > > >> > c88361932af5: Loading layer > > >> > [==================================================>] 508B/508B > > >> > Loaded image: kubernetesui/dashboard:v2.7.0 > > >> > 0b031aac6569: Loading layer > > >> > [==================================================>] 804kB/804kB > > >> > cc0597072b83: Loading layer > > >> > [==================================================>] > 1.076MB/1.076MB > > >> > 5a8045f731fc: Loading layer > > >> > [==================================================>] > 90.58MB/90.58MB > > >> > f5a5ef1a005f: Loading layer > > >> > [==================================================>] > 1.188MB/1.188MB > > >> > 5cf3bfe5417d: Loading layer > > >> > [==================================================>] > 8.506MB/8.506MB > > >> > Loaded image: registry.k8s.io/etcd:3.5.4-0 > > >> > 8d7366c22fd8: Loading layer > > >> > [==================================================>] > 804.1kB/804.1kB > > >> > 7d5f5bba7c45: Loading layer > > >> > [==================================================>] > 694.9kB/694.9kB > > >> > 8154c9483a6c: Loading layer > > >> > [==================================================>] > 32.72MB/32.72MB > > >> > Loaded image: registry.k8s.io/kube-apiserver:v1.25.0 > > >> > c5f1a47b47bc: Loading layer > > >> > [==================================================>] > 29.76MB/29.76MB > > >> > Loaded image: registry.k8s.io/kube-controller-manager:v1.25.0 > > >> > e4d19dd12df2: Loading layer > > >> > [==================================================>] > 7.631MB/7.631MB > > >> > ba3637ccd78a: Loading layer > > >> > [==================================================>] > 12.63MB/12.63MB > > >> > Loaded image: registry.k8s.io/kube-proxy:v1.25.0 > > >> > 026969d859f7: Loading layer > > >> > [==================================================>] > 14.29MB/14.29MB > > >> > Loaded image: registry.k8s.io/kube-scheduler:v1.25.0 > > >> > d01384fea991: Loading layer > > >> > [==================================================>] > 19.74MB/19.74MB > > >> > bcec7eb9e567: Loading layer > > >> > [==================================================>] 530B/530B > > >> > Loaded image: kubernetesui/metrics-scraper:v1.0.8 > > >> > 961e93cda9dd: Loading layer > > >> > [==================================================>] 307kB/307kB > > >> > Loaded image: registry.k8s.io/pause:3.8 > > >> > 1b3ee35aacca: Loading layer > > >> > [==================================================>] > 2.796MB/2.796MB > > >> > 17d89e2a5199: Loading layer > > >> > [==================================================>] > 7.425MB/7.425MB > > >> > 148b94859892: Loading layer > > >> > [==================================================>] 8.1MB/8.1MB > > >> > dbea472c1ced: Loading layer > > >> > [==================================================>] > 4.376MB/4.376MB > > >> > e81f1a367c17: Loading layer > > >> > [==================================================>] 131B/131B > > >> > 351848f6b295: Loading layer > > >> > [==================================================>] > 8.216MB/8.216MB > > >> > Loaded image: weaveworks/weave-kube:latest > > >> > b20f7ff12b24: Loading layer > > >> > [==================================================>] > 727.9kB/727.9kB > > >> > 1d49e386162e: Loading layer > > >> > [==================================================>] > 9.281MB/9.281MB > > >> > 20145eeac146: Loading layer > > >> > [==================================================>] 297B/297B > > >> > d4cc72748587: Loading layer > > >> > [==================================================>] 600B/600B > > >> > Loaded image: weaveworks/weave-npc:latest > > >> > net.bridge.bridge-nf-call-iptables = 1 > > >> > [init] Using Kubernetes version: v1.25.0 > > >> > [preflight] Running pre-flight checks > > >> > [WARNING SystemVerification]: missing optional cgroups: > blkio > > >> > error execution phase preflight: [preflight] Some fatal errors > occurred: > > >> > [ERROR CRI]: container runtime is not running: output: E0411 > > >> > 11:38:11.935458 12988 remote_runtime.go:948] "Status from runtime > > >> service > > >> > failed" err="rpc error: code = Unimplemented desc = unknown service > > >> > runtime.v1alpha2.RuntimeService" > > >> > time="2024-04-11T11:38:11Z" level=fatal msg="getting status of > runtime: > > >> rpc > > >> > error: code = Unimplemented desc = unknown service > > >> > runtime.v1alpha2.RuntimeService" > > >> > , error: exit status 1 > > >> > [preflight] If you know what you are doing, you can make a check > > >> non-fatal > > >> > with `--ignore-preflight-errors=...` > > >> > To see the stack trace of this error execute with --v=5 or higher > > >> > [init] Using Kubernetes version: v1.25.0 > > >> > [preflight] Running pre-flight checks > > >> > [WARNING SystemVerification]: missing optional cgroups: > blkio > > >> > error execution phase preflight: [preflight] Some fatal errors > occurred: > > >> > [ERROR CRI]: container runtime is not running: output: E0411 > > >> > 11:38:12.079422 13015 remote_runtime.go:948] "Status from runtime > > >> service > > >> > failed" err="rpc error: code = Unimplemented desc = unknown service > > >> > runtime.v1alpha2.RuntimeService" > > >> > time="2024-04-11T11:38:12Z" level=fatal msg="getting status of > runtime: > > >> rpc > > >> > error: code = Unimplemented desc = unknown service > > >> > runtime.v1alpha2.RuntimeService" > > >> > , error: exit status 1 > > >> > [preflight] If you know what you are doing, you can make a check > > >> non-fatal > > >> > with `--ignore-preflight-errors=...` > > >> > To see the stack trace of this error execute with --v=5 or higher > > >> > [init] Using Kubernetes version: v1.25.0 > > >> > [preflight] Running pre-flight checks > > >> > [WARNING SystemVerification]: missing optional cgroups: > blkio > > >> > error execution phase preflight: [preflight] Some fatal errors > occurred: > > >> > [ERROR CRI]: container runtime is not running: output: E0411 > > >> > 11:38:12.228669 13043 remote_runtime.go:948] "Status from runtime > > >> service > > >> > failed" err="rpc error: code = Unimplemented desc = unknown service > > >> > runtime.v1alpha2.RuntimeService" > > >> > time="2024-04-11T11:38:12Z" level=fatal msg="getting status of > runtime: > > >> rpc > > >> > error: code = Unimplemented desc = unknown service > > >> > runtime.v1alpha2.RuntimeService" > > >> > , error: exit status 1 > > >> > [preflight] If you know what you are doing, you can make a check > > >> non-fatal > > >> > with `--ignore-preflight-errors=...` > > >> > To see the stack trace of this error execute with --v=5 or higher > > >> > Error: kubeadm init failed! > > >> > > > >> > > > >> > Le jeu. 11 avr. 2024 à 12:03, Rohit Yadav < > rohit.ya...@shapeblue.com> a > > >> > écrit : > > >> > > > >> > > Hi Benoit, > > >> > > > > >> > > The CKS feature has been improving over versions, I don't know if > what > > >> > > you're trying to achieve is possible with it. Maybe try a > different > > >> > version > > >> > > of the data iso: > > >> > > http://download.cloudstack.org/cks/ > > >> > > > > >> > > Alternatively, you can also have a look at the CAPC project: > > >> > > https://cluster-api-cloudstack.sigs.k8s.io > > >> > > > > >> > > > > >> > > Regards. > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > ________________________________ > > >> > > From: benoit lair <kurushi4...@gmail.com> > > >> > > Sent: Thursday, April 11, 2024 14:16 > > >> > > To: us...@cloudstack.apache.org <us...@cloudstack.apache.org>; > dev < > > >> > > dev@cloudstack.apache.org> > > >> > > Subject: Re: ACS 4.16 - Change SystemVM template for CKS > > >> > > > > >> > > Any advices ? > > >> > > > > >> > > Best regards > > >> > > > > >> > > Le lun. 8 avr. 2024 à 16:53, benoit lair <kurushi4...@gmail.com> > a > > >> > écrit : > > >> > > > > >> > > > I am opened to every alternatives of changing system vm > templates > > >> > > > I just need to run K8s clyusters 1.28 with my CS 4.16 :) > > >> > > > > > >> > > > Le lun. 8 avr. 2024 à 16:52, benoit lair <kurushi4...@gmail.com> > a > > >> > > écrit : > > >> > > > > > >> > > >> Hello Folks, > > >> > > >> > > >> > > >> I am trying to install K8s cluster with community iso 1.28.4 > > >> > > >> I am with a ACS 4.16.0 environment > > >> > > >> It seems K8S is not working out of the box with v > 1.23.3 due > to > > >> > > >> containerd.io version > > >> > > >> > > >> > > >> I would like to release a template who will work with K8s > > 1.23.3 > > >> on > > >> > > acs > > >> > > >> 4.16 > > >> > > >> How can i tell CS to take another system vm template, avoiding > to > > >> mess > > >> > > >> normal features with VR and VPC VR keeping the systemvm > template > > >> > issued > > >> > > >> with CS 4.16 ? > > >> > > >> > > >> > > >> I am with XCP-NG 8.2.1 in production > > >> > > >> > > >> > > >> Thanks for your help or advises > > >> > > >> Best regards, > > >> > > >> Benoit > > >> > > >> > > >> > > > > > >> > > > > >> > > > >> > > > >
