shwstppr opened a new pull request #3680: [WIP: DO NOT MERGE] CloudStack Kubernetes Service URL: https://github.com/apache/cloudstack/pull/3680 ## Description <!--- Describe your changes in detail --> Specification document https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Kubernetes+Service Kubernetes Service plugin adds Kubernetes integration in the CloudStack. Plugin is disabled by default and an admin can enable it using the Global Setting. It enables users to run containerized services using Kubernetes clusters. Kubernetes Service plugin uses a CoreOS based template for node VMs for the Kubernetes cluster. For installation of Kubernetes binaries on cluster nodes, a binaries ISO can be created for a particular Kubernetes version and can be added as a supported version by an admin. This allows faster, offline installation Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters. For deployment and setup of Kubernetes on cluster nodes, the plugin uses the Kubernetes tool, kubeadm. kubeadm is the command-line tool for easily provisioning a secure Kubernetes cluster on top of physical or cloud servers or virtual machines. Under the hood, master node(s) of the cluster starts a Kubernetes cluster using kubeadm init command with a custom token and worker nodes join this Kubernetes cluster using kubeadm join command with the same token. More about kubeadm, https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/. Weave Net CNI provider plugin is used for cluster netoworking. More about Weave Net provide plugin, https://www.weave.works/docs/net/latest/kubernetes/kube-addon/. To access Kubernetes dashboard securely, the plugin provides access to kubeconfig file data which allows using the Kubernetes tool, kubectl, to run proxy locally and thereby access dashboard. More about kubectl, https://kubernetes.io/docs/reference/kubectl/overview/ The service allows creation of Kubernetes clusters using UI or API. Both UI and API functionalities to list, delete, scale upgrade, stop and start these clusters. Enabling the Kubernetes Service Kubernetes Service plugin is disabled by default. To enable it, go to Global Settings and set the following global configuration to true: `cloud.kubernetes.service.enabled` Restart the Management Server to enable the set configuration values. `service cloudstack-management restart` Once the Kubernetes service is running new API will become accessible and UI will start show Kubernetes Service tab. **Kubernetes Supported Versions** For faster and offline installation of Kubernetes and docker images on node virtual machines of a Kubernetes cluster, the service provides the functionality to manage supported Kubernetes versions. Script named create-kubernetes-binaries-iso.sh has been provided in the cloudstack-common package for creating a new setup iso with the desired version of Kubernetes binaries and corresponding docker images. Usage, `./create-kubernetes-binaries-iso.sh OUTPUT_PATH KUBERNETES_VERSION CNI_VERSION CRICTL_VERSION WEAVENET_NETWORK_YAML_CONFIG DASHBOARD_YAML_CONFIG` eg, `./create-binaries-iso.sh ./ 1.12.5 0.7.1 1.12.0 "https://cloud.weave.works/k8s/net?k8s-version=1.12.5"; https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta1/aio/deploy/recommended.yaml` ISOs created using the script can be added as supported Kubernetes version for Kubernetes service. _Working with Kubernetes supported version_ New Kubernetes versions can be added using both UI and API. UI provides the following form to add new supported version, `addKubernetesSupportedVersion` API can be used by an admin to add a new supported version for the service. It takes following input parameters, ``` name (the name of the Kubernetes supported version) semanticversion (the semantic version of the Kubernetes; Required) zoneid (the ID of the zone in which Kubernetes supported version will be available) isoid (the ID of the binaries ISO for Kubernetes supported version) url (the URL of the binaries ISO for Kubernetes supported version) checksum (the checksum value of the binaries ISO) ``` ``` > add kubernetessupportedversion name=v1.13.2 semanticversion=1.13.2 url=http://172.20.0.1/files/setup-1.13.2.iso zoneid=34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6 { "kubernetessupportedversion": { "id": "6668e999-fe6c-4a91-88d8-d10bcf280d02", "isoid": "78d45e9b-a482-46f4-8cbc-cf7964564b85", "isoname": "v1.13.2-Kubernetes-Binaries-ISO", "isostate": "Active", "semanticversion": "1.13.2", "name": "v1.13.2", "supportsha": false, "zoneid": "34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6", "zonename": "KVM-advzone1" } } ``` Note: The minimum Kubernetes version that can be added in the service is **1.11**. `listKubernetesSupportedVersion` API can be used to list existing supported versions. It takes id parameter as input to list details of a single supported version. `deleteKubernetesSupportedVersion` API has been provided for admins to delete an existing supported version if it is not used by any Kubernetes cluster in the service. id parameter of the API can be used to pass Kubernetes version to be deleted. Note: `addKubernetesSupportedVersion` and `deleteKubernetesSupportedVersion` API are available to admin only **Kubernetes clusters** Kubernetes service provides the functionality of running and managing Kubernetes clusters. Highly available, scalable Kubernetes clusters can be created to run containerized deployments without having trouble to set up Kubernetes on each container node manually. Service will automatically provision the desired number of virtual machines as per cluster size using the binaries from the given Kubernetes version. Additionally, the service provides the functionality to upgrade and scale clusters. Running clusters can be upgraded to a newer minor or patch Kubernetes version at a time. Running clusters can also be scaled for the number of worker nodes up and down and for the service offering used by each node. The provides functionality to create Kubernetes clusters for Shared, Isolated and VPC networks in the CloudStack but such networks must be accessible to CloudStack management server for provisioning virtual machines of the cluster. Template and default network offering must be set Global Settings for the service to create Kubernetes clusters. Following Global Setting value must be set to the name of Template to be used for deploying node virtual machines while creating a Kubernetes cluster, `cloud.kubernetes.cluster.template.name` CoreOS templates for CloudStack can be found here, http://dl.openvm.eu/cloudstack/coreos/x86_64/ Following Global Setting value must be set to the name of Network Offering to be used for creating a new network when no network has been selected while creating a Kubernetes cluster, `cloud.kubernetes.cluster.network.offering` A new network offering named DefaultNetworkOfferingforKubernetesService has been added since 4.14.0 Note: - Multi-master, HA cluster can be created for Kubernetes version 1.16 and above only. - While creating multi-master, HA cluster over a shared network, an external load-balancer must be manually setup. This load-balancer should have port-forwarding rules for SSH, Kubernetes API server access. Service assumes SSH access to cluster nodes is available from port 2222 to (2222 + cluster node count -1). Similarly, for API access 6443 must be forwarded to master nodes. Over CloudStack isolated network these rules are automatically provisioned. _Managing Kubernetes clusters_ For Kubernetes cluster management, service provides create, stop, start, scale, upgrade and delete APIs and similar features in the UI. APIs available, `createKubernetesCluster` ``` > create kubernetescluster name=Test description=Test-Cluster zoneid=34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6 size=1 noderootdisksize=10 serviceofferingid=a4f280a1-9122-40a8-8f0c-3adb91060f2a kubernetesversionid=6668e999-fe6c-4a91-88d8-d10bcf280d02 { "kubernetescluster": { "associatednetworkname": "Test-network", "cpunumber": "4", "description": "Test-Cluster", "endpoint": "https://172.20.20.12:6443/";, "id": "74e3cc02-bbf7-438f-bfb0-9c193e90c1fb", "kubernetesversionid": "6668e999-fe6c-4a91-88d8-d10bcf280d02", "kubernetesversionname": "v1.13.2", "masternodes": 1, "memory": "4096", "name": "Test", "networkid": "148af2cb-4b94-42a2-b701-3b6aa884cbb0", "serviceofferingid": "a4f280a1-9122-40a8-8f0c-3adb91060f2a", "serviceofferingname": "CKS Instance", "size": 1, "state": "Running", "templateid": "17607ed6-1756-4ed7-b0f4-dbab5feff5b2", "virtualmachineids": [ "da2cb67e-e852-4ecd-b16f-a8f16eb2c962", "4179864a-88ad-4d6d-890c-c9b73c53589b" ], "zoneid": "34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6", "zonename": "KVM-advzone1" } } ``` `stopKubernetesCluster` can be used to stop a running cluster. It takes id of the cluster as the input parameter. `startKubernetesCluster` can be used to start a stopped cluster. It takes id of the cluster as the input parameter. `scaleKubernetesCluster` can be used to scale a running or stopped cluster. Note: Only upscaling is supported while scaling clusters for service offering. Olny a running Kubernetes cluster can be scaled for size. `upgradeKubernetesCluster` can be used to upgrade a running cluster. `deleteKubernetesCluster` can be used to delete a cluster. It takes id of the cluster as the input parameter. `getKubernetesClusterConfig` API can be used to retrieve kubeconfig file data for a cluster. It takes id of the cluster as the input parameter. <!-- For new features, provide link to FS, dev ML discussion etc. --> <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. --> <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged --> <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" --> <!-- Fixes: # --> ## Types of changes <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) ## Screenshots (if appropriate):          ## How Has This Been Tested? <!-- Please describe in detail how you tested your changes. --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
