I have evaluated to the best of my ability: - Rancher + RancherOS - Rancher + CoreOS - CoreOS Kubernetes (Manual installation) - CoreOS Kubernetes (Tectonic) - CoreOS SelfHosted (Bootkube+matchbox)
Quick Summary: With RancherOS my experience is that you have to be careful to install a OS version with a compatilbe docker for kubernetes. I also dont like to be locked into docker, prefer systemd as an init system as the rancherOS init system, in comparison, is limited. Rancher + CoreOS installation experience is the best. However only 1.5 is supported. 1.6 is not yet released and i could not get a cluster started using there development branch. Also for advanced K8 users, advanced customization of the installation process I found difficult (e.g. custom CA for certificate authentication, etc.). In order to make the provisioning process smooth you will want to use CoreOS matchbox. CoreOS manual installation is excellent however it does not explain how to provision the systemd units onto the machines. I am a big CoreOS fleet fan so I initially created fleet units to deploy the documented systemd units. CoreOS has now deprecated Fleet so this approach is not also not recommended. Also the documentation can sometime be out of date. CoreOS bootkube+matchbox I also struggled with CoreOS tectonic but ironically am very happy with bootkube+matchbox on CoreOS (which is what tectonic is based on). I would highly recommend it. https://github.com/coreos/matchbox/blob/master/Documentation/bootkube.md I would also recommend bootkube over CoreOS manual installation as the documentation is more likely to be maintained and up to date as it is the basis for tectonic. More interesting is that bootkube gives you access to features such as "self hosted etcd" where etcd is deployed into kubernetes itsself. With bootkube+matchbox I am able to iPXE boot a bare metal server kubernetes master in 5 minutes with no manual intervention. New workers are also automatcially joined with no manual intervention simply from an iPXE boot. The advantage of the iPXE approach is also that you can seemlessly test it using virtualization and the process is the same (there are some small issue like network adapters taking more time to come online on bare metal but in general i dont have to make any changes). One strange part of the boot-kube flow is having to ssh on to each machine and "scp" certificates and kubeconfig. I extended the matchbox bootkube-install example to dynamically download all required assets such that no manual intervention is required. This is one of the advantages of the CoreOS ignition process where you can setup indirection to download all your kubernetes assets from a remote server. Some examples scripts below: #generated bootkube config bootkube render --asset-dir=assets --api-servers=https://192.168.122.10:443 --api-server-alt-names=DNS=k8.example.com,IP=192.168.122.10 --etcd-servers=https://192.168.122.10:2379 --experimental-self-hosted-etcd #start matchbox ipxe server docker run -p 8080:8080 --rm -v $PWD/examples:/var/lib/matchbox:Z -v $PWD/examples/groups/bootkube-install:/var/lib/matchbox/groups:Z quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug #examples/groups/bootkube-install/node1.json extended for dynamic bootkube { "id": "node1", "name": "Controller Node", "profile": "bootkube-controller", "selector": { "mac": "18:03:73:3f:6d:bf", "os": "installed" }, "metadata": { "cluster_id" : "test_agent", "config_server": "http://172.20.20.219:1979/k8/test_agent";, "ip_address" : "192.168.122.10", "ip_gateway" : "192.168.122.1", "net_mask" : "23", "domain_name": "k8.example.com", "etcd_initial_cluster": "k8=https://192.168.122.10:2380";, "etcd_endpoints": "https://192.168.122.10:2379";, "etcd_name": "k8", "k8s_dns_service_ip": "10.3.0.10", "ssh_authorized_keys": [ "ssh-rsa ..." ] } } #examples/ignition/bootkube-controller.yaml extended for dynamic bootkube networkd: units: - name: 00-static.network contents: | [Match] Name=e* [Network] DNS=8.8.8.8 Address={{.ip_address}}/{{.net_mask}} Gateway={{.ip_gateway}} systemd: units: - name: bootkube.service enable: true contents: | [Unit] Description=Bootstrap a Kubernetes control plane with a temp api-server [Service] Type=simple RemainAfterExit=yes Restart=always RestartSec=20 WorkingDirectory=/opt/bootkube ExecStartPre=/bin/sh -c 'wget -r -np -nH -x --cut-dirs=1 {{.config_server}}/' ExecStartPre=/bin/sh -c 'mv {{.cluster_id}} /opt/bootkube/assets' ExecStart=/opt/bootkube/bootkube-start [Install] WantedBy=multi-user.target storage: files: - path: /etc/ssl/etcd/etcd-client.crt filesystem: root mode: 0644 contents: remote: url: "{{.config_server}}/tls/etcd-client.crt" On Tuesday, June 27, 2017 at 7:06:48 AM UTC+2, luce...@gmail.com wrote: > > Using Kubernetes 1.6 on Bar Metal servers (production) seems to be > difficult. > > Tested CoreOs and Tectonic, not working on Bar Metal servers. > Bar Metal servers and conjure-up kubernetes not possible. (+ Maas) > > Rancher is working but the whole system is based on the docker daemon, > systemd is a better choice as base system for Kubernetes. (Like CoreOs does) > > > A possible solution is kubeadm but kubeadm is not ready for production! > > Are there other possibilities? > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
