Thank you for your post. I too am trying different methods/stacks as well to pxe boot a 42u bare metal rack into a cluster. os running in ram of course. what was your experience with Rancher and core os together? Do you recommend Bootkube+matchbox over coreosmatchbox/ rancher? I like ranchers extra environment / name spaces, acl web interface, logging and cli access. However, I dont know how to automate creating and joining the cluster. it appears rancher was primarily designed for a hard disk install, and manual cluster creation. Could this be automatied? or is it just not worth it.
(I already use pxe matchbox /ignition / coreos so maybe I should dump rancher and run bootkube) Thoughts? On Wednesday, June 28, 2017 at 4:55:24 AM UTC-5, mrpanigale wrote: > I have evaluated to the best of my ability: > - Rancher + RancherOS > - Rancher + CoreOS > - CoreOS Kubernetes (Manual installation) > - CoreOS Kubernetes (Tectonic) > > - CoreOS SelfHosted (Bootkube+matchbox) > > > Quick Summary: > With RancherOS my experience is that you have to be careful to install a OS > version with a compatilbe docker for kubernetes. I also dont like to be > locked into docker, prefer systemd as an init system as the rancherOS init > system, in comparison, is limited. > > > Rancher + CoreOS installation experience is the best. However only 1.5 is > supported. 1.6 is not yet released and i could not get a cluster started > using there development branch. Also for advanced K8 users, advanced > customization of the installation process I found difficult (e.g. custom CA > for certificate authentication, etc.). In order to make the provisioning > process smooth you will want to use CoreOS matchbox. > > > CoreOS manual installation is excellent however it does not explain how to > provision the systemd units onto the machines. I am a big CoreOS fleet fan so > I initially created fleet units to deploy the documented systemd units. > CoreOS has now deprecated Fleet so this approach is not also not recommended. > Also the documentation can sometime be out of date. > > > CoreOS bootkube+matchboxI also struggled with CoreOS tectonic but ironically > am very happy with bootkube+matchbox on CoreOS (which is what tectonic is > based on). I would highly recommend it. > https://github.com/coreos/matchbox/blob/master/Documentation/bootkube.md > I would also recommend bootkube over CoreOS manual installation as the > documentation is more likely to be maintained and up to date as it is the > basis for tectonic. > More interesting is that bootkube gives you access to features such as "self > hosted etcd" where etcd is deployed into kubernetes itsself. > > > > With bootkube+matchbox I am able to iPXE boot a bare metal server kubernetes > master in 5 minutes with no manual intervention. New workers are also > automatcially joined with no manual intervention simply from an iPXE boot. > The advantage of the iPXE approach is also that you can seemlessly test it > using virtualization and the process is the same (there are some small issue > like network adapters taking more time to come online on bare metal but in > general i dont have to make any changes). > > > One strange part of the boot-kube flow is having to ssh on to each machine > and "scp" certificates and kubeconfig. > I extended the matchbox bootkube-install example to dynamically download all > required assets such that no manual intervention is required. This is one of > the advantages of the CoreOS ignition process where you can setup indirection > to download all your kubernetes assets from a remote server. > > > Some examples scripts below: > > > #generated bootkube config > bootkube render --asset-dir=assets --api-servers=https://192.168.122.10:443 > --api-server-alt-names=DNS=k8.example.com,IP=192.168.122.10 > --etcd-servers=https://192.168.122.10:2379 --experimental-self-hosted-etcd > > > > #start matchbox ipxe server > docker run -p 8080:8080 --rm -v $PWD/examples:/var/lib/matchbox:Z -v > $PWD/examples/groups/bootkube-install:/var/lib/matchbox/groups:Z > quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug > > > > #examples/groups/bootkube-install/node1.json extended for dynamic bootkube > > { > "id": "node1", > "name": "Controller Node", > "profile": "bootkube-controller", > "selector": { > "mac": "18:03:73:3f:6d:bf", > "os": "installed" > }, > "metadata": { > "cluster_id" : "test_agent", > "config_server": "http://172.20.20.219:1979/k8/test_agent";, > "ip_address" : "192.168.122.10", > "ip_gateway" : "192.168.122.1", > "net_mask" : "23", > "domain_name": "k8.example.com", > "etcd_initial_cluster": "k8=https://192.168.122.10:2380";, > "etcd_endpoints": "https://192.168.122.10:2379";, > "etcd_name": "k8", > "k8s_dns_service_ip": "10.3.0.10", > "ssh_authorized_keys": [ > "ssh-rsa ..." ] > } > } > > > > > #examples/ignition/bootkube-controller.yaml extended for dynamic bootkube > > networkd: > units: > - name: 00-static.network > contents: | > [Match] > Name=e* > [Network] > DNS=8.8.8.8 > Address={{.ip_address}}/{{.net_mask}} > Gateway={{.ip_gateway}} > systemd: > units: > - name: bootkube.service > enable: true > contents: | > [Unit] > Description=Bootstrap a Kubernetes control plane with a temp > api-server > [Service] > Type=simple > RemainAfterExit=yes > Restart=always > RestartSec=20 > WorkingDirectory=/opt/bootkube > ExecStartPre=/bin/sh -c 'wget -r -np -nH -x --cut-dirs=1 > {{.config_server}}/' > ExecStartPre=/bin/sh -c 'mv {{.cluster_id}} /opt/bootkube/assets' > ExecStart=/opt/bootkube/bootkube-start > [Install] > WantedBy=multi-user.target > > storage: > files: > - path: /etc/ssl/etcd/etcd-client.crt > filesystem: root > mode: 0644 > contents: > remote: > url: "{{.config_server}}/tls/etcd-client.crt" > > > On Tuesday, June 27, 2017 at 7:06:48 AM UTC+2, luce...@gmail.com wrote:Using > Kubernetes 1.6 on Bar Metal servers (production) seems to be difficult. > Tested CoreOs and Tectonic, not working on Bar Metal servers. > Bar Metal servers and conjure-up kubernetes not possible. (+ Maas) > Rancher is working but the whole system is based on the docker daemon, > systemd is a better choice as base system for Kubernetes. (Like CoreOs does) > > A possible solution is kubeadm but kubeadm is not ready for production! > Are there other possibilities? -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
