Hey z3ro, My current position is that I would recommend PXE+matchbox+bootkube. We have been working with this deployment approach very successfully since my last post and all our internal clusters are deployed using iPXE.
Ultimately the following issues where deciding factors against Rancher: - Rancher deploys kubernetes using a technology called Cattle. This adds additional complexity and knowledge requirements especially when troubleshooting (think having to learn Puppet/Ansible/Chef + OpenStack neutron networking in addition). We preferred the bootkube "self hosted kubernetes" approach as deploying and upgrading kubernetes was a consistent experience to deploying our business applications. - Rancher's approach to ingress, in my oppinion, is inflexible. For example, it was very difficult to impossible to expose a "hostNetwork=true" running container or deploy a custom ingress controller. This is very important when planning for the future and reducing vendor lock-in. For example, we are already evaluating istio.io technology and maintaining some kind of flexibilty at the networking layer is important. Ultimately, I think, you would want to be able to solve problems in multiple ways if you need to and Rancher is not flexible enough. Again all my opinion. - We love CEPH, StorageClasses and PVCs. Unfortunately, integration with Rancher was not possible because it was not possible to extend the Rancher deployment scripts to mount additional kernel modules required for CEPH integration. One of our engineers after two months was able to fork the Rancher kubernetes catalog and somehow get this working but this means more maintenance merging updates from the Rancher community. It also highlighted the steps/complexity required to customize the deployment process, if needed. - Compared to the matchbox + bootkube approach, deployment customization is straight forward and after some time you gain more confidence. We ended up pulling upstream updates before bootkube or matchbox performed official releases. We simply monitor this URL for new hyperkube releases and test the new kubernetes images as they arrive. https://quay.io/repository/coreos/hyperkube?tab=tags - We love Network policy. Unfortunately we were not able to deploy Project Calico within Rancher for similar reasons to not being able to deploy CEPH/PVC integration. Network policy is important for us as we use the security features, lucky for us bootkube has a flag to generate Calico deployment assets and the deployment process is painless - Security patches and existing CoreOS experience. Our company has already been using CoreOS in production with fleet since 2014. Keeing things CoreOS allows us to take advantage of the automated security updates operator integration with kubernetes (https://github.com/coreos/container-linux-update-operator). Unfortunately we were not able to find recommendations for patching within a Rancher cluster kind regards, Andrew On Wednesday, September 20, 2017 at 4:59:59 PM UTC+2, z3ro wrote: > > Thank you for your post. > > I too am trying different methods/stacks as well to pxe boot a 42u bare > metal rack into a cluster. os running in ram of course. what was your > experience with Rancher and core os together? Do you recommend > Bootkube+matchbox over coreosmatchbox/ rancher? I like ranchers extra > environment / name spaces, acl web interface, logging and cli access. > However, I dont know how to automate creating and joining the cluster. it > appears rancher was primarily designed for a hard disk install, and manual > cluster creation. Could this be automatied? or is it just not worth it. > > (I already use pxe matchbox /ignition / coreos so maybe I should dump > rancher and run bootkube) > > Thoughts? > > > > On Wednesday, June 28, 2017 at 4:55:24 AM UTC-5, mrpanigale wrote: > > I have evaluated to the best of my ability: > > - Rancher + RancherOS > > - Rancher + CoreOS > > - CoreOS Kubernetes (Manual installation) > > - CoreOS Kubernetes (Tectonic) > > > > - CoreOS SelfHosted (Bootkube+matchbox) > > > > > > Quick Summary: > > With RancherOS my experience is that you have to be careful to install a > OS version with a compatilbe docker for kubernetes. I also dont like to be > locked into docker, prefer systemd as an init system as the rancherOS init > system, in comparison, is limited. > > > > > > Rancher + CoreOS installation experience is the best. However only 1.5 > is supported. 1.6 is not yet released and i could not get a cluster started > using there development branch. Also for advanced K8 users, advanced > customization of the installation process I found difficult (e.g. custom CA > for certificate authentication, etc.). In order to make the provisioning > process smooth you will want to use CoreOS matchbox. > > > > > > CoreOS manual installation is excellent however it does not explain how > to provision the systemd units onto the machines. I am a big CoreOS fleet > fan so I initially created fleet units to deploy the documented systemd > units. CoreOS has now deprecated Fleet so this approach is not also not > recommended. Also the documentation can sometime be out of date. > > > > > > CoreOS bootkube+matchboxI also struggled with CoreOS tectonic but > ironically am very happy with bootkube+matchbox on CoreOS (which is what > tectonic is based on). I would highly recommend it. > https://github.com/coreos/matchbox/blob/master/Documentation/bootkube.md > > I would also recommend bootkube over CoreOS manual installation as the > documentation is more likely to be maintained and up to date as it is the > basis for tectonic. > > More interesting is that bootkube gives you access to features such as > "self hosted etcd" where etcd is deployed into kubernetes itsself. > > > > > > > > With bootkube+matchbox I am able to iPXE boot a bare metal server > kubernetes master in 5 minutes with no manual intervention. New workers are > also automatcially joined with no manual intervention simply from an iPXE > boot. The advantage of the iPXE approach is also that you can seemlessly > test it using virtualization and the process is the same (there are some > small issue like network adapters taking more time to come online on bare > metal but in general i dont have to make any changes). > > > > > > One strange part of the boot-kube flow is having to ssh on to each > machine and "scp" certificates and kubeconfig. > > I extended the matchbox bootkube-install example to dynamically download > all required assets such that no manual intervention is required. This is > one of the advantages of the CoreOS ignition process where you can setup > indirection to download all your kubernetes assets from a remote server. > > > > > > Some examples scripts below: > > > > > > #generated bootkube config > > bootkube render --asset-dir=assets --api-servers= > https://192.168.122.10:443 > --api-server-alt-names=DNS=k8.example.com,IP=192.168.122.10 > --etcd-servers=https://192.168.122.10:2379 > --experimental-self-hosted-etcd > > > > > > > > #start matchbox ipxe server > > docker run -p 8080:8080 --rm -v $PWD/examples:/var/lib/matchbox:Z -v > $PWD/examples/groups/bootkube-install:/var/lib/matchbox/groups:Z > quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug > > > > > > > > #examples/groups/bootkube-install/node1.json extended for dynamic > bootkube > > > > { > > "id": "node1", > > "name": "Controller Node", > > "profile": "bootkube-controller", > > "selector": { > > "mac": "18:03:73:3f:6d:bf", > > "os": "installed" > > }, > > "metadata": { > > "cluster_id" : "test_agent", > > "config_server": "http://172.20.20.219:1979/k8/test_agent";, > > "ip_address" : "192.168.122.10", > > "ip_gateway" : "192.168.122.1", > > "net_mask" : "23", > > "domain_name": "k8.example.com", > > "etcd_initial_cluster": "k8=https://192.168.122.10:2380";, > > "etcd_endpoints": "https://192.168.122.10:2379";, > > "etcd_name": "k8", > > "k8s_dns_service_ip": "10.3.0.10", > > "ssh_authorized_keys": [ > > "ssh-rsa ..." ] > > } > > } > > > > > > > > > > #examples/ignition/bootkube-controller.yaml extended for dynamic > bootkube > > > > networkd: > > units: > > - name: 00-static.network > > contents: | > > [Match] > > Name=e* > > [Network] > > DNS=8.8.8.8 > > Address={{.ip_address}}/{{.net_mask}} > > Gateway={{.ip_gateway}} > > systemd: > > units: > > - name: bootkube.service > > enable: true > > contents: | > > [Unit] > > Description=Bootstrap a Kubernetes control plane with a temp > api-server > > [Service] > > Type=simple > > RemainAfterExit=yes > > Restart=always > > RestartSec=20 > > WorkingDirectory=/opt/bootkube > > ExecStartPre=/bin/sh -c 'wget -r -np -nH -x --cut-dirs=1 > {{.config_server}}/' > > ExecStartPre=/bin/sh -c 'mv {{.cluster_id}} > /opt/bootkube/assets' > > ExecStart=/opt/bootkube/bootkube-start > > [Install] > > WantedBy=multi-user.target > > > > storage: > > files: > > - path: /etc/ssl/etcd/etcd-client.crt > > filesystem: root > > mode: 0644 > > contents: > > remote: > > url: "{{.config_server}}/tls/etcd-client.crt" > > > > > > On Tuesday, June 27, 2017 at 7:06:48 AM UTC+2, luce...@gmail.com > wrote:Using Kubernetes 1.6 on Bar Metal servers (production) seems to be > difficult. > > Tested CoreOs and Tectonic, not working on Bar Metal servers. > > Bar Metal servers and conjure-up kubernetes not possible. (+ Maas) > > Rancher is working but the whole system is based on the docker > daemon, systemd is a better choice as base system for Kubernetes. (Like > CoreOs does) > > > > A possible solution is kubeadm but kubeadm is not ready for > production! > > Are there other possibilities? > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
