Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 26/09/14 00:01, Angus Lees wrote: On Thu, 25 Sep 2014 04:01:38 PM Fox, Kevin M wrote: Doesn't nova with a docker driver and heat autoscaling handle case 2 and 3 for control jobs? Has anyone tried yet? For reference, the cases were: - Something to deploy the code (docker / distro packages / pip install / etc) - Something to choose where to deploy - Something to respond to machine outages / autoscaling and re-deploy as necessary I tried for a while, yes. The problems I ran into (and I'd be interested to know if there are solutions to these): - I'm trying to deploy into VMs on rackspace public cloud (just because that's what I have). This means I can't use the nova docker driver, without constructing an entire self-contained openstack undercloud first. - heat+cloud-init (afaics) can't deal with circular dependencies (like nova- neutron) since the machines need to exist first before you can refer to their IPs. I suspect that this now can actually be done with Software Deployment resources (which separate the creation of the actual servers from the inputs to the software configured on them, thus breaking the circular dependency). From what I can see, TripleO gets around this by always scheduling them on the same machine and just using the known local IP. Other installs declare fixed IPs up front - on rackspace I can't do that (easily). I can't use loadbalancers via heat for this because the loadbalancers need to know the backend node addresses, which means the nodes have to exist first and you're back to a circular dependency. Again, Sofware Deployments should resolve that. For comparision, with kubernetes you declare the loadbalancer-equivalents (services) up front with a search expression for the backends. In a second pass you create the backends (pods) which can refer to any of the loadbalanced endpoints. The loadbalancers then reconfigure themselves on the fly to find the new backends. You _can_ do a similar lazy-loadbalancer-reconfig thing with openstack too, but not with heat and not just out of the box. - My experiences using heat for anything complex have been extremely frustrating. The version on rackspace public cloud is ancient and limited, and quite easy to get into a state where the only fix is to destroy the entire stack and recreate it. I don't know that Rackspace's version is ancient (afaik they chase trunk, so it's probably more recent than even Icehouse), but the Juno release should be much better on this front. I'm sure these are fixed in newer versions of heat, but last time I tried I was unable to run it standalone against an arms-length keystone because some of the recursive heat callbacks became confused about which auth token to use. (I'm sure this can be fixed, if it wasn't already just me using it wrong in the first place.) Yeah, Heat only really supports a fairly limited subset of functionality in standalone mode. (I mean, it's probably the majority, but it's everything except the _really_ useful parts ;) We're hearing more from users that they care about this, and by happy coincidence making technical changes from which a true standalone mode should fall out. - As far as I know, nothing in a heat/loadbalancer/nova stack will actually reschedule jobs away from a failed machine. That's correct, although it is something we have already started looking at and plan to implement over the next few cycles. There's also no lazy discovery/nameservice mechanism, so updating IP address declarations in cloud- configs tend to ripple through the heat config and cause all sorts of VMs/containers to be reinstalled without any sort of throttling or rolling update. Yes, I can't wait for Designate :) cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
-Original Message- From: Angus Lees [mailto:gusl...@gmail.com] On Behalf Of Angus Lees Sent: Thursday, September 25, 2014 9:01 PM To: openstack-dev@lists.openstack.org Cc: Fox, Kevin M Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On Thu, 25 Sep 2014 04:01:38 PM Fox, Kevin M wrote: Doesn't nova with a docker driver and heat autoscaling handle case 2 and 3 for control jobs? Has anyone tried yet? For reference, the cases were: - Something to deploy the code (docker / distro packages / pip install / etc) - Something to choose where to deploy - Something to respond to machine outages / autoscaling and re-deploy as necessary I tried for a while, yes. The problems I ran into (and I'd be interested to know if there are solutions to these): - I'm trying to deploy into VMs on rackspace public cloud (just because that's what I have). This means I can't use the nova docker driver, without constructing an entire self-contained openstack undercloud first. That's true. But you are essentially doing the same with Kubernetes. Installing a self contained undercloud. If that's the case, it would be nice to use the same docker containers to build an undercloud to deploy the over cloud and reuse almost all of the work, rather then deploy two different systems. - heat+cloud-init (afaics) can't deal with circular dependencies (like nova- neutron) since the machines need to exist first before you can refer to their IPs. Could this be made to work with by either a floating ip or load balancer with a floating ip in front so you know before hand what its going to be? From what I can see, TripleO gets around this by always scheduling them on the same machine and just using the known local IP. Other installs declare fixed IPs up front - on rackspace I can't do that (easily). I can't use loadbalancers via heat for this because the loadbalancers need to know the backend node addresses, which means the nodes have to exist first and you're back to a circular dependency. Starting in icehouse, heat does not need to know the backend ip's on lb creation. There is a PoolMember resource that acts similarly to a cinder mount resource. It lets you create the lb in one stack, and then an instance template that launches the vm, installs stuff, then binds to the lb in another. For an example, see: https://github.com/EMSL-MSC/heat-templates/tree/master/cfn/389/ BaseCluster and RO_Replica templates. For comparision, with kubernetes you declare the loadbalancer-equivalents (services) up front with a search expression for the backends. In a second pass you create the backends (pods) which can refer to any of the loadbalanced endpoints. The loadbalancers then reconfigure themselves on the fly to find the new backends. You _can_ do a similar lazy- loadbalancer-reconfig thing with openstack too, but not with heat and not just out of the box. As above, you should be able to do this with heat now. - My experiences using heat for anything complex have been extremely frustrating. The version on rackspace public cloud is ancient and limited, and quite easy to get into a state where the only fix is to destroy the entire stack and recreate it. I'm sure these are fixed in newer versions of heat, but last time I tried I was unable to run it standalone against an arms-length keystone because some of the recursive heat callbacks became confused about which auth token to use. Newer versions of heat are better. and its getting better all the time. I agree its not as fault tolerant as it needs to be yet in all cases. For now, I usually end up breaking my systems into a few separate stacks that I can deal with updates in a way that if there is a failure, nothing critical gets lost. Usually that means launching/deleting/relaunching components rather then ever updating them. (I'm sure this can be fixed, if it wasn't already just me using it wrong in the first place.) - As far as I know, nothing in a heat/loadbalancer/nova stack will actually reschedule jobs away from a failed machine. That's a problem currently, yes. There's also no lazy discovery/nameservice mechanism, so updating IP address declarations in cloud- configs tend to ripple through the heat config and cause all sorts of VMs/containers to be reinstalled without any sort of throttling or rolling update. I try and use floating ip's for these. Designate support so that you have a nice pretty name for it would be even better. Alternatively, I've had good luck with private Neutron networks. When you own the net, you can assign the vm any ip you want in it. In cases like Ceph where you have mons that need to be at known ip's, I just make sure the vm's running the mons are always launched with those fixed ips. Then you never have to reconfig slaves. So: I think there's some things to learn from
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Thu, Sep 25, 2014 at 6:02 AM, Clint Byrum cl...@fewbar.com wrote: However, this does make me think that Keystone domains should be exposable to services inside your cloud for use as SSO. It would be quite handy if the keystone users used for the VMs that host Kubernetes could use the same credentials to manage the containers. I was exactly thinking about the same and looking at the code here : https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/client/request.go#L263 it seems to use some basic HTTP auth which should be enough with the REMOTE_USER/apache feature of keystone : http://docs.openstack.org/developer/keystone/external-auth.html#using-httpd-authentication but if we want to have proper full integration with OpenStack we would probably at some point want to teach modularity and a keystone plugin to give to k8 Chmouel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Mike Spreitzer's message of 2014-09-24 22:01:54 -0700: Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM: Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: ... ... Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. In what sense would Kubernetes manage baremetal (at all)? By from a single host do you mean that a client on one host can manage remote baremetal and containers? I can see that Kubernetes allows a client on one host to get containers placed remotely --- but so does the Docker driver for Nova. I mean that one box would need to host Ironic, Docker, and Nova, for the purposes of deploying OpenStack. We call it the undercloud, or sometimes the Deployment Cloud. It's not necessarily something that Nova/Neutron cannot do by design, but it doesn't work now. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. Clint, in another branch of this email tree you referred to the VMs that host Kubernetes. How does that square with Steve's text that seems to imply bare metal minions? That was in a more general context, discussing using Kubernetes for general deployment. Could have just as easily have said hosts, machines, or instances. I can see that some people have had much more detailed design discussions than I have yet found. Perhaps it would be helpful to share an organized presentation of the design thoughts in more detail. I personally have not had any detailed discussions about this before it was announced. I've just dug into the design and some of the code of Kubernetes because it is quite interesting to me. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. So I am still not clear on what Steve is trying to say is the main use case. Kubernetes is even farther from balancing among PDUs than Nova is. At least Nova has a framework in which this issue can be posed and solved. I mean a framework that actually can carry the necessary information. The Kubernetes scheduler interface is extremely impoverished in the information it passes and it uses GO structs --- which, like C structs, can not be subclassed. I don't think this is totally clear yet. The thing that Steven seems to be trying to solve is deploying OpenStack using docker, and Kubernetes may very well be a better choice than Nova for this. There are some really nice features, and a lot of the benefits we've been citing about image based deployments are realized in docker without the pain of a full OS image to redeploy all the time. The structs vs. classes argument is completely out of line and has nothing to do with where Kubernetes might go in the future. It's like saying because cars use internal combustion engines they are limited. It is just a facet of how it works today. Nova's filter scheduler includes a fatal bug that bites when balancing and you want more than one element per area, see https://bugs.launchpad.net/nova/+bug/1373478. However: (a) you might not need more than one element per area and (b) fixing that bug is a much smaller job than expanding the mind of K8s. Perhaps. I am quite a fan of set based design, and Kubernetes is a narrowly focused single implementation solution, where Nova is a broadly focused abstraction layer for VM's. I think it is worthwhile to push a bit into the Kubernetes space and see whether the limitations are important or not. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/25/2014 12:01 AM, Clint Byrum wrote: Excerpts from Mike Spreitzer's message of 2014-09-24 22:01:54 -0700: Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM: Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: ... ... Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. In what sense would Kubernetes manage baremetal (at all)? By from a single host do you mean that a client on one host can manage remote baremetal and containers? I can see that Kubernetes allows a client on one host to get containers placed remotely --- but so does the Docker driver for Nova. I mean that one box would need to host Ironic, Docker, and Nova, for the purposes of deploying OpenStack. We call it the undercloud, or sometimes the Deployment Cloud. It's not necessarily something that Nova/Neutron cannot do by design, but it doesn't work now. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. Clint, in another branch of this email tree you referred to the VMs that host Kubernetes. How does that square with Steve's text that seems to imply bare metal minions? That was in a more general context, discussing using Kubernetes for general deployment. Could have just as easily have said hosts, machines, or instances. I can see that some people have had much more detailed design discussions than I have yet found. Perhaps it would be helpful to share an organized presentation of the design thoughts in more detail. I personally have not had any detailed discussions about this before it was announced. I've just dug into the design and some of the code of Kubernetes because it is quite interesting to me. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. So I am still not clear on what Steve is trying to say is the main use case. Kubernetes is even farther from balancing among PDUs than Nova is. At least Nova has a framework in which this issue can be posed and solved. I mean a framework that actually can carry the necessary information. The Kubernetes scheduler interface is extremely impoverished in the information it passes and it uses GO structs --- which, like C structs, can not be subclassed. I don't think this is totally clear yet. The thing that Steven seems to be trying to solve is deploying OpenStack using docker, and Kubernetes may very well be a better choice than Nova for this. There are some really nice features, and a lot of the benefits we've been citing about image based deployments are realized in docker without the pain of a full OS image to redeploy all the time. This is precisely the problem I want to solve. I looked at Nova+Docker as a solution, and it seems to me the runway to get to a successful codebase is longer with more risk. That is why this is an experiment to see if a Kubernetes based approach would work. if at the end of the day we throw out Kubernetes as a scheduler once we have the other problems solved and reimplement Kubernetes in Nova+Docker, I think that would be an acceptable outcome, but not something I want to *start* with but *finish* with. Regards -steve The structs vs. classes argument is completely out of line and has nothing to do with where Kubernetes might go in the future. It's like saying because cars use internal combustion engines they are limited. It is just a facet of how it works today. Nova's filter scheduler includes a fatal bug that bites when balancing and you want more than one element per area, see https://bugs.launchpad.net/nova/+bug/1373478. However: (a) you might not need more than one element per area and (b) fixing that bug is a much smaller job than expanding the mind of K8s. Perhaps. I am quite a fan of set based design, and Kubernetes is a narrowly focused single implementation solution, where Nova is a broadly focused abstraction layer for VM's. I think it is worthwhile to push a bit into the Kubernetes space and see whether the limitations are important or not. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/24/2014 10:01 PM, Mike Spreitzer wrote: Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM: Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: ... ... Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. In what sense would Kubernetes manage baremetal (at all)? By from a single host do you mean that a client on one host can manage remote baremetal and containers? I can see that Kubernetes allows a client on one host to get containers placed remotely --- but so does the Docker driver for Nova. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. Clint, in another branch of this email tree you referred to the VMs that host Kubernetes. How does that square with Steve's text that seems to imply bare metal minions? I can see that some people have had much more detailed design discussions than I have yet found. Perhaps it would be helpful to share an organized presentation of the design thoughts in more detail. Mike, I have had no such design discussions. Thus far the furthest along we are in the project is determining we need Docker containers for each of the OpenStack daemons. We are working a bit on how that design should operate. For example, our current model on reconfiguration of a docker container is to kill the docker container and start a fresh one with the new configuration. This is literally where the design discussions have finished. We have not had much discussion about Kubernetes at all other then I know it is a docker scheduler and I know it can get the job done :) I think other folks design discussions so far on this thread are speculation about what an architecture should look like. That is great - lets have those Monday 2000 UTC in #openstack-medeting in our first Kolla meeting. Regards -steve If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. So I am still not clear on what Steve is trying to say is the main use case. Kubernetes is even farther from balancing among PDUs than Nova is. At least Nova has a framework in which this issue can be posed and solved. I mean a framework that actually can carry the necessary information. The Kubernetes scheduler interface is extremely impoverished in the information it passes and it uses GO structs --- which, like C structs, can not be subclassed. Nova's filter scheduler includes a fatal bug that bites when balancing and you want more than one element per area, see https://bugs.launchpad.net/nova/+bug/1373478. However: (a) you might not need more than one element per area and (b) fixing that bug is a much smaller job than expanding the mind of K8s. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Doesn't nova with a docker driver and heat autoscaling handle case 2 and 3 for control jobs? Has anyone tried yet? Thanks, Kevin From: Angus Lees [g...@inodes.org] Sent: Wednesday, September 24, 2014 6:33 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On Wed, 24 Sep 2014 10:31:19 PM Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. I've no idea what Steven's motivation is, but here's my reasoning for going down a similar path: OpenStack deployment is basically two types of software: 1. Control jobs, various API servers, etc that are basically just regular python wsgi apps. 2. Compute/network node agents that run under hypervisors, configure host networking, etc. The 2nd group probably wants to run on baremetal and is mostly identical on all such machines, but the 1st group wants higher level PaaS type things. In particular, for the control jobs you want: - Something to deploy the code (docker / distro packages / pip install / etc) - Something to choose where to deploy - Something to respond to machine outages / autoscaling and re-deploy as necessary These last few don't have strong existing options within OpenStack yet (as far as I'm aware). Having explored a few different approaches recently, kubernetes is certainly not the only option - but is a reasonable contender here. So: I certainly don't see kubernetes as competing with anything in OpenStack - but as filling a gap in job management with something that has a fairly lightweight config syntax and is relatively simple to deploy on VMs or baremetal. I also think the phrase integrating kubernetes into OpenStack is overstating the task at hand. The primary downside I've discovered so far seems to be that kubernetes is very young and still has an awkward cli, a few easy to encounter bugs, etc. - Gus From: Steven Dake [mailto:sd...@redhat.com] Sent: September-24-14 7:41 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). Yup it is fairly large in size. Time will tell if this approach will work. This is an experiment as Robert and others on the thread have pointed out :). Regards -steve On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.commailto:sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Then you still need all the kubernetes api/daemons for the master and slaves. If you ignore the complexity this adds, then it seems simpler then just using openstack for it. but really, it still is an under/overcloud kind of setup, your just using kubernetes for the undercloud, and openstack for the overcloud? Thanks, Kevin From: Steven Dake [sd...@redhat.com] Sent: Wednesday, September 24, 2014 8:02 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling and once scheduled, Kubernetes would be responsible for connecting the various containers together. Regards -steve From: Steven Dake [mailto:sd...@redhat.com] Sent: September-24-14 7:41 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). Yup it is fairly large in size. Time will tell if this approach will work. This is an experiment as Robert and others on the thread have pointed out :). Regards -steve On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.commailto:sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Why can't you manage baremetal and containers from a single host with nova/neutron? Is this a current missing feature, or has the development teams said they will never implement it? Thanks, Kevin From: Clint Byrum [cl...@fewbar.com] Sent: Wednesday, September 24, 2014 9:13 PM To: openstack-dev Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling I am not sure what is meant here by pass config options nor how it would be done prior to scheduling; can you please clarify? I do not imagine Kubernetes would *choose* the config values, K8s does not know anything about configuring OpenStack. Before scheduling, there is no running container to pass anything to. Docker containers tend to use environment variables passed to the initial command to configure things. The Kubernetes API allows setting these environment variables on creation of the container. and once scheduled, Kubernetes would be responsible for connecting the various containers together. Kubernetes has a limited role in connecting containers together. K8s creates the networking environment in which the containers *can* communicate, and passes environment variables into containers telling them from what protocol://host:port/ to import each imported endpoint. Kubernetes creates a universal reverse proxy on each minion, to provide endpoints that do not vary as the servers move around. It is up to stuff outside Kubernetes to decide what should be connected to what, and it is up to the containers to read the environment variables and actually connect. This is a nice simple interface though, and I like that it is narrowly defined, not trying to be anything that containers want to share with other containers. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
First, Kevin, please try to figure out a way to reply in-line when you're replying to multiple levels of threads. Even if you have to copy and quote it manually.. it took me reading your message and the previous message 3 times to understand the context. Second, I don't think anybody minds having a control plane for each level of control. The point isn't to replace the undercloud, but to replace nova rebuild as the way you push out new software while retaining the benefits of the image approach. Excerpts from Fox, Kevin M's message of 2014-09-25 09:07:10 -0700: Then you still need all the kubernetes api/daemons for the master and slaves. If you ignore the complexity this adds, then it seems simpler then just using openstack for it. but really, it still is an under/overcloud kind of setup, your just using kubernetes for the undercloud, and openstack for the overcloud? Thanks, Kevin From: Steven Dake [sd...@redhat.com] Sent: Wednesday, September 24, 2014 8:02 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling and once scheduled, Kubernetes would be responsible for connecting the various containers together. Regards -steve From: Steven Dake [mailto:sd...@redhat.com] Sent: September-24-14 7:41 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). Yup it is fairly large in size. Time will tell if this approach will work. This is an experiment as Robert and others on the thread have pointed out :). Regards -steve On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.commailto:sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Ah. So the goal of project Kolla then is to deploy OpenStack via Docker using whatever means that works, not, to deploy OpenStack using Docker+Kubernetes, where the first stab at an implementation is using Kubernetes. That seems like a much more reasonable goal to me. Thanks, Kevin From: Steven Dake [sd...@redhat.com] Sent: Thursday, September 25, 2014 8:30 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/25/2014 12:01 AM, Clint Byrum wrote: Excerpts from Mike Spreitzer's message of 2014-09-24 22:01:54 -0700: Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM: Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: ... ... Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. In what sense would Kubernetes manage baremetal (at all)? By from a single host do you mean that a client on one host can manage remote baremetal and containers? I can see that Kubernetes allows a client on one host to get containers placed remotely --- but so does the Docker driver for Nova. I mean that one box would need to host Ironic, Docker, and Nova, for the purposes of deploying OpenStack. We call it the undercloud, or sometimes the Deployment Cloud. It's not necessarily something that Nova/Neutron cannot do by design, but it doesn't work now. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. Clint, in another branch of this email tree you referred to the VMs that host Kubernetes. How does that square with Steve's text that seems to imply bare metal minions? That was in a more general context, discussing using Kubernetes for general deployment. Could have just as easily have said hosts, machines, or instances. I can see that some people have had much more detailed design discussions than I have yet found. Perhaps it would be helpful to share an organized presentation of the design thoughts in more detail. I personally have not had any detailed discussions about this before it was announced. I've just dug into the design and some of the code of Kubernetes because it is quite interesting to me. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. So I am still not clear on what Steve is trying to say is the main use case. Kubernetes is even farther from balancing among PDUs than Nova is. At least Nova has a framework in which this issue can be posed and solved. I mean a framework that actually can carry the necessary information. The Kubernetes scheduler interface is extremely impoverished in the information it passes and it uses GO structs --- which, like C structs, can not be subclassed. I don't think this is totally clear yet. The thing that Steven seems to be trying to solve is deploying OpenStack using docker, and Kubernetes may very well be a better choice than Nova for this. There are some really nice features, and a lot of the benefits we've been citing about image based deployments are realized in docker without the pain of a full OS image to redeploy all the time. This is precisely the problem I want to solve. I looked at Nova+Docker as a solution, and it seems to me the runway to get to a successful codebase is longer with more risk. That is why this is an experiment to see if a Kubernetes based approach would work. if at the end of the day we throw out Kubernetes as a scheduler once we have the other problems solved and reimplement Kubernetes in Nova+Docker, I think that would be an acceptable outcome, but not something I want to *start* with but *finish* with. Regards -steve The structs vs. classes argument is completely out of line and has nothing to do with where Kubernetes might go in the future. It's like saying because cars use internal combustion engines they are limited. It is just a facet of how it works today. Nova's filter scheduler includes a fatal bug that bites when balancing and you want more than one element per area, see https://bugs.launchpad.net/nova/+bug/1373478. However: (a) you might not need more than one element per area
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Fox, Kevin M's message of 2014-09-25 09:13:26 -0700: Why can't you manage baremetal and containers from a single host with nova/neutron? Is this a current missing feature, or has the development teams said they will never implement it? It's a bug. But it is also a complexity that isn't really handled well in Nova's current design. Nova wants to send the workload onto the machine, and that is it. In this case, you have two workloads, one hosted on the other, and Nova has no model for that. You end up in a weird situation where one (baremetal) is host for other (containers) and no real way to separate the two or identify that dependency. I think it's worth pursuing in OpenStack, but Steven is solving deployment of OpenStack today with tools that exist today. I think Kolla may very well prove that the container approach is too different from Nova's design and wants to be more separate, at which point our big tent will be in an interesting position: Do we adopt Kubernetes and put an OpenStack API on it, or do we re-implement it. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
-Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: Thursday, September 25, 2014 9:35 AM To: openstack-dev Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker First, Kevin, please try to figure out a way to reply in-line when you're replying to multiple levels of threads. Even if you have to copy and quote it manually.. it took me reading your message and the previous message 3 times to understand the context. I'm sorry. I think your frustration with it mirrors the frustration I have with having to use this blankity blank microsoft webmail that doesn't support inline commenting, or having to rdesktop to a windows terminal server so I can reply inline. :/ Second, I don't think anybody minds having a control plane for each level of control. The point isn't to replace the undercloud, but to replace nova rebuild as the way you push out new software while retaining the benefits of the image approach. I don't quite follow. Wouldn't you be using heat autoscaling, not nova directly? Thanks, Kevin Excerpts from Fox, Kevin M's message of 2014-09-25 09:07:10 -0700: Then you still need all the kubernetes api/daemons for the master and slaves. If you ignore the complexity this adds, then it seems simpler then just using openstack for it. but really, it still is an under/overcloud kind of setup, your just using kubernetes for the undercloud, and openstack for the overcloud? Thanks, Kevin From: Steven Dake [sd...@redhat.com] Sent: Wednesday, September 24, 2014 8:02 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling and once scheduled, Kubernetes would be responsible for connecting the various containers together. Regards -steve From: Steven Dake [mailto:sd...@redhat.com] Sent: September-24-14 7:41 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
-Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: Thursday, September 25, 2014 9:44 AM To: openstack-dev Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker Excerpts from Fox, Kevin M's message of 2014-09-25 09:13:26 -0700: Why can't you manage baremetal and containers from a single host with nova/neutron? Is this a current missing feature, or has the development teams said they will never implement it? It's a bug. But it is also a complexity that isn't really handled well in Nova's current design. Nova wants to send the workload onto the machine, and that is it. In this case, you have two workloads, one hosted on the other, and Nova has no model for that. You end up in a weird situation where one (baremetal) is host for other (containers) and no real way to separate the two or identify that dependency. Ideally, like you say, you should be able to have one host managed by two different nova drivers in the same cell. But I think today, you can simply use two different cells and it should work? One cell for deploying bare metal images, of which one image is provided that contains the nova docker compute resources. The other cell used to support launching docker instances on those hosts. To the end user, it still looks like one unified cloud like we all want, but under the hood, its two separate subclouds. An under and an overcloud. I think it's worth pursuing in OpenStack, but Steven is solving deployment of OpenStack today with tools that exist today. I think Kolla may very well prove that the container approach is too different from Nova's design and wants to be more separate, at which point our big tent will be in an interesting position: Do we adopt Kubernetes and put an OpenStack API on it, or do we re-implement it. That is a very interesting question, worth pursuing. I think either way, most of the work is going to be in dockerizing the services. So that alone is worth playing with too. I managed to get libvirt to work in docker once. It was a pain. Getting nova and neutron bits in that container too would be even harder. I'm waiting to try again until I know that systemd will run nicely inside a docker container. It would make managing the startup/stopping of the container much easier to get right. Thanks, Kevin ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Thu, 25 Sep 2014 04:01:38 PM Fox, Kevin M wrote: Doesn't nova with a docker driver and heat autoscaling handle case 2 and 3 for control jobs? Has anyone tried yet? For reference, the cases were: - Something to deploy the code (docker / distro packages / pip install / etc) - Something to choose where to deploy - Something to respond to machine outages / autoscaling and re-deploy as necessary I tried for a while, yes. The problems I ran into (and I'd be interested to know if there are solutions to these): - I'm trying to deploy into VMs on rackspace public cloud (just because that's what I have). This means I can't use the nova docker driver, without constructing an entire self-contained openstack undercloud first. - heat+cloud-init (afaics) can't deal with circular dependencies (like nova- neutron) since the machines need to exist first before you can refer to their IPs. From what I can see, TripleO gets around this by always scheduling them on the same machine and just using the known local IP. Other installs declare fixed IPs up front - on rackspace I can't do that (easily). I can't use loadbalancers via heat for this because the loadbalancers need to know the backend node addresses, which means the nodes have to exist first and you're back to a circular dependency. For comparision, with kubernetes you declare the loadbalancer-equivalents (services) up front with a search expression for the backends. In a second pass you create the backends (pods) which can refer to any of the loadbalanced endpoints. The loadbalancers then reconfigure themselves on the fly to find the new backends. You _can_ do a similar lazy-loadbalancer-reconfig thing with openstack too, but not with heat and not just out of the box. - My experiences using heat for anything complex have been extremely frustrating. The version on rackspace public cloud is ancient and limited, and quite easy to get into a state where the only fix is to destroy the entire stack and recreate it. I'm sure these are fixed in newer versions of heat, but last time I tried I was unable to run it standalone against an arms-length keystone because some of the recursive heat callbacks became confused about which auth token to use. (I'm sure this can be fixed, if it wasn't already just me using it wrong in the first place.) - As far as I know, nothing in a heat/loadbalancer/nova stack will actually reschedule jobs away from a failed machine. There's also no lazy discovery/nameservice mechanism, so updating IP address declarations in cloud- configs tend to ripple through the heat config and cause all sorts of VMs/containers to be reinstalled without any sort of throttling or rolling update. So: I think there's some things to learn from the kubernetes approach, which is why I'm trying to gain more experience with it. I know I'm learning more about the various OpenStack components along the way too ;) -- - Gus ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Fri, Sep 26, 2014 at 2:01 PM, Angus Lees g...@inodes.org wrote: On Thu, 25 Sep 2014 04:01:38 PM Fox, Kevin M wrote: Doesn't nova with a docker driver and heat autoscaling handle case 2 and 3 for control jobs? Has anyone tried yet? For reference, the cases were: - Something to deploy the code (docker / distro packages / pip install / etc) - Something to choose where to deploy - Something to respond to machine outages / autoscaling and re-deploy as necessary I tried for a while, yes. The problems I ran into (and I'd be interested to know if there are solutions to these): - I'm trying to deploy into VMs on rackspace public cloud (just because that's what I have). This means I can't use the nova docker driver, without constructing an entire self-contained openstack undercloud first. - heat+cloud-init (afaics) can't deal with circular dependencies (like nova- neutron) since the machines need to exist first before you can refer to their IPs. From what I can see, TripleO gets around this by always scheduling them on the same machine and just using the known local IP. Other installs declare fixed IPs up front - on rackspace I can't do that (easily). I can't use loadbalancers via heat for this because the loadbalancers need to know the backend node addresses, which means the nodes have to exist first and you're back to a circular dependency. For comparision, with kubernetes you declare the loadbalancer-equivalents (services) up front with a search expression for the backends. In a second pass you create the backends (pods) which can refer to any of the loadbalanced endpoints. The loadbalancers then reconfigure themselves on the fly to find the new backends. You _can_ do a similar lazy-loadbalancer-reconfig thing with openstack too, but not with heat and not just out of the box. Do you have a minimal template that shows what you are trying to do? (just to demonstrate the circular dependency). - My experiences using heat for anything complex have been extremely frustrating. The version on rackspace public cloud is ancient and limited, and quite easy to get into a state where the only fix is to destroy the entire stack and recreate it. I'm sure these are fixed in newer versions of heat, but last time I tried I was unable to run it standalone against an arms-length keystone because some of the recursive heat callbacks became confused about which auth token to use. Gus we are working at improving standalone (Steven Baker has some patch out for this). (I'm sure this can be fixed, if it wasn't already just me using it wrong in the first place.) - As far as I know, nothing in a heat/loadbalancer/nova stack will actually reschedule jobs away from a failed machine. There's also no lazy This might go part of the way there, the other part of it is detecting the failed machine and some how marking it as failed. https://review.openstack.org/#/c/105907/ discovery/nameservice mechanism, so updating IP address declarations in cloud- configs tend to ripple through the heat config and cause all sorts of VMs/containers to be reinstalled without any sort of throttling or rolling update. So: I think there's some things to learn from the kubernetes approach, which is why I'm trying to gain more experience with it. I know I'm learning more about the various OpenStack components along the way too ;) This is valuable feedback, we need to improve Heat to make these use case work better. But I also don't believe there is one tool for all jobs, so see little harm in trying other things out too. Thanks Angus -- - Gus ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 24 September 2014 16:38, Jay Pipes jaypi...@gmail.com wrote: On 09/23/2014 10:29 PM, Steven Dake wrote: There is a deployment program - tripleo is just one implementation. Nope, that is not correct. Like it or not (I personally don't), Triple-O is *the* Deployment Program for OpenStack: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n284 Saying Triple-O is just one implementation of a deployment program is like saying Heat is just one implementation of an orchestration program. It isn't. It's *the* implemenation of an orchestration program that has been blessed by the TC: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n112 Thats not what Steve said. He said that the codebase they are creating is a *project* with a target home of the OpenStack Deployment *program*, aka TripleO. The TC blesses social structure and code separately: no part of TripleO has had its code blessed by the TC yet (incubation/graduation), but the team was blessed. I've no opinion on the Murano bits you raise. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Jay Pipes's message of 2014-09-23 21:38:37 -0700: On 09/23/2014 10:29 PM, Steven Dake wrote: There is a deployment program - tripleo is just one implementation. Nope, that is not correct. Like it or not (I personally don't), Triple-O is *the* Deployment Program for OpenStack: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n284 Saying Triple-O is just one implementation of a deployment program is like saying Heat is just one implementation of an orchestration program. It isn't. It's *the* implemenation of an orchestration program that has been blessed by the TC: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n112 That was written before we learned everything we've learned in the last 12 months. I think it is unfair to simply point to this and imply that bending or even changing it is not open for discussion. We went through this with Heat and various projects that want to extend heat (eg Murano) and one big mistake I think Murano folks made was not figuring out where there code would go prior to writing it. I'm only making a statement as to where I think it should belong. Sorry, I have to call you to task on this. You think it was a mistake for the Murano folks to not figure out where the code would go prior to writing it? For the record, Murano existed nearly 2 years ago, as a response to various customer requests. Having the ability to properly deploy Windows applications like SQL Server and Active Directory into an OpenStack cloud was more important to the Murano developers than trying to predict what the whims of the OpenStack developer and governance model would be months or years down the road. Tell me, did any of Heat's code exist prior to deciding to propose it for incubation? Saying that Murano developers should have thought about where their code would live is holding them to a higher standard than any of the other developer communities. Did folks working on disk-image-builder pre-validate with the TC or the mailing list that the dib code would live in the triple-o program? No, of course not. It was developed naturally and then placed into the program that fit it best. Murano was developed naturally in exactly the same way, and the Murano developers have been nothing but accommodating to every request made of them by the TC (and those requests have been entirely different over the last 18 months, ranging from split it out to just propose another program) and by the PTLs for projects that requested they split various parts of Murano out into existing programs. The Murano developers have done no power grab, have deliberately tried to be as community-focused and amenable to all requests as possible, and yet they are treated with disdain by a number of folks in the core Heat developer community, including yourself, Clint and Zane. And honestly, I don't get it... all Murano is doing is generating Heat templates and trying to fill in some pieces that Heat isn't interested in doing. I don't see why there is so much animosity towards a project that has, to my knowledge, acted in precisely the ways that we've asked projects to act in the OpenStack community: with openness, transparency, and community good will. Disdain is hardly the right word. Disdain implies we don't have any respect at all for Murano. I cannot speak for others, but I do have respect. I'm just not interested in Murano. FWIW, I think what Steven Dake is saying is that he does not want to end up in the same position Murano is in. I think that is unlikely, as we're seeing many projects hitting the same wall, which is the cause for discussing changing how we include or exclude projects. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Wed, Sep 24, 2014 at 12:40 AM, Steven Dake sd...@redhat.com wrote: I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base Congratulations this sounds promising! If I understand correctly reading your POC there is two part to Kolla the docker images repository of openstack services and a future service (or kubernetes plugin?[1]) driving the communication and deployments to kubernetes. I think making sure that we separate the two would be nice to have, since if we can plug those images within devstack, thanks to the abstraction of how we run processes that was introduced by Dean (http://git.io/Px1nMg) perhaps that would be a nice way to make devstack more robust and have a nice side effect to have a pretty good testing for those docker images. Chmouel [1] CAVEAT: I don't know very well kubernetes, ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/24/2014 03:57 AM, Clint Byrum wrote: Excerpts from Jay Pipes's message of 2014-09-23 21:38:37 -0700: On 09/23/2014 10:29 PM, Steven Dake wrote: There is a deployment program - tripleo is just one implementation. Nope, that is not correct. Like it or not (I personally don't), Triple-O is *the* Deployment Program for OpenStack: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n284 Saying Triple-O is just one implementation of a deployment program is like saying Heat is just one implementation of an orchestration program. It isn't. It's *the* implemenation of an orchestration program that has been blessed by the TC: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n112 That was written before we learned everything we've learned in the last 12 months. I think it is unfair to simply point to this and imply that bending or even changing it is not open for discussion. My statement above is a reflection of the current reality of OpenStack governance policies and organizational structure. It's neither fair nor unfair. We went through this with Heat and various projects that want to extend heat (eg Murano) and one big mistake I think Murano folks made was not figuring out where there code would go prior to writing it. I'm only making a statement as to where I think it should belong. Sorry, I have to call you to task on this. You think it was a mistake for the Murano folks to not figure out where the code would go prior to writing it? For the record, Murano existed nearly 2 years ago, as a response to various customer requests. Having the ability to properly deploy Windows applications like SQL Server and Active Directory into an OpenStack cloud was more important to the Murano developers than trying to predict what the whims of the OpenStack developer and governance model would be months or years down the road. Tell me, did any of Heat's code exist prior to deciding to propose it for incubation? Saying that Murano developers should have thought about where their code would live is holding them to a higher standard than any of the other developer communities. Did folks working on disk-image-builder pre-validate with the TC or the mailing list that the dib code would live in the triple-o program? No, of course not. It was developed naturally and then placed into the program that fit it best. Murano was developed naturally in exactly the same way, and the Murano developers have been nothing but accommodating to every request made of them by the TC (and those requests have been entirely different over the last 18 months, ranging from split it out to just propose another program) and by the PTLs for projects that requested they split various parts of Murano out into existing programs. The Murano developers have done no power grab, have deliberately tried to be as community-focused and amenable to all requests as possible, and yet they are treated with disdain by a number of folks in the core Heat developer community, including yourself, Clint and Zane. And honestly, I don't get it... all Murano is doing is generating Heat templates and trying to fill in some pieces that Heat isn't interested in doing. I don't see why there is so much animosity towards a project that has, to my knowledge, acted in precisely the ways that we've asked projects to act in the OpenStack community: with openness, transparency, and community good will. Disdain is hardly the right word. Disdain implies we don't have any respect at all for Murano. I cannot speak for others, but I do have respect. I'm just not interested in Murano. OK. FWIW, I think what Steven Dake is saying is that he does not want to end up in the same position Murano is in. Perhaps. I just took offense to the implication (big mistake .. the Murano folks made) that somehow it was the Murano developer team's fault that they didn't have the foresight to predict the mess that the governance structure and policies have caused projects that want to be in the openstack/ code namespace but need to go through several arbitrary Trials by Fire before the TC to do so. I think that is unlikely, as we're seeing many projects hitting the same wall, which is the cause for discussing changing how we include or exclude projects. Hey, I'm all for changing the way we build the OpenStack tent. I just didn't think it was right to call out the Murano team in the way that it was. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Wed, Sep 24, 2014 at 9:16 AM, Jay Pipes jaypi...@gmail.com wrote: On 09/24/2014 03:19 AM, Robert Collins wrote: On 24 September 2014 16:38, Jay Pipes jaypi...@gmail.com wrote: On 09/23/2014 10:29 PM, Steven Dake wrote: There is a deployment program - tripleo is just one implementation. Nope, that is not correct. Like it or not (I personally don't), Triple-O is *the* Deployment Program for OpenStack: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n284 Saying Triple-O is just one implementation of a deployment program is like saying Heat is just one implementation of an orchestration program. It isn't. It's *the* implemenation of an orchestration program that has been blessed by the TC: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n112 Thats not what Steve said. He said that the codebase they are creating is a *project* with a target home of the OpenStack Deployment *program*, aka TripleO. The TC blesses social structure and code separately: no part of TripleO has had its code blessed by the TC yet (incubation/graduation), but the team was blessed. There are zero programs in the OpenStack governance repository that have competing implementations for the same thing. Like it or not, the TC process of blessing these teams has effectively blessed a single implementation of something. And it looks to me like what's being proposed here is that there is a group of folks who intend to work on Knoll, and they are indicating that they plan to participate and would like to be a part of that team. Personally, as a TripleO team member, I welcome that approach and their willingness to participate and share experience with the Deployment program. Meaning: exactly what you seem to claim is not possible due to some perceived blessing, is indeed in fact happening, or trying to come about. It would be great if Heat was already perfect and great at doing container orchestration *really* well. I'm not saying Kubernetes is either, but I'm not going to dismiss it just b/c it might compete with Heat. I see lots of other integration points with OpenStack services (using heat/nova/ironic to deploy kubernetes host, still using ironic to deploy baremetal storage nodes due to the iscsi issue, etc). -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Wed, Sep 24, 2014 at 9:41 AM, James Slagle james.sla...@gmail.com wrote: And it looks to me like what's being proposed here is that there is a group of folks who intend to work on Knoll, and they are indicating Oops, I meant Kolla, obviously :-). -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/24/2014 09:41 AM, James Slagle wrote: On Wed, Sep 24, 2014 at 9:16 AM, Jay Pipes jaypi...@gmail.com wrote: There are zero programs in the OpenStack governance repository that have competing implementations for the same thing. Like it or not, the TC process of blessing these teams has effectively blessed a single implementation of something. And it looks to me like what's being proposed here is that there is a group of folks who intend to work on Knoll, and they are indicating that they plan to participate and would like to be a part of that team. Personally, as a TripleO team member, I welcome that approach and their willingness to participate and share experience with the Deployment program. Nobody is saying what the Kolla folks are doing is not laudable. I'm certainly not saying that. I think it's great to participate and be open from the start. What I took umbrage with was the statement that it was the Murano developers who made the mistake years ago of basically not being in the right place at the right time. Meaning: exactly what you seem to claim is not possible due to some perceived blessing, is indeed in fact happening, or trying to come about. :) Talking about something on the ML is not the same thing as having that thing happen in real life. Kolla folks can and should discuss their end goal of being in the openstack/ code namespace and offering an alternate implementation for deploying OpenStack. That doesn't mean that the Technical Committee will allow this, though. Which is what I'm saying... the real world right now does not match this perception that a group can just state where they want to end up in the openstack/ code namespace and by just being up front about it, that magically happens. It would be great if Heat was already perfect and great at doing container orchestration *really* well. I'm not saying Kubernetes is either, but I'm not going to dismiss it just b/c it might compete with Heat. I see lots of other integration points with OpenStack services (using heat/nova/ironic to deploy kubernetes host, still using ironic to deploy baremetal storage nodes due to the iscsi issue, etc). Again, I'm not dismissing Kolla whatsoever. I think it's a great initiative. I'd point out that Fuel has been doing deployment with Docker containers for a while now, also out in the open, but on stackforge. Would the deployment program welcome Fuel into the openstack/ code namespace as well? Something to think about. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Wed, Sep 24, 2014 at 10:03 AM, Jay Pipes jaypi...@gmail.com wrote: On 09/24/2014 09:41 AM, James Slagle wrote: Meaning: exactly what you seem to claim is not possible due to some perceived blessing, is indeed in fact happening, or trying to come about. :) Talking about something on the ML is not the same thing as having that thing happen in real life. Hence the trying to come about. And the only thing proposed for real life right now is a project under stackforge whose long term goal is to merge into the Deployment program. I don't get the opposition to a long term goal. Kolla folks can and should discuss their end goal of being in the openstack/ code namespace and offering an alternate implementation for deploying OpenStack. That doesn't mean that the Technical Committee will allow this, though. Certainly true. Perhaps the mission statement for the Deployment program needs some tweaking. Perhaps it will be covered by whatever plays out within the larger OpenStack changes that are being discussed about the future of programs/projects/etc. Personally, I think there is some room for interpretation in the existing mission statement around the wherever possible phrase. Where it's not possible, OpenStack does not have to be used. So again, we probably need to update for clarity. I think the Deployment program should work with the TC to help define what it wants to be. Which is what I'm saying... the real world right now does not match this perception that a group can just state where they want to end up in the openstack/ code namespace and by just being up front about it, that magically happens. I'm not sure who you are arguing against that has that perception :). I've reread the thread, and I see desires being voiced to join an existing program, and some initial support offered in favor of that, minus your responses ;-). Obviously patches would have to be proposed to the governance repo to add projects under the program, those would have to be approved by people with +2 in governance, etc. No one claims it will be magically done. It would be great if Heat was already perfect and great at doing container orchestration *really* well. I'm not saying Kubernetes is either, but I'm not going to dismiss it just b/c it might compete with Heat. I see lots of other integration points with OpenStack services (using heat/nova/ironic to deploy kubernetes host, still using ironic to deploy baremetal storage nodes due to the iscsi issue, etc). Again, I'm not dismissing Kolla whatsoever. I think it's a great initiative. I'd point out that Fuel has been doing deployment with Docker containers for a while now, also out in the open, but on stackforge. Would the deployment program welcome Fuel into the openstack/ code namespace as well? Something to think about. Based on what you're saying about the Deployment program, you seem to indicate the TC would say No. I don't speak for the program. In the past, I've personally expressed support for alternative implementations where they make sense for OpenStack as a whole, and I still feel that way. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long term, as we experiment we believe it is best to hold a separate weekly one hour IRC meeting on Mondays at 2000 UTC in #openstack-meeting [3]. This project has been discussed with the current TripleO PTL (Robert Collins) and he seemed very supportive and agreed with the organization of the project outlined above. James Slagle, a TripleO core developer, has kindly offered to liase between Kolla and the broader TripleO community. I personally feel it is necessary to start from a nearly empty repository when kicking off a new project. As a result, there is limited code in the repository [4] at this time. I suspect folks will start cranking out a kick-ass implementation once the Kolla/Stackforge integration support is reviewed by the infra team [5]. The initial core team is composed of Steven Dake, Ryan Hallisey, James Lebocki, Jeff Peeler, James Slagle, Lars Kellogg-Sedman, and David Vossel. The core team will be reviewed every 6 weeks to add fresh developers. Please join the core team in designing and inventing this rockin' new technology! Regards -steve ~~ [1] https://github.com/docker/docker [2] https://github.com/GoogleCloudPlatform/kubernetes [3] https://wiki.openstack.org/wiki/Meetings/Kolla [4] https://github.com/jlabocki/superhappyfunshow [5] https://review.openstack.org/#/c/122972/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Steven Dake's message of 2014-09-23 15:40:29 -0700: *Hi folks,*** * I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. You had me at Docker.. Kubernetes establishes robust declarative primitives for maintaining the desired state requested by the user. We see these primitives as the main value added by Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers require active controllers, not just imperative orchestration. But the bit above has me a bit nervous... I'm not exactly ignorant of what declarative orchestration is, and of late I've found it to be more trouble than I had previously imagined it would be. All of the features above are desirable in any application, whether docker managed or not, and have been discussed for Heat specifically. I'm not entirely sure I want these things in my OpenStack deployment, but it will be interesting to see if there are operators who want them bad enough to deal with the inherent complexities of trying to write such a thing for an application as demanding as OpenStack. Anyway, I would definitely be interested in seeing if we can plug it into the interfaces we already have for image building, config file and system state management. Thanks for sharing, and see you in the deployment trenches. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan From: Steven Dake [mailto:sd...@redhat.com] Sent: September-24-14 7:41 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). Yup it is fairly large in size. Time will tell if this approach will work. This is an experiment as Robert and others on the thread have pointed out :). Regards -steve On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.commailto:sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long term, as we experiment we believe it is best to hold a separate weekly one hour IRC meeting on Mondays at 2000 UTC in #openstack-meeting [3]. This project has been discussed with the current TripleO PTL (Robert Collins) and he seemed very supportive and agreed with the organization of the project outlined above. James Slagle, a TripleO core developer, has kindly offered to liase between Kolla and the broader TripleO community. I personally feel it is necessary to start from a nearly empty repository when kicking off a new project. As a result, there is limited code in the repository [4] at this time. I suspect folks will start cranking out a kick-ass implementation once the Kolla/Stackforge integration support is reviewed by the infra team [5]. The initial core team is composed of Steven Dake, Ryan Hallisey, James Lebocki, Jeff Peeler, James Slagle, Lars Kellogg-Sedman, and David Vossel. The core team will be reviewed every 6 weeks to add fresh developers. Please join the core team in designing and inventing this rockin' new technology! Regards -steve ~~ [1] https://github.com/docker/docker [2] https://github.com/GoogleCloudPlatform/kubernetes [3] https://wiki.openstack.org/wiki/Meetings/Kolla [4] https://github.com/jlabocki/superhappyfunshow [5] https://review.openstack.org/#/c/122972/ ___ OpenStack-dev mailing list OpenStack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Wed, 24 Sep 2014 10:31:19 PM Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. I've no idea what Steven's motivation is, but here's my reasoning for going down a similar path: OpenStack deployment is basically two types of software: 1. Control jobs, various API servers, etc that are basically just regular python wsgi apps. 2. Compute/network node agents that run under hypervisors, configure host networking, etc. The 2nd group probably wants to run on baremetal and is mostly identical on all such machines, but the 1st group wants higher level PaaS type things. In particular, for the control jobs you want: - Something to deploy the code (docker / distro packages / pip install / etc) - Something to choose where to deploy - Something to respond to machine outages / autoscaling and re-deploy as necessary These last few don't have strong existing options within OpenStack yet (as far as I'm aware). Having explored a few different approaches recently, kubernetes is certainly not the only option - but is a reasonable contender here. So: I certainly don't see kubernetes as competing with anything in OpenStack - but as filling a gap in job management with something that has a fairly lightweight config syntax and is relatively simple to deploy on VMs or baremetal. I also think the phrase integrating kubernetes into OpenStack is overstating the task at hand. The primary downside I've discovered so far seems to be that kubernetes is very young and still has an awkward cli, a few easy to encounter bugs, etc. - Gus From: Steven Dake [mailto:sd...@redhat.com] Sent: September-24-14 7:41 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). Yup it is fairly large in size. Time will tell if this approach will work. This is an experiment as Robert and others on the thread have pointed out :). Regards -steve On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.commailto:sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Angus Lees's message of 2014-09-24 18:33:22 -0700: These last few don't have strong existing options within OpenStack yet (as far as I'm aware). Having explored a few different approaches recently, kubernetes is certainly not the only option - but is a reasonable contender here. So: I certainly don't see kubernetes as competing with anything in OpenStack - but as filling a gap in job management with something that has a fairly lightweight config syntax and is relatively simple to deploy on VMs or baremetal. I also think the phrase integrating kubernetes into OpenStack is overstating the task at hand. Reading more on the design and intention, I think it very much competes with Heat. The moment it learns to talk to Nova, it becomes a convergence engine for OpenStack. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling and once scheduled, Kubernetes would be responsible for connecting the various containers together. Regards -steve *From:*Steven Dake [mailto:sd...@redhat.com] *Sent:* September-24-14 7:41 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/24/2014 10:12 AM, Joshua Harlow wrote: Sounds like an interesting project/goal and will be interesting to see where this goes. A few questions/comments: How much golang will people be exposed to with this addition? Joshua, I expect very little. We intend to use Kubernetes as an upstream project, rather then something we contribute to directly. Seeing that this could be the first 'go' using project it will be interesting to see where this goes (since afaik none of the infra support exists, and people aren't likely to familiar with go vs python in the openstack community overall). What's your thoughts on how this will affect the existing openstack container effort? I don't think it will have any impact on the existing Magnum project. At some point if Magnum implements scheduling of docker containers, we may add support for Magnum in addition to Kubernetes, but it is impossible to tell at this point. I don't want to derail either project by trying to force them together unnaturally so early. I see that kubernetes isn't exactly a small project either (~90k LOC, for those who use these types of metrics), so I wonder how that will affect people getting involved here, aka, who has the resources/operators/other... available to actually setup/deploy/run kubernetes, when operators are likely still just struggling to run openstack itself (at least operators are getting used to the openstack warts, a new set of kubernetes warts could not be so helpful). Yup it is fairly large in size. Time will tell if this approach will work. This is an experiment as Robert and others on the thread have pointed out :). Regards -steve On Sep 23, 2014, at 3:40 PM, Steven Dake sd...@redhat.com mailto:sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long term, as we experiment we believe it is best to hold a separate weekly one hour IRC meeting on Mondays at 2000 UTC in #openstack-meeting [3]. This project has been discussed with the current TripleO PTL (Robert Collins) and he seemed very supportive and agreed with the organization of the project outlined above. James Slagle
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling I am not sure what is meant here by pass config options nor how it would be done prior to scheduling; can you please clarify? I do not imagine Kubernetes would *choose* the config values, K8s does not know anything about configuring OpenStack. Before scheduling, there is no running container to pass anything to. and once scheduled, Kubernetes would be responsible for connecting the various containers together. Kubernetes has a limited role in connecting containers together. K8s creates the networking environment in which the containers *can* communicate, and passes environment variables into containers telling them from what protocol://host:port/ to import each imported endpoint. Kubernetes creates a universal reverse proxy on each minion, to provide endpoints that do not vary as the servers move around. It is up to stuff outside Kubernetes to decide what should be connected to what, and it is up to the containers to read the environment variables and actually connect. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Steven Dake's message of 2014-09-24 20:02:49 -0700: On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. Agreed. Containers are a special kind of workload, not a VM. However, I do think that a container manager could be taught manage vms and vice versa. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling and once scheduled, Kubernetes would be responsible for connecting the various containers together. I like it. This is just good old fashioned encapsulation, finally applied to ops. I also like that it rides on top of OpenStack entirely, and doesn't have to be integrated. However, this does make me think that Keystone domains should be exposable to services inside your cloud for use as SSO. It would be quite handy if the keystone users used for the VMs that host Kubernetes could use the same credentials to manage the containers. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling I am not sure what is meant here by pass config options nor how it would be done prior to scheduling; can you please clarify? I do not imagine Kubernetes would *choose* the config values, K8s does not know anything about configuring OpenStack. Before scheduling, there is no running container to pass anything to. Docker containers tend to use environment variables passed to the initial command to configure things. The Kubernetes API allows setting these environment variables on creation of the container. and once scheduled, Kubernetes would be responsible for connecting the various containers together. Kubernetes has a limited role in connecting containers together. K8s creates the networking environment in which the containers *can* communicate, and passes environment variables into containers telling them from what protocol://host:port/ to import each imported endpoint. Kubernetes creates a universal reverse proxy on each minion, to provide endpoints that do not vary as the servers move around. It is up to stuff outside Kubernetes to decide what should be connected to what, and it is up to the containers to read the environment variables and actually connect. This is a nice simple interface though, and I like that it is narrowly defined, not trying to be anything that containers want to share with other containers. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM: Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: ... ... Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. In what sense would Kubernetes manage baremetal (at all)? By from a single host do you mean that a client on one host can manage remote baremetal and containers? I can see that Kubernetes allows a client on one host to get containers placed remotely --- but so does the Docker driver for Nova. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. Clint, in another branch of this email tree you referred to the VMs that host Kubernetes. How does that square with Steve's text that seems to imply bare metal minions? I can see that some people have had much more detailed design discussions than I have yet found. Perhaps it would be helpful to share an organized presentation of the design thoughts in more detail. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. So I am still not clear on what Steve is trying to say is the main use case. Kubernetes is even farther from balancing among PDUs than Nova is. At least Nova has a framework in which this issue can be posed and solved. I mean a framework that actually can carry the necessary information. The Kubernetes scheduler interface is extremely impoverished in the information it passes and it uses GO structs --- which, like C structs, can not be subclassed. Nova's filter scheduler includes a fatal bug that bites when balancing and you want more than one element per area, see https://bugs.launchpad.net/nova/+bug/1373478. However: (a) you might not need more than one element per area and (b) fixing that bug is a much smaller job than expanding the mind of K8s. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
All hail our container overlords! -Rob On 24 September 2014 10:40, Steven Dake sd...@redhat.com wrote: Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long term, as we experiment we believe it is best to hold a separate weekly one hour IRC meeting on Mondays at 2000 UTC in #openstack-meeting [3]. This project has been discussed with the current TripleO PTL (Robert Collins) and he seemed very supportive and agreed with the organization of the project outlined above. James Slagle, a TripleO core developer, has kindly offered to liase between Kolla and the broader TripleO community. I personally feel it is necessary to start from a nearly empty repository when kicking off a new project. As a result, there is limited code in the repository [4] at this time. I suspect folks will start cranking out a kick-ass implementation once the Kolla/Stackforge integration support is reviewed by the infra team [5]. The initial core team is composed of Steven Dake, Ryan Hallisey, James Lebocki, Jeff Peeler, James Slagle, Lars Kellogg-Sedman, and David Vossel. The core team will be reviewed every 6 weeks to add fresh developers. Please join the core team in designing and inventing this rockin' new technology! Regards -steve ~~ [1] https://github.com/docker/docker [2] https://github.com/GoogleCloudPlatform/kubernetes [3] https://wiki.openstack.org/wiki/Meetings/Kolla [4] https://github.com/jlabocki/superhappyfunshow [5] https://review.openstack.org/#/c/122972/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
I'm interested in how this relates/conflicts with the TripleO goal of using OpenStack to deploy OpenStack. It looks like (maybe just superficially) that Kubernetes is simply a combination of (nova + docker driver) = container schedualer and (heat) = orchestration. They both schedule containers, will need advanced scheduling like ensure these two containers are on different servers (nova ServerGroups), autoscale resources, hook up things together, have a json document that describes the desired state, etc... If that's the case, it seems odd to use an OpenStack competing product to deploy a competitor of Kubernetes. Two software stacks to learn how to debug rather then just one. Maybe I'm just totally misunderstanding what Kubernetes is trying to accomplish though. I'm not trying to stur up trouble here. I just really want to understand how these two technologies fit together. Thanks, Kevin From: Steven Dake [sd...@redhat.com] Sent: Tuesday, September 23, 2014 3:40 PM To: OpenStack Development Mailing List Subject: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker Hi folks, I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long term, as we experiment we believe it is best to hold a separate weekly one hour IRC meeting on Mondays at 2000 UTC in #openstack-meeting [3]. This project has been discussed with the current TripleO PTL (Robert Collins) and he seemed very supportive and agreed with the organization of the project outlined above. James Slagle, a TripleO core developer, has kindly offered to liase between Kolla and the broader TripleO community. I personally feel it is necessary to start from a nearly empty repository when kicking off a new project. As a result, there is limited code in the repository [4] at this time. I suspect folks will start cranking out a kick-ass implementation once the Kolla/Stackforge integration support is reviewed by the infra team [5]. The initial core team is composed of Steven Dake, Ryan Hallisey, James Lebocki, Jeff Peeler, James Slagle, Lars Kellogg-Sedman, and David Vossel. The core team will be reviewed every 6 weeks to add fresh developers. Please join the core team in designing and inventing this rockin' new technology! Regards -steve ~~ [1] https://github.com/docker/docker [2] https://github.com/GoogleCloudPlatform/kubernetes [3] https://wiki.openstack.org/wiki/Meetings/Kolla [4] https://github.com/jlabocki/superhappyfunshow [5] https://review.openstack.org/#/c/122972/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/23/2014 05:38 PM, Fox, Kevin M wrote: I'm interested in how this relates/conflicts with the TripleO goal of using OpenStack to deploy OpenStack. It looks like (maybe just superficially) that Kubernetes is simply a combination of (nova + docker driver) = container schedualer and (heat) = orchestration. They both schedule containers, will need advanced scheduling like ensure these two containers are on different servers (nova ServerGroups), autoscale resources, hook up things together, have a json document that describes the desired state, etc... If that's the case, it seems odd to use an OpenStack competing product to deploy a competitor of Kubernetes. Two software stacks to learn how to debug rather then just one. Kevin, Thanks for the feedback. There are two orthogonal points you address re competitiveness. One is the deployment program (which Kolla intends to be a part of). The deployment program includes an implementation (tripleo). TripleO is focused around using OpenStack to deploy OpenStack. Kolla is focused around using Kubernetes to deploy OpenStack. But they both fit into the same program, and at some point they may even be remerged into both using OpenStack to deploy OpenStack. Time will tell. IMO Kubernetes is not competitive with OpenStack. The way in which the Kolla project uses them is in fact complimentary. In a perfect world OpenStack's container service (Magnum) + Heat could be used instead of Kubernetes. The problem with that approach is the container service for OpenStack is not functional and not integrated into the release. It is indeed true that another software stack must be learned. We hope to abstract most/all of the differences so the actual maintenance difference (ie what must be learned) presents a small learning footprint. Maybe I'm just totally misunderstanding what Kubernetes is trying to accomplish though. I'm not trying to stur up trouble here. I just really want to understand how these two technologies fit together. I don't see you stirring up trouble :) Essentially this project proposes an alternative method for deploying OpenStack (ie not using OpenStack, but using Kubernetes). I did run the idea by Robert Collins (current TripleO PTL) first before we got cracking on the code base. He indicated the approach was worth experimenting with. Regards -steve Thanks, Kevin *From:* Steven Dake [sd...@redhat.com] *Sent:* Tuesday, September 23, 2014 3:40 PM *To:* OpenStack Development Mailing List *Subject:* [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker *Hi folks,*** * I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long term, as we experiment we believe it is best to hold a separate weekly one hour IRC meeting on Mondays at 2000 UTC in #openstack-meeting [3]. This project has been discussed with the current TripleO PTL (Robert Collins) and he seemed very supportive and agreed with the organization of the project outlined above. James Slagle, a TripleO core developer, has kindly offered to liase between Kolla and the broader TripleO community. I personally feel it is necessary to start from a nearly empty repository when kicking off a new project. As a result, there is limited code in the repository [4] at this time. I suspect folks will start cranking out a kick-ass implementation once the Kolla/Stackforge integration support is reviewed by the infra team [5]. The initial core team is composed of Steven Dake, Ryan Hallisey, James Lebocki, Jeff Peeler, James Slagle, Lars Kellogg-Sedman, and David Vossel. The core team will be reviewed every 6 weeks to add fresh developers. Please join the core team
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
I don't know if anyone else has noticed, but you can not install Cinder inside a container. Cinder requires an iSCSI package that fails to install; its install script tries to launch the daemon, and that fails. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Tue, Sep 23, 2014 at 7:06 PM, Steven Dake sd...@redhat.com wrote: On 09/23/2014 05:38 PM, Fox, Kevin M wrote: I'm interested in how this relates/conflicts with the TripleO goal of using OpenStack to deploy OpenStack. It looks like (maybe just superficially) that Kubernetes is simply a combination of (nova + docker driver) = container schedualer and (heat) = orchestration. They both schedule containers, will need advanced scheduling like ensure these two containers are on different servers (nova ServerGroups), autoscale resources, hook up things together, have a json document that describes the desired state, etc... If that's the case, it seems odd to use an OpenStack competing product to deploy a competitor of Kubernetes. Two software stacks to learn how to debug rather then just one. Kevin, Thanks for the feedback. There are two orthogonal points you address re competitiveness. One is the deployment program (which Kolla intends to be a part of). The deployment program includes an implementation (tripleo). TripleO is focused around using OpenStack to deploy OpenStack. Kolla is focused around using Kubernetes to deploy OpenStack. But they both fit into the same program, and at some point they may even be remerged into both using OpenStack to deploy OpenStack. Time will tell. IMO Kubernetes is not competitive with OpenStack. The way in which the Kolla project uses them is in fact complimentary. In a perfect world OpenStack's container service (Magnum) + Heat could be used instead of Kubernetes. The problem with that approach is the container service for OpenStack is not functional and not integrated into the release. It is indeed true that another software stack must be learned. We hope to abstract most/all of the differences so the actual maintenance difference (ie what must be learned) presents a small learning footprint. Maybe I'm just totally misunderstanding what Kubernetes is trying to accomplish though. I'm not trying to stur up trouble here. I just really want to understand how these two technologies fit together. I don't see you stirring up trouble :) Essentially this project proposes an alternative method for deploying OpenStack (ie not using OpenStack, but using Kubernetes). I did run the idea by Robert Collins (current TripleO PTL) first before we got cracking on the code base. He indicated the approach was worth experimenting with. So I think it's a cool idea and worth looking at as you have said. But I'm very confused by your statements, it seems to me that there's a misunderstanding, either in what Triple'O is, or something else entirely. Given that Triple'O stands for Openstack On Openstack I'm not sure how you separate the OpenStack piece from the project? Don't get me wrong, I'm certainly not saying one way is better than the other etc. just that the statements here are a bit confusing tome. Also, it seems REALLY strange that Triple'O hasn't even graduated and my limited understanding is that it still has a ways to go and we're proposing alternate implementations of it. Very odd IMO. That being said, I'd love to see some details on what you have in mind here. I don't necessarily see why it needs to be an OpenStack Project per-say as opposed to a really cool Open Source Project for deploying OpenStack containers (or whatever it is exactly that you have in mind). Regards -steve Thanks, Kevin -- *From:* Steven Dake [sd...@redhat.com] *Sent:* Tuesday, September 23, 2014 3:40 PM *To:* OpenStack Development Mailing List *Subject:* [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker *Hi folks,* * I'm pleased to announce the development of a new project Kolla which is Greek for glue :). Kolla has a goal of providing an implementation that deploys OpenStack using Kubernetes and Docker. This project will begin as a StackForge project separate from the TripleO/Deployment program code base. Our long term goal is to merge into the TripleO/Deployment program rather then create a new program. Docker is a container technology for delivering hermetically sealed applications and has about 620 technical contributors [1]. We intend to produce docker images for a variety of platforms beginning with Fedora 20. We are completely open to any distro support, so if folks want to add new Linux distribution to Kolla please feel free to submit patches :) Kubernetes at the most basic level is a Docker scheduler produced by and used within Google [2]. Kubernetes has in excess of 100 technical contributors. Kubernetes is more then just a scheduler, it provides additional functionality such as load balancing and scaling and has a significant roadmap. The #tripleo channel on Freenode will be used for Kolla developer and user communication. Even though we plan to become part of the Deployment program long
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On Tue, Sep 23, 2014 at 7:30 PM, Mike Spreitzer mspre...@us.ibm.com wrote: I don't know if anyone else has noticed, but you can not install Cinder inside a container. Cinder requires an iSCSI package that fails to install; its install script tries to launch the daemon, and that fails. Yes, certainly have. I've had discussions with a number of folks on the topic of how to solve the storage problem but I think people are focused on bigger goals at the moment, including myself. Maybe that's something we can change in the coming months. Or maybe somebody has made some progress and I just haven't heard about it. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/23/2014 06:30 PM, Mike Spreitzer wrote: I don't know if anyone else has noticed, but you can not install Cinder inside a container. Cinder requires an iSCSI package that fails to install; its install script tries to launch the daemon, and that fails. Regards, Mike Mike, Mordred pointed that out late last week - that essentially iscsi + containers are bust. This is something that needs fixing in upstream kernel.org. I don't know the details, but I assume at some point we will need to figure out what the problem is and sort out a solution and beg the upstream to fix it ;-). Regards -steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Ok, so perhaps I need to be a bit more verbose here. TripleO already had a plan to use docker for some services via Nova, but that ran into some issues; specifically running two computes (ironic and docker) on a single machine plays havoc with the assumption in Neutron that the compute hostname shall be the same as the neutron agent queue name in oslo,messaging, and that breaks vif plugging.. Dan Prince was driving a discussion with nova core about the best way to address that. Doing docker for everything, as the base layer is interesting - how does one deploy the machines and do their base network? Is that still Ironic and Neutron? There's a bunch of learning that needs to be done to answer those questions and the best way to learn is to experiment. There are lots of possible outcomes, and I don't want to pre-judge them! Now, with respect to the 'and must use openstack components where one exists angle' - well the big tent model is quite clearly weakening what that means [with the presumed exception of Ring 0] - but Ironic isn't in Ring 0 in any of the design sketches - so its not at all clear to me whether we as a group want to keep enforcing that, or perhaps be much more inclusive and say 'pick good components that do the job'. So again, lets see what happens and how it pans out: the folk hacking on this are already well established 'stackers, we need have no fear that they are going to ignore the capabilities present in the OpenStack ecosystem: much more interesting to me will be to see what things they /can/ reuse, and what they cannot. Finally, yes, docker/lxc containers and iscsi don't mix very well today as netlink isn't namespaced - we spent a bunch of time early on in TripleO investigating this, before shelving it as a 'later' problem:- if there is an answer and this team finds it - great! If not, they'll need some way to deploy the iscsi using components outside of docker... and that may bring us full circle back to deploying images via Ironic rather than docker - or it may not! we'll just have to see :) -Rob On 24 September 2014 13:30, Mike Spreitzer mspre...@us.ibm.com wrote: I don't know if anyone else has noticed, but you can not install Cinder inside a container. Cinder requires an iSCSI package that fails to install; its install script tries to launch the daemon, and that fails. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/23/2014 07:15 PM, John Griffith wrote: On Tue, Sep 23, 2014 at 7:06 PM, Steven Dake sd...@redhat.com mailto:sd...@redhat.com wrote: On 09/23/2014 05:38 PM, Fox, Kevin M wrote: I'm interested in how this relates/conflicts with the TripleO goal of using OpenStack to deploy OpenStack. It looks like (maybe just superficially) that Kubernetes is simply a combination of (nova + docker driver) = container schedualer and (heat) = orchestration. They both schedule containers, will need advanced scheduling like ensure these two containers are on different servers (nova ServerGroups), autoscale resources, hook up things together, have a json document that describes the desired state, etc... If that's the case, it seems odd to use an OpenStack competing product to deploy a competitor of Kubernetes. Two software stacks to learn how to debug rather then just one. Kevin, Thanks for the feedback. There are two orthogonal points you address re competitiveness. One is the deployment program (which Kolla intends to be a part of). The deployment program includes an implementation (tripleo). TripleO is focused around using OpenStack to deploy OpenStack. Kolla is focused around using Kubernetes to deploy OpenStack. But they both fit into the same program, and at some point they may even be remerged into both using OpenStack to deploy OpenStack. Time will tell. IMO Kubernetes is not competitive with OpenStack. The way in which the Kolla project uses them is in fact complimentary. In a perfect world OpenStack's container service (Magnum) + Heat could be used instead of Kubernetes. The problem with that approach is the container service for OpenStack is not functional and not integrated into the release. It is indeed true that another software stack must be learned. We hope to abstract most/all of the differences so the actual maintenance difference (ie what must be learned) presents a small learning footprint. Maybe I'm just totally misunderstanding what Kubernetes is trying to accomplish though. I'm not trying to stur up trouble here. I just really want to understand how these two technologies fit together. I don't see you stirring up trouble :) Essentially this project proposes an alternative method for deploying OpenStack (ie not using OpenStack, but using Kubernetes). I did run the idea by Robert Collins (current TripleO PTL) first before we got cracking on the code base. He indicated the approach was worth experimenting with. So I think it's a cool idea and worth looking at as you have said. But I'm very confused by your statements, it seems to me that there's a misunderstanding, either in what Triple'O is, or something else entirely. Given that Triple'O stands for Openstack On Openstack I'm not sure how you separate the OpenStack piece from the project? Don't get me wrong, I'm certainly not saying one way is better than the other etc. just that the statements here are a bit confusing tome. John, There is a deployment program - tripleo is just one implementation. We went through this with Heat and various projects that want to extend heat (eg Murano) and one big mistake I think Murano folks made was not figuring out where there code would go prior to writing it. I'm only making a statement as to where I think it should belong. Also, it seems REALLY strange that Triple'O hasn't even graduated and my limited understanding is that it still has a ways to go and we're proposing alternate implementations of it. Very odd IMO. Our goal is deploying OpenStack using containers. TripleO could have this same goal, but at the present it does not (I could be mistaken here, please feel free to correct if I am incorrect). It rather prefers to deploy on bare metal. We are just focusing on this particular point, (openstack in containers on bare metal). I spoke with Robert for quite awhile about integration time and we were both in agreement early or late integration is not a concern for us - getting something working for containers seemed more compelling. That being said, I'd love to see some details on what you have in mind here. I don't necessarily see why it needs to be an OpenStack Project per-say as opposed to a really cool Open Source Project for deploying OpenStack containers (or whatever it is exactly that you have in mind). It doesn't have to necessarily go into the deployment program. My main motivation at this point for using stackforge and attaching it to OpenStack is I desperately want to use the OpenStack workflow since our workflow rocks! Regards -steve Regards -steve Thanks, Kevin *From:* Steven Dake [sd...@redhat.com mailto:sd...@redhat.com] *Sent:* Tuesday,
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
There are other options. 3 docker containers with ceph in them perhaps. Kevin From: Steven Dake Sent: Tuesday, September 23, 2014 7:21:39 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker On 09/23/2014 06:30 PM, Mike Spreitzer wrote: I don't know if anyone else has noticed, but you can not install Cinder inside a container. Cinder requires an iSCSI package that fails to install; its install script tries to launch the daemon, and that fails. Regards, Mike Mike, Mordred pointed that out late last week - that essentially iscsi + containers are bust. This is something that needs fixing in upstream kernel.org. I don't know the details, but I assume at some point we will need to figure out what the problem is and sort out a solution and beg the upstream to fix it ;-). Regards -steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
On 09/23/2014 10:29 PM, Steven Dake wrote: There is a deployment program - tripleo is just one implementation. Nope, that is not correct. Like it or not (I personally don't), Triple-O is *the* Deployment Program for OpenStack: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n284 Saying Triple-O is just one implementation of a deployment program is like saying Heat is just one implementation of an orchestration program. It isn't. It's *the* implemenation of an orchestration program that has been blessed by the TC: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n112 We went through this with Heat and various projects that want to extend heat (eg Murano) and one big mistake I think Murano folks made was not figuring out where there code would go prior to writing it. I'm only making a statement as to where I think it should belong. Sorry, I have to call you to task on this. You think it was a mistake for the Murano folks to not figure out where the code would go prior to writing it? For the record, Murano existed nearly 2 years ago, as a response to various customer requests. Having the ability to properly deploy Windows applications like SQL Server and Active Directory into an OpenStack cloud was more important to the Murano developers than trying to predict what the whims of the OpenStack developer and governance model would be months or years down the road. Tell me, did any of Heat's code exist prior to deciding to propose it for incubation? Saying that Murano developers should have thought about where their code would live is holding them to a higher standard than any of the other developer communities. Did folks working on disk-image-builder pre-validate with the TC or the mailing list that the dib code would live in the triple-o program? No, of course not. It was developed naturally and then placed into the program that fit it best. Murano was developed naturally in exactly the same way, and the Murano developers have been nothing but accommodating to every request made of them by the TC (and those requests have been entirely different over the last 18 months, ranging from split it out to just propose another program) and by the PTLs for projects that requested they split various parts of Murano out into existing programs. The Murano developers have done no power grab, have deliberately tried to be as community-focused and amenable to all requests as possible, and yet they are treated with disdain by a number of folks in the core Heat developer community, including yourself, Clint and Zane. And honestly, I don't get it... all Murano is doing is generating Heat templates and trying to fill in some pieces that Heat isn't interested in doing. I don't see why there is so much animosity towards a project that has, to my knowledge, acted in precisely the ways that we've asked projects to act in the OpenStack community: with openness, transparency, and community good will. Sorry to be so blunt, but this has been weighing on me. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
