Re: [Openstack-operators] Case studies on Openstack HA architecture
On Mon, Aug 28, 2017 at 6:24 PM, Imtiaz Chowdhurywrote: > Thanks Curtis, Robert, David and Mohammed for your responses. > > As a follow up question, do you use any deployment automation tools for > setting up the HA control plane? I can see the value of deploying each > service in separate virtual environment or containers but automating such > deployment requires developing some new tools. Openstack-ansible is one > potential deployment tool that I am aware of but that had limited support > CentOS. > I think it's safe to say that all "distros" and "installers" for openstack will converge on using containers of some kind. As to what and how that is actually done, there are a lot of opinions, and I think you would have to perform your own due diligence around what makes sense to you in terms of technologies and implementations. I don't think there is a right answer. :) To me it seems to be boiling down to: 1) Kubernetes is overkill (and use something simpler) or 2) If you don't use Kubernetes then you will just end up building it yourself [1]. I don't think it's actually that simple, but those are the talking points. I think, as has been mentioned in this thread so far, the major opensource openstack "installers" or "managers" (?) are openstack-ansible, kolla, tripleo, and now openstack-helm. Not sure if I'm missing any, sorry if I did. Frankly I find this all pretty interesting. :) Thanks, Curtis. [1]: http://blog.reactiveops.com/is-kubernetes-overkill > Imtiaz > > On 8/28/17, 2:23 PM, "Curtis" wrote: > > On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury > wrote: > > Hi Openstack operators, > > > > > > > > Most Openstack HA deployment use 3 node database cluster, 3 node > rabbitMQ > > cluster and 3 Controllers. I am wondering whether there are any studies > done > > that show the pros and cons of co-locating database and messaging > service > > with the Openstack control services. In other words, I am very > interested > > in learning about advantages and disadvantages, in terms of ease of > > deployment, upgrade and overall API performance, of having 3 all-in-one > > Openstack controller over a more distributed deployment model. > > I'm not aware of any actual case studies, but this is the (current) > default model for tripleo and its downstream product, so there would > be a lot of deployments like this out there in the wild. In the > default deployment everything but compute is on these 3 nodes running > on the physical OS. > > Do you mean 3 physical servers with everything running on the physical OS? > > My opinion is that 3 physical nodes to run all the control plane > services is quite common, but in custom deployments I either run vms > and containers on those or just containers. I'd use at least lxc to > segregate services into their own containers. > > I would also suggest that using those same physical servers as > north/south "network nodes" (which you probably don't have as I > believe workday is a big opencontrail user) or hosts for stateful > metric systems (ie. mongodb) can cause issues performance wise, but > co-located mysql/galera and rabbit on the same nodes as the rest of > the openstack control plane hasn't been a problem for me yet, but with > containers I could split them out fairly easily if needed. > > Thanks, > Curtis. > > > > > > > > > References to any work done in this area will be highly appreciated. > > > > > > > > Thanks, > > Imtiaz > > > > > > > > > > ___ > > OpenStack-operators mailing list > > OpenStack-operators@lists.openstack.org > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc= > > > > > > -- > Blog: serverascode.com > > -- Blog: serverascode.com ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
> Thanks Curtis, Robert, David and Mohammed for your responses. >As a follow up question, do you use any deployment automation tools for > setting up the HA control plane? > I can see the value of deploying each service in separate virtual > environment or containers but automating such deployment requires developing > some new tools. Openstack-ansible is one potential deployment tool that I am > aware of but that had limited support CentOS. Here we currently have a physical loadbalancer which provides the ALL the HA logic. The services are installed on vms managed by Puppet. IMHO a loadbalancer is the right place to solve HA since you only have to do it once. (Depending on your Neutron implementation you might also need something for Neutron) To rant a bit about deployment/automation: I am not necessarily a fan of management with puppet since module dependencies can become a nightmare even with things like R10K. e.g. you want to deploy a newer version of keystone, this requires a newer openstack-common puppet module. Now you have a change that affects everything (other OpenStack puppet modules also use openstack-common) and might now need upgrades as well. To solve these kind of things we are looking at containers and investigated two possible deployment scenario’s: OpenStack helm (+containers build by kolla) which is pretty nice and uses k8s. The problem is that it is still early days for both k8s and helm. Things that stuck out most: * Helm: Nice for a deployment from scratch. Integration with our environment is a bit of a pain (e.g. if you want to start with just one service) It would need a lot of work to above into the product and not everything would be easy to get into upstream. Still a very early implementation needs quite a bit of TLC. If you can live with what comes out of the box it might be a nice solution. * K8S: It is a relatively complex product and it is still missing some features especially for self-hosted installations. After some deliberation, we decided to go with the “hashi” stack (with kola build containers). This stack has more of a unix philosophy, simple processes that do 1 thing well: * Nomad –scheduling * Consul - service discovery and KV store * Vault – secret management * Fabio zeroconf loadbalancer which integrates with Consul In general this stack is really easy to understand for everyone. (just work with it half a day and you really understand what is going on under the hood) There are no overlay networks :) Lots of the stuff can break without much impact. E.g. Nomad is only relevant when you want to start/stop containers it can crash or turned off the rest of the time. Another pro for this is that there is a significant amount of knowledge around these products in house. To give an example on complexity: if you look at deployment of k8s itself and the hashi stack: * deployment of k8s with kargo: you have a very large playblook which takes 30 minutes to run to setup a cluster. * deployment of the hashi stuff: is just one binary for each component with 1 config file basically done in a few minutes if even that. Cheers, Robert van Leeuwen ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
kolla has various containerization tools. one based on ansible, another based on kubernetes. From: Imtiaz Chowdhury [imtiaz.chowdh...@workday.com] Sent: Monday, August 28, 2017 5:24 PM To: Curtis Cc: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] Case studies on Openstack HA architecture Thanks Curtis, Robert, David and Mohammed for your responses. As a follow up question, do you use any deployment automation tools for setting up the HA control plane? I can see the value of deploying each service in separate virtual environment or containers but automating such deployment requires developing some new tools. Openstack-ansible is one potential deployment tool that I am aware of but that had limited support CentOS. Imtiaz On 8/28/17, 2:23 PM, "Curtis" <serverasc...@gmail.com> wrote: On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury <imtiaz.chowdh...@workday.com> wrote: > Hi Openstack operators, > > > > Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ > cluster and 3 Controllers. I am wondering whether there are any studies done > that show the pros and cons of co-locating database and messaging service > with the Openstack control services. In other words, I am very interested > in learning about advantages and disadvantages, in terms of ease of > deployment, upgrade and overall API performance, of having 3 all-in-one > Openstack controller over a more distributed deployment model. I'm not aware of any actual case studies, but this is the (current) default model for tripleo and its downstream product, so there would be a lot of deployments like this out there in the wild. In the default deployment everything but compute is on these 3 nodes running on the physical OS. Do you mean 3 physical servers with everything running on the physical OS? My opinion is that 3 physical nodes to run all the control plane services is quite common, but in custom deployments I either run vms and containers on those or just containers. I'd use at least lxc to segregate services into their own containers. I would also suggest that using those same physical servers as north/south "network nodes" (which you probably don't have as I believe workday is a big opencontrail user) or hosts for stateful metric systems (ie. mongodb) can cause issues performance wise, but co-located mysql/galera and rabbit on the same nodes as the rest of the openstack control plane hasn't been a problem for me yet, but with containers I could split them out fairly easily if needed. Thanks, Curtis. > > > > References to any work done in this area will be highly appreciated. > > > > Thanks, > Imtiaz > > > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc= > -- Blog: serverascode.com ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
Thanks Curtis, Robert, David and Mohammed for your responses. As a follow up question, do you use any deployment automation tools for setting up the HA control plane? I can see the value of deploying each service in separate virtual environment or containers but automating such deployment requires developing some new tools. Openstack-ansible is one potential deployment tool that I am aware of but that had limited support CentOS. Imtiaz On 8/28/17, 2:23 PM, "Curtis"wrote: On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury wrote: > Hi Openstack operators, > > > > Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ > cluster and 3 Controllers. I am wondering whether there are any studies done > that show the pros and cons of co-locating database and messaging service > with the Openstack control services. In other words, I am very interested > in learning about advantages and disadvantages, in terms of ease of > deployment, upgrade and overall API performance, of having 3 all-in-one > Openstack controller over a more distributed deployment model. I'm not aware of any actual case studies, but this is the (current) default model for tripleo and its downstream product, so there would be a lot of deployments like this out there in the wild. In the default deployment everything but compute is on these 3 nodes running on the physical OS. Do you mean 3 physical servers with everything running on the physical OS? My opinion is that 3 physical nodes to run all the control plane services is quite common, but in custom deployments I either run vms and containers on those or just containers. I'd use at least lxc to segregate services into their own containers. I would also suggest that using those same physical servers as north/south "network nodes" (which you probably don't have as I believe workday is a big opencontrail user) or hosts for stateful metric systems (ie. mongodb) can cause issues performance wise, but co-located mysql/galera and rabbit on the same nodes as the rest of the openstack control plane hasn't been a problem for me yet, but with containers I could split them out fairly easily if needed. Thanks, Curtis. > > > > References to any work done in this area will be highly appreciated. > > > > Thanks, > Imtiaz > > > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc= > -- Blog: serverascode.com ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhurywrote: > Hi Openstack operators, > > > > Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ > cluster and 3 Controllers. I am wondering whether there are any studies done > that show the pros and cons of co-locating database and messaging service > with the Openstack control services. In other words, I am very interested > in learning about advantages and disadvantages, in terms of ease of > deployment, upgrade and overall API performance, of having 3 all-in-one > Openstack controller over a more distributed deployment model. I'm not aware of any actual case studies, but this is the (current) default model for tripleo and its downstream product, so there would be a lot of deployments like this out there in the wild. In the default deployment everything but compute is on these 3 nodes running on the physical OS. Do you mean 3 physical servers with everything running on the physical OS? My opinion is that 3 physical nodes to run all the control plane services is quite common, but in custom deployments I either run vms and containers on those or just containers. I'd use at least lxc to segregate services into their own containers. I would also suggest that using those same physical servers as north/south "network nodes" (which you probably don't have as I believe workday is a big opencontrail user) or hosts for stateful metric systems (ie. mongodb) can cause issues performance wise, but co-located mysql/galera and rabbit on the same nodes as the rest of the openstack control plane hasn't been a problem for me yet, but with containers I could split them out fairly easily if needed. Thanks, Curtis. > > > > References to any work done in this area will be highly appreciated. > > > > Thanks, > Imtiaz > > > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > -- Blog: serverascode.com ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
> > Hi Openstack operators, > > Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ > cluster and 3 Controllers. > I am wondering whether there are any studies done that show the pros and cons > of co-locating database and messaging service with the Openstack control > services. > In other words, I am very interested in learning about advantages and > disadvantages, in terms of ease of deployment, > upgrade and overall API performance, of having 3 all-in-one Openstack > controller over a more distributed deployment model. In general, a host with less running services is easier to troubleshoot. However, there is no fundamental issue with sharing mysql/rabbit on a single controller assuming the host is quick enough. For the OpenStack API services I would highly recommend “splitting” things up. It does not necessarily need to be on different physical hosts but at least use something to prevent package dependencies between the different OpenStack components (virtualenv/containers/vms). If you do not do this it makes upgrades of an individual component on a node near impossible. Cheers, Robert van Leeuwen ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
We've also always done the same. We deploy any cloud with a minimum of 3 (quorum, etc) but when the database or message queue needs to scale up, the other components probably need to as well so making an equal replica of everything across the environment. Sent from my iPhone > On Aug 26, 2017, at 9:13 AM, David Medberrywrote: > > I'm not aware of any studies as per se, but we have long run rabbitmq, MySQL, > and all the API endpoints on the same three nodes. > > > >> On Aug 25, 2017 6:12 PM, "Imtiaz Chowdhury" >> wrote: >> Hi Openstack operators, >> >> >> >> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ >> cluster and 3 Controllers. I am wondering whether there are any studies done >> that show the pros and cons of co-locating database and messaging service >> with the Openstack control services. In other words, I am very interested >> in learning about advantages and disadvantages, in terms of ease of >> deployment, upgrade and overall API performance, of having 3 all-in-one >> Openstack controller over a more distributed deployment model. >> >> >> >> References to any work done in this area will be highly appreciated. >> >> >> >> Thanks, >> Imtiaz >> >> >> >> >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Case studies on Openstack HA architecture
I'm not aware of any studies as per se, but we have long run rabbitmq, MySQL, and all the API endpoints on the same three nodes. On Aug 25, 2017 6:12 PM, "Imtiaz Chowdhury"wrote: > Hi Openstack operators, > > > > Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ > cluster and 3 Controllers. I am wondering whether there are any studies > done that show the pros and cons of co-locating database and messaging > service with the Openstack control services. In other words, I am very > interested in learning about advantages and disadvantages, in terms of ease > of deployment, upgrade and overall API performance, of having 3 all-in-one > Openstack controller over a more distributed deployment model. > > > > References to any work done in this area will be highly appreciated. > > > > Thanks, > Imtiaz > > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] Case studies on Openstack HA architecture
Hi Openstack operators, Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ cluster and 3 Controllers. I am wondering whether there are any studies done that show the pros and cons of co-locating database and messaging service with the Openstack control services. In other words, I am very interested in learning about advantages and disadvantages, in terms of ease of deployment, upgrade and overall API performance, of having 3 all-in-one Openstack controller over a more distributed deployment model. References to any work done in this area will be highly appreciated. Thanks, Imtiaz ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators