Re: [Openstack-operators] Case studies on Openstack HA architecture

[Curtis]

On Mon, Aug 28, 2017 at 6:24 PM, Imtiaz Chowdhury
 wrote:
> Thanks Curtis, Robert, David and Mohammed for your responses.
>
> As a follow up question, do you use any deployment automation tools for 
> setting up the HA control plane? I can see the value of deploying each 
> service in separate virtual environment or containers but automating such 
> deployment requires developing some new tools. Openstack-ansible is one 
> potential deployment tool that I am aware of but that had limited support 
> CentOS.
>

I think it's safe to say that all "distros" and "installers" for
openstack will converge on using containers of some kind. As to what
and how that is actually done, there are a lot of opinions, and I
think you would have to perform your own due diligence around what
makes sense to you in terms of technologies and implementations. I
don't think there is a right answer. :)

To me it seems to be boiling down to: 1) Kubernetes is overkill (and
use something simpler) or 2) If you don't use Kubernetes then you will
just end up building it yourself [1]. I don't think it's actually that
simple, but those are the talking points.

I think, as has been mentioned in this thread so far, the major
opensource openstack "installers" or "managers" (?) are
openstack-ansible, kolla, tripleo, and now openstack-helm. Not sure if
I'm missing any, sorry if I did.

Frankly I find this all pretty interesting. :)

Thanks,
Curtis.

[1]: http://blog.reactiveops.com/is-kubernetes-overkill


> Imtiaz
>
> On 8/28/17, 2:23 PM, "Curtis"  wrote:
>
> On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
>  wrote:
> > Hi Openstack operators,
> >
> >
> >
> > Most Openstack HA deployment use 3 node database cluster, 3 node 
> rabbitMQ
> > cluster and 3 Controllers. I am wondering whether there are any studies 
> done
> > that show the pros and cons of co-locating database and messaging 
> service
> > with the Openstack control services.  In other words, I am very 
> interested
> > in learning about advantages and disadvantages, in terms of ease of
> > deployment, upgrade and overall API performance, of having 3 all-in-one
> > Openstack controller over a more distributed deployment model.
>
> I'm not aware of any actual case studies, but this is the (current)
> default model for tripleo and its downstream product, so there would
> be a lot of deployments like this out there in the wild. In the
> default deployment everything but compute is on these 3 nodes running
> on the physical OS.
>
> Do you mean 3 physical servers with everything running on the physical OS?
>
> My opinion is that 3 physical nodes to run all the control plane
> services is quite common, but in custom deployments I either run vms
> and containers on those or just containers. I'd use at least lxc to
> segregate services into their own containers.
>
> I would also suggest that using those same physical servers as
> north/south "network nodes" (which you probably don't have as I
> believe workday is a big opencontrail user) or hosts for stateful
> metric systems (ie. mongodb) can cause issues performance wise, but
> co-located mysql/galera and rabbit on the same nodes as the rest of
> the openstack control plane hasn't been a problem for me yet, but with
> containers I could split them out fairly easily if needed.
>
> Thanks,
> Curtis.
>
> >
> >
> >
> > References to any work done in this area will be highly appreciated.
> >
> >
> >
> > Thanks,
> > Imtiaz
> >
> >
> >
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc=
> >
>
>
>
> --
> Blog: serverascode.com
>
>



-- 
Blog: serverascode.com

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[Van Leeuwen, Robert]

> Thanks Curtis, Robert, David and Mohammed for your responses. 
>As a follow up question, do you use any deployment automation tools for 
> setting up the HA control plane?
>  I can see the value of deploying each service in separate virtual 
> environment or containers but automating such deployment requires developing 
> some new tools. Openstack-ansible is one potential deployment tool that I am 
> aware of but that had limited support CentOS. 

Here we currently have a physical loadbalancer which provides the ALL the HA 
logic. 
The services are installed on vms managed by Puppet.
IMHO a loadbalancer is the right place to solve HA since you only have to do it 
once. 
(Depending on your Neutron implementation you might also need something for 
Neutron) 

To rant a bit about deployment/automation:
I am not necessarily a fan of management with puppet since module dependencies 
can become a nightmare even with things like R10K.
e.g. you want to deploy a newer version of keystone, this requires a newer 
openstack-common puppet module. 
Now you have a change that affects everything (other OpenStack puppet modules 
also use openstack-common) and might now need upgrades as well.

To solve these kind of things we are looking at containers and investigated two 
possible deployment scenario’s:
OpenStack helm (+containers build by kolla) which is pretty nice and uses k8s.
The problem is that it is still early days for both k8s and helm. 
Things that stuck out most:
* Helm: Nice for a deployment from scratch. 
   Integration with our environment is a bit of a pain (e.g. if you want to 
start with just one service)
   It would need a lot of work to above into the product and not everything 
would be easy to get into upstream.
   Still a very early implementation needs quite a bit of TLC. 
If you can live with what comes out of the box it might be a nice solution.
* K8S: It is a relatively complex product and it is still missing some features 
especially for self-hosted installations.

After some deliberation, we decided to go with the “hashi” stack (with kola 
build containers). 
This stack has more of a unix philosophy, simple processes that do 1 thing well:
* Nomad –scheduling 
* Consul - service discovery and KV store
* Vault – secret management
* Fabio zeroconf loadbalancer which integrates with Consul
In general this stack is really easy to understand for everyone. (just work 
with it half a day and you really understand what is going on under the hood)
There are no overlay networks :)
Lots of the stuff can break without much impact. E.g. Nomad is only relevant 
when you want to start/stop containers it can crash or turned off the rest of 
the time.
Another pro for this is that there is a significant amount of knowledge around 
these products in house.

To give an example on complexity: if you look at deployment of k8s itself and 
the hashi stack: 
* deployment of k8s with kargo: you have a very large playblook which takes 30 
minutes to run to setup a cluster.
* deployment of the hashi stuff: is just one binary for each component with 1 
config file basically done in a few minutes if even that.


Cheers,
Robert van Leeuwen

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[Fox, Kevin M]

kolla has various containerization tools. one based on ansible, another based 
on kubernetes.

From: Imtiaz Chowdhury [imtiaz.chowdh...@workday.com]
Sent: Monday, August 28, 2017 5:24 PM
To: Curtis
Cc: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] Case studies on Openstack HA architecture

Thanks Curtis, Robert, David and Mohammed for your responses.

As a follow up question, do you use any deployment automation tools for setting 
up the HA control plane? I can see the value of deploying each service in 
separate virtual environment or containers but automating such deployment 
requires developing some new tools. Openstack-ansible is one potential 
deployment tool that I am aware of but that had limited support CentOS.

Imtiaz

On 8/28/17, 2:23 PM, "Curtis" <serverasc...@gmail.com> wrote:

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
<imtiaz.chowdh...@workday.com> wrote:
> Hi Openstack operators,
>
>
>
> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
> cluster and 3 Controllers. I am wondering whether there are any studies 
done
> that show the pros and cons of co-locating database and messaging service
> with the Openstack control services.  In other words, I am very interested
> in learning about advantages and disadvantages, in terms of ease of
> deployment, upgrade and overall API performance, of having 3 all-in-one
> Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.

>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> 
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc=
>



--
Blog: serverascode.com


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[Imtiaz Chowdhury]

Thanks Curtis, Robert, David and Mohammed for your responses. 

As a follow up question, do you use any deployment automation tools for setting 
up the HA control plane? I can see the value of deploying each service in 
separate virtual environment or containers but automating such deployment 
requires developing some new tools. Openstack-ansible is one potential 
deployment tool that I am aware of but that had limited support CentOS. 

Imtiaz

On 8/28/17, 2:23 PM, "Curtis"  wrote:

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
 wrote:
> Hi Openstack operators,
>
>
>
> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
> cluster and 3 Controllers. I am wondering whether there are any studies 
done
> that show the pros and cons of co-locating database and messaging service
> with the Openstack control services.  In other words, I am very interested
> in learning about advantages and disadvantages, in terms of ease of
> deployment, upgrade and overall API performance, of having 3 all-in-one
> Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.

>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> 
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=FzzkP-wZtxwR_lHv11gV2RDLNwSuEtI-Ttdh3mloOUA=byM_0ToLQ8JCji20rvyQJYr-Zm4pHsZ5TK4CFkuZbbk=mLtaaDedoNufBf-kCreLrdZ-McNRAuuesR3xWIT76Vc=
 
>



-- 
Blog: serverascode.com


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[Curtis]

On Fri, Aug 25, 2017 at 6:11 PM, Imtiaz Chowdhury
 wrote:
> Hi Openstack operators,
>
>
>
> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
> cluster and 3 Controllers. I am wondering whether there are any studies done
> that show the pros and cons of co-locating database and messaging service
> with the Openstack control services.  In other words, I am very interested
> in learning about advantages and disadvantages, in terms of ease of
> deployment, upgrade and overall API performance, of having 3 all-in-one
> Openstack controller over a more distributed deployment model.

I'm not aware of any actual case studies, but this is the (current)
default model for tripleo and its downstream product, so there would
be a lot of deployments like this out there in the wild. In the
default deployment everything but compute is on these 3 nodes running
on the physical OS.

Do you mean 3 physical servers with everything running on the physical OS?

My opinion is that 3 physical nodes to run all the control plane
services is quite common, but in custom deployments I either run vms
and containers on those or just containers. I'd use at least lxc to
segregate services into their own containers.

I would also suggest that using those same physical servers as
north/south "network nodes" (which you probably don't have as I
believe workday is a big opencontrail user) or hosts for stateful
metric systems (ie. mongodb) can cause issues performance wise, but
co-located mysql/galera and rabbit on the same nodes as the rest of
the openstack control plane hasn't been a problem for me yet, but with
containers I could split them out fairly easily if needed.

Thanks,
Curtis.

>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



-- 
Blog: serverascode.com

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[Van Leeuwen, Robert]

>
> Hi Openstack operators,
>
> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ 
> cluster and 3 Controllers.
> I am wondering whether there are any studies done that show the pros and cons 
> of co-locating database and messaging service with the Openstack control 
> services.
> In other words, I am very interested in learning about advantages and 
> disadvantages, in terms of ease of deployment,
> upgrade and overall API performance, of having 3 all-in-one Openstack 
> controller over a more distributed deployment model.

In general, a host with less running services is easier to troubleshoot.
However, there is no fundamental issue with sharing mysql/rabbit on a single 
controller assuming the host is quick enough.

For the OpenStack API services I would highly recommend “splitting” things up.
It does not necessarily need to be on different physical hosts but at least use 
something to prevent package dependencies between the different OpenStack 
components (virtualenv/containers/vms).
If you do not do this it makes upgrades of an individual component on a node 
near impossible.

Cheers,
Robert van Leeuwen
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[Mohammed Naser]

We've also always done the same.  We deploy any cloud with a minimum of 3 
(quorum, etc) but when the database or message queue needs to scale up, the 
other components probably need to as well so making an equal replica of 
everything across the environment. 

Sent from my iPhone

> On Aug 26, 2017, at 9:13 AM, David Medberry  wrote:
> 
> I'm not aware of any studies as per se, but we have long run rabbitmq, MySQL, 
> and all the API endpoints on the same three nodes. 
> 
> 
> 
>> On Aug 25, 2017 6:12 PM, "Imtiaz Chowdhury"  
>> wrote:
>> Hi Openstack operators,
>> 
>>  
>> 
>> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ 
>> cluster and 3 Controllers. I am wondering whether there are any studies done 
>> that show the pros and cons of co-locating database and messaging service 
>> with the Openstack control services.  In other words, I am very interested 
>> in learning about advantages and disadvantages, in terms of ease of 
>> deployment, upgrade and overall API performance, of having 3 all-in-one 
>> Openstack controller over a more distributed deployment model.
>> 
>>  
>> 
>> References to any work done in this area will be highly appreciated.
>> 
>>  
>> 
>> Thanks, 
>> Imtiaz
>> 
>>  
>> 
>> 
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Case studies on Openstack HA architecture

[David Medberry]

I'm not aware of any studies as per se, but we have long run rabbitmq,
MySQL, and all the API endpoints on the same three nodes.



On Aug 25, 2017 6:12 PM, "Imtiaz Chowdhury" 
wrote:

> Hi Openstack operators,
>
>
>
> Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ
> cluster and 3 Controllers. I am wondering whether there are any studies
> done that show the pros and cons of co-locating database and messaging
> service with the Openstack control services.  In other words, I am very
> interested in learning about advantages and disadvantages, in terms of ease
> of deployment, upgrade and overall API performance, of having 3 all-in-one
> Openstack controller over a more distributed deployment model.
>
>
>
> References to any work done in this area will be highly appreciated.
>
>
>
> Thanks,
> Imtiaz
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Case studies on Openstack HA architecture

[Imtiaz Chowdhury]

Hi Openstack operators,

Most Openstack HA deployment use 3 node database cluster, 3 node rabbitMQ 
cluster and 3 Controllers. I am wondering whether there are any studies done 
that show the pros and cons of co-locating database and messaging service with 
the Openstack control services.  In other words, I am very interested in 
learning about advantages and disadvantages, in terms of ease of deployment, 
upgrade and overall API performance, of having 3 all-in-one Openstack 
controller over a more distributed deployment model.

References to any work done in this area will be highly appreciated.

Thanks,
Imtiaz

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators