Re: [openstack-dev] [kolla] Multi-Regions Support
- Mail original - > De: "Jay Pipes" <jaypi...@gmail.com> > À: openstack-dev@lists.openstack.org > Envoyé: Vendredi 6 Janvier 2017 21:42:46 > Objet: Re: [openstack-dev] [kolla] Multi-Regions Support > > On 01/06/2017 03:23 PM, Sam Yaple wrote: > > This should be read as MariaDB+Galera for replication. It is a > > highly-available database. > > Don't get me wrong. I love me some Galera. :) However, what the poster > is really working towards is an implementation of the VCPE and eVCPE use > cases for ETSI NFV. These use cases require a highly distributed compute > fabric that can withstand long disruptions in network connectivity > (between POPs/COs and the last mile of network service) while still > being able to service compute and network functions at the customer premise. > > Galera doesn't tolerate network disruption of any significant length of > time. At all. If there is a Keystone services running on the customer > premise that is connecting to a Galera database, and that Galera > database's connectivity to its peers is disrupted, down goes the whole > on-premise cloud fabric. And that's exactly what I believe the original > poster is attempting to avoid. Thus my not understanding the choice here. > Jay, you are thinking too far ;) The goal of this thread is to see how Kolla can deploy a multi region scenario. Remarks/contributions to progress in that direction is the goal of the initial post. Best, Matt > Best, > -jay > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] Multi-Regions Support
I was kind of hoping k2k federation would solve that. one keystone per region to provide a local keystone to talk to, and a centeral keystone users authenticate with. Just waiting for horizon to gain support before trying though. Thanks, Kevin From: Jay Pipes [jaypi...@gmail.com] Sent: Friday, January 06, 2017 12:42 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [kolla] Multi-Regions Support On 01/06/2017 03:23 PM, Sam Yaple wrote: > This should be read as MariaDB+Galera for replication. It is a > highly-available database. Don't get me wrong. I love me some Galera. :) However, what the poster is really working towards is an implementation of the VCPE and eVCPE use cases for ETSI NFV. These use cases require a highly distributed compute fabric that can withstand long disruptions in network connectivity (between POPs/COs and the last mile of network service) while still being able to service compute and network functions at the customer premise. Galera doesn't tolerate network disruption of any significant length of time. At all. If there is a Keystone services running on the customer premise that is connecting to a Galera database, and that Galera database's connectivity to its peers is disrupted, down goes the whole on-premise cloud fabric. And that's exactly what I believe the original poster is attempting to avoid. Thus my not understanding the choice here. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] Multi-Regions Support
On 01/06/2017 03:23 PM, Sam Yaple wrote: This should be read as MariaDB+Galera for replication. It is a highly-available database. Don't get me wrong. I love me some Galera. :) However, what the poster is really working towards is an implementation of the VCPE and eVCPE use cases for ETSI NFV. These use cases require a highly distributed compute fabric that can withstand long disruptions in network connectivity (between POPs/COs and the last mile of network service) while still being able to service compute and network functions at the customer premise. Galera doesn't tolerate network disruption of any significant length of time. At all. If there is a Keystone services running on the customer premise that is connecting to a Galera database, and that Galera database's connectivity to its peers is disrupted, down goes the whole on-premise cloud fabric. And that's exactly what I believe the original poster is attempting to avoid. Thus my not understanding the choice here. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] Multi-Regions Support
On Fri, Jan 6, 2017 at 8:01 PM, Jay Pipeswrote: > On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote: > >> Hello, >> >> TL;DR: We make a multi-regions deployment with Kolla. It requires to >> patch the code a little bit, and you can find the diff on our >> GitHub[1]. This patch is just a first attempt to support multi-regions >> in Kolla and it raises questions. Some modifications are not done in >> an idiomatic way and we do not expect this to be merged in Kolla. The >> reminder of this mail explains our patch and states our questions. >> >> At Inria/Discovery[2], we evaluate OpenStack at scale for the >> Performance Working Group. So far, we focus on one single OpenStack >> region deployment with hundreds of computes and we always go with >> Kolla for our deployment. Over the last few days, we tried to achieve >> a multi-regions OpenStack deployment with Kolla. We want to share with >> you our current deployment workflow, patches we had to apply on Kolla >> to support multi-regions, and also ask you if we do things correctly. >> >> First of all, our multi-regions deployment follows the one described >> by the OpenStack documentation[3]. >> > > I don't see an "Admin Region" as part of the OpenStack documentation for > multi-region deployment. I also see LDAP mentioned as the recommended > authentication/IdM store. > > > Concretely, the deployment > >> considers /one/ Administrative Region (AR) that contains Keystone and >> Horizon. >> > > That's not a region. Those should be shared resources *across* regions. > > > This is a Kolla-based deployment, so Keystone is hidden > >> behind an HAProxy, and has MariaDB and memcached as backend. >> > > I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL > system REDIS"? But here, you're using MariaDB -- a non-distributed database > -- for the Keystone component which is the very thing that is the most > highly distributed of all state storage in OpenStack. > This should be read as MariaDB+Galera for replication. It is a highly-available database. > > So, you are replacing the Nova DB (which doesn't need to be distributed at > all, since it's a centralized control plane piece) within the regions with > a "distributed" NoSQL store (and throwing away transactional safety I might > add) but you're going with a non-distributed traditional RDBMS for the very > piece that needs to be shared, distributed, and highly-available across > OpenStack. I don't understand that. > > At the > >> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full >> OpenStack, except Keystone. We got something as follows at the end of >> the deployment: >> >> Admin Region (AR): >> - control: >> * Horizon >> * HAProxy >> * Keyston >> * MariaDB >> * memcached >> > > Again, that's not a region. Those are merely shared services between > regions. > > > OpenStack Region x (OSRx): >> - control: >> * HAProxy >> * nova-api/conductor/scheduler >> * neutron-server/l3/dhcp/... >> * glance-api/registry >> * MariaDB >> * RabbitMQ >> >> - compute1: >> * nova-compute >> * neutron-agent >> >> - compute2: ... >> >> We do the deployment by running Kolla n+1 times. The first run deploys >> the Administrative Region (AR) and the other runs deploy OpenStack >> Regions (OSR). For each run, we fix the value of `openstack_region_name' >> variable to the name of the current region. >> >> In the context of multi-regions, Keystone (in the AR) should be >> available to all OSRs. This means, there are as many Keystone >> endpoints as regions. For instance, if we consider two OSRs, the >> result of listing endpoints at the end of the AR deployment looks like >> this: >> >> >> $ openstack endpoint list >> >> | Region | Serv Name | Serv Type | Interface | URL >> | >> |+---+---+---+- >> -| >> | AR | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR1 | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | OSR1 | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | OSR1 | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR2 | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | OSR2 | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | OSR2 | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> > > There shouldn't be an AR region. If the Keystone authentication domain is > indeed shared between OpenStack regions, then an administrative user should > be able to hit any Keystone endpoint in any OpenStack region and add > users/projects/roles, etc. to the shared Keystone data store (or if using > LDAP, the admin should be able to add
Re: [openstack-dev] [kolla] Multi-Regions Support
On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote: Hello, TL;DR: We make a multi-regions deployment with Kolla. It requires to patch the code a little bit, and you can find the diff on our GitHub[1]. This patch is just a first attempt to support multi-regions in Kolla and it raises questions. Some modifications are not done in an idiomatic way and we do not expect this to be merged in Kolla. The reminder of this mail explains our patch and states our questions. At Inria/Discovery[2], we evaluate OpenStack at scale for the Performance Working Group. So far, we focus on one single OpenStack region deployment with hundreds of computes and we always go with Kolla for our deployment. Over the last few days, we tried to achieve a multi-regions OpenStack deployment with Kolla. We want to share with you our current deployment workflow, patches we had to apply on Kolla to support multi-regions, and also ask you if we do things correctly. First of all, our multi-regions deployment follows the one described by the OpenStack documentation[3]. I don't see an "Admin Region" as part of the OpenStack documentation for multi-region deployment. I also see LDAP mentioned as the recommended authentication/IdM store. > Concretely, the deployment considers /one/ Administrative Region (AR) that contains Keystone and Horizon. That's not a region. Those should be shared resources *across* regions. > This is a Kolla-based deployment, so Keystone is hidden behind an HAProxy, and has MariaDB and memcached as backend. I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL system REDIS"? But here, you're using MariaDB -- a non-distributed database -- for the Keystone component which is the very thing that is the most highly distributed of all state storage in OpenStack. So, you are replacing the Nova DB (which doesn't need to be distributed at all, since it's a centralized control plane piece) within the regions with a "distributed" NoSQL store (and throwing away transactional safety I might add) but you're going with a non-distributed traditional RDBMS for the very piece that needs to be shared, distributed, and highly-available across OpenStack. I don't understand that. > At the same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full OpenStack, except Keystone. We got something as follows at the end of the deployment: Admin Region (AR): - control: * Horizon * HAProxy * Keyston * MariaDB * memcached Again, that's not a region. Those are merely shared services between regions. OpenStack Region x (OSRx): - control: * HAProxy * nova-api/conductor/scheduler * neutron-server/l3/dhcp/... * glance-api/registry * MariaDB * RabbitMQ - compute1: * nova-compute * neutron-agent - compute2: ... We do the deployment by running Kolla n+1 times. The first run deploys the Administrative Region (AR) and the other runs deploy OpenStack Regions (OSR). For each run, we fix the value of `openstack_region_name' variable to the name of the current region. In the context of multi-regions, Keystone (in the AR) should be available to all OSRs. This means, there are as many Keystone endpoints as regions. For instance, if we consider two OSRs, the result of listing endpoints at the end of the AR deployment looks like this: $ openstack endpoint list | Region | Serv Name | Serv Type | Interface | URL | |+---+---+---+--| | AR | keystone | identity | public| http://10.24.63.248:5000/v3 | | AR | keystone | identity | internal | http://10.24.63.248:5000/v3 | | AR | keystone | identity | admin | http://10.24.63.248:35357/v3 | | OSR1 | keystone | identity | public| http://10.24.63.248:5000/v3 | | OSR1 | keystone | identity | internal | http://10.24.63.248:5000/v3 | | OSR1 | keystone | identity | admin | http://10.24.63.248:35357/v3 | | OSR2 | keystone | identity | public| http://10.24.63.248:5000/v3 | | OSR2 | keystone | identity | internal | http://10.24.63.248:5000/v3 | | OSR2 | keystone | identity | admin | http://10.24.63.248:35357/v3 | There shouldn't be an AR region. If the Keystone authentication domain is indeed shared between OpenStack regions, then an administrative user should be able to hit any Keystone endpoint in any OpenStack region and add users/projects/roles, etc. to the shared Keystone data store (or if using LDAP, the admin should be able to add a user to ActiveDirectory/ApacheDS in any OpenStack region and have that user information immediately show up in any of the other regions). Best, -jay This requires patching the `keystone/tasks/register.yml' play[4] to re-execute the `Creating admin project, user, role, service, and endpoint' task for all regions we consider. An example of such a patch is given on our GitHub[5]. In this example, the `openstack_regions' variable
Re: [openstack-dev] [kolla] Multi-Regions Support
Hello, Thanks for your feedback. > The original value is this: > auth_uri = {{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{ > keystone_public_port }} > > Kolla does SSL via haproxy, not directly at the service itself. So internal > traffic is http, external is https. In this case, 'kolla_internal_fqdn' > points to the haproxy vip. Got it, but if you look at the default definition of `keystone_internal_url', inside `group_vars/all.yml', you have: 332 keystone_internal_url: "{{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{ keystone_public_port }}/v3" As you can see, the definition is close to the original value of `auth_uri' (except the v3). So, it seems to be a better idea to set `auth_uri' to `keystone_internal_url' instead of the canonical form. Especially in the context of multi-regions. In this context, you cannot change the value of `{{ kolla_internal_fqdn }}' because, as you say, it has to target the region's HAProxy vip. However, you have to set the value of `auth_uri' to target the Administrative Region HAProxy vip. So, by defining in nova/neutron/glance conf: [keystone_authtoken] auth_uri = {{ keystone_internal_url }} auth_url = {{ keystone_admin_url }} Then you can target the Administrative Region HAProxy vip by redefining `keystone_*_url' without redefining `kolla_internal_fqdn'. > Its not legacy code, but I am sure it can be changed to work multiregion in > the way you are attempting to do it. I was talking about legacy code because, in the stable/newton branch, the `auth_uri' is already set to `{{ keystone_internal_url }}', but solely for Kubernetes engine[1]. Shall we proceed by pushing to Gerrit with a dedicated page in the documentation? Also, our patch is based on stable/newton, is it better if we follow the kolla-ansible master? [1] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L151 On Thu, Jan 5, 2017 at 7:01 PM, Sam Yaplewrote: > On Thu, Jan 5, 2017 at 2:12 PM, Ronan-Alexandre Cherrueau > wrote: >> >> Hello, >> >> >> TL;DR: We make a multi-regions deployment with Kolla. It requires to >> patch the code a little bit, and you can find the diff on our >> GitHub[1]. This patch is just a first attempt to support multi-regions >> in Kolla and it raises questions. Some modifications are not done in >> an idiomatic way and we do not expect this to be merged in Kolla. The >> reminder of this mail explains our patch and states our questions. >> >> >> At Inria/Discovery[2], we evaluate OpenStack at scale for the >> Performance Working Group. So far, we focus on one single OpenStack >> region deployment with hundreds of computes and we always go with >> Kolla for our deployment. Over the last few days, we tried to achieve >> a multi-regions OpenStack deployment with Kolla. We want to share with >> you our current deployment workflow, patches we had to apply on Kolla >> to support multi-regions, and also ask you if we do things correctly. >> >> First of all, our multi-regions deployment follows the one described >> by the OpenStack documentation[3]. Concretely, the deployment >> considers /one/ Administrative Region (AR) that contains Keystone and >> Horizon. This is a Kolla-based deployment, so Keystone is hidden >> behind an HAProxy, and has MariaDB and memcached as backend. At the >> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full >> OpenStack, except Keystone. We got something as follows at the end of >> the deployment: >> >> >> Admin Region (AR): >> - control: >> * Horizon >> * HAProxy >> * Keyston >> * MariaDB >> * memcached >> >> OpenStack Region x (OSRx): >> - control: >> * HAProxy >> * nova-api/conductor/scheduler >> * neutron-server/l3/dhcp/... >> * glance-api/registry >> * MariaDB >> * RabbitMQ >> >> - compute1: >> * nova-compute >> * neutron-agent >> >> - compute2: ... >> >> >> We do the deployment by running Kolla n+1 times. The first run deploys >> the Administrative Region (AR) and the other runs deploy OpenStack >> Regions (OSR). For each run, we fix the value of `openstack_region_name' >> variable to the name of the current region. >> >> In the context of multi-regions, Keystone (in the AR) should be >> available to all OSRs. This means, there are as many Keystone >> endpoints as regions. For instance, if we consider two OSRs, the >> result of listing endpoints at the end of the AR deployment looks like >> this: >> >> >> $ openstack endpoint list >> >> | Region | Serv Name | Serv Type | Interface | URL >> | >> >> |+---+---+---+--| >> | AR | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR1 |
Re: [openstack-dev] [kolla] Multi-Regions Support
On Thu, Jan 5, 2017 at 2:12 PM, Ronan-Alexandre Cherrueau < ronan-alexandre.cherru...@inria.fr> wrote: > Hello, > > > TL;DR: We make a multi-regions deployment with Kolla. It requires to > patch the code a little bit, and you can find the diff on our > GitHub[1]. This patch is just a first attempt to support multi-regions > in Kolla and it raises questions. Some modifications are not done in > an idiomatic way and we do not expect this to be merged in Kolla. The > reminder of this mail explains our patch and states our questions. > > > At Inria/Discovery[2], we evaluate OpenStack at scale for the > Performance Working Group. So far, we focus on one single OpenStack > region deployment with hundreds of computes and we always go with > Kolla for our deployment. Over the last few days, we tried to achieve > a multi-regions OpenStack deployment with Kolla. We want to share with > you our current deployment workflow, patches we had to apply on Kolla > to support multi-regions, and also ask you if we do things correctly. > > First of all, our multi-regions deployment follows the one described > by the OpenStack documentation[3]. Concretely, the deployment > considers /one/ Administrative Region (AR) that contains Keystone and > Horizon. This is a Kolla-based deployment, so Keystone is hidden > behind an HAProxy, and has MariaDB and memcached as backend. At the > same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full > OpenStack, except Keystone. We got something as follows at the end of > the deployment: > > > Admin Region (AR): > - control: > * Horizon > * HAProxy > * Keyston > * MariaDB > * memcached > > OpenStack Region x (OSRx): > - control: > * HAProxy > * nova-api/conductor/scheduler > * neutron-server/l3/dhcp/... > * glance-api/registry > * MariaDB > * RabbitMQ > > - compute1: > * nova-compute > * neutron-agent > > - compute2: ... > > > We do the deployment by running Kolla n+1 times. The first run deploys > the Administrative Region (AR) and the other runs deploy OpenStack > Regions (OSR). For each run, we fix the value of `openstack_region_name' > variable to the name of the current region. > > In the context of multi-regions, Keystone (in the AR) should be > available to all OSRs. This means, there are as many Keystone > endpoints as regions. For instance, if we consider two OSRs, the > result of listing endpoints at the end of the AR deployment looks like > this: > > > $ openstack endpoint list > > | Region | Serv Name | Serv Type | Interface | URL > | > |+---+---+---+-- > | > | AR | keystone | identity | public| > http://10.24.63.248:5000/v3 | > | AR | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | AR | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > | OSR1 | keystone | identity | public| > http://10.24.63.248:5000/v3 | > | OSR1 | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | OSR1 | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > | OSR2 | keystone | identity | public| > http://10.24.63.248:5000/v3 | > | OSR2 | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | OSR2 | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > > > This requires patching the `keystone/tasks/register.yml' play[4] to > re-execute the `Creating admin project, user, role, service, and > endpoint' task for all regions we consider. An example of such a patch > is given on our GitHub[5]. In this example, the `openstack_regions' > variable is a list that contains the name of all regions (see [6]). As > a drawback, the patch implies to know in advance all OSR. A better > implementation would execute the `Creating admin project, user, role, > service, and endpoint' task every time a new OSR is going to be > deployed. But this requires to move the task somewhere else in the > Kolla workflow and we have no idea where this should be. > > In the AR, we also have to change the Horizon configuration file to > handle multi-regions[7]. The modification could be done easily and > idiomatically by setting the `node_custome_config' variable to the > `multi-regions' directory[8] and benefits from Kolla merging config > system. > > Also, deploying OSRs requires patching the kolla-toolbox as it seems > not region-aware. In particular, patch the `kolla_keystone_service.py' > module[9] that is responsible for contacting Keystone and creating a > new endpoint when we register a new OpenStack service. > > > 73 for _endpoint in cloud.keystone_client.endpoints.list(): > 74 if _endpoint.service_id == service.id and \ > 75 _endpoint.interface == interface: > 76endpoint = _endpoint > 77if endpoint.url != url: > 78 changed = True > 79 cloud.keystone_client.endpoints.update( > 80endpoint, url=url) >
Re: [openstack-dev] [kolla] Multi-Regions Support
Great stuff! Thank you Ronan. I'd love to see this guide refactored and submitted to our docs (also take a longer look how to make full fledged support in kolla tree). Looking for volunteers:) On 5 January 2017 at 07:59, Jeffrey Zhangwrote: > Thanks Ronan, > > Great approach. > > I am always expecting multi-region support in Kolla. I hope what you did can > be merged into Kolla. > > > > On Thu, Jan 5, 2017 at 10:12 PM, Ronan-Alexandre Cherrueau > wrote: >> >> Hello, >> >> >> TL;DR: We make a multi-regions deployment with Kolla. It requires to >> patch the code a little bit, and you can find the diff on our >> GitHub[1]. This patch is just a first attempt to support multi-regions >> in Kolla and it raises questions. Some modifications are not done in >> an idiomatic way and we do not expect this to be merged in Kolla. The >> reminder of this mail explains our patch and states our questions. >> >> >> At Inria/Discovery[2], we evaluate OpenStack at scale for the >> Performance Working Group. So far, we focus on one single OpenStack >> region deployment with hundreds of computes and we always go with >> Kolla for our deployment. Over the last few days, we tried to achieve >> a multi-regions OpenStack deployment with Kolla. We want to share with >> you our current deployment workflow, patches we had to apply on Kolla >> to support multi-regions, and also ask you if we do things correctly. >> >> First of all, our multi-regions deployment follows the one described >> by the OpenStack documentation[3]. Concretely, the deployment >> considers /one/ Administrative Region (AR) that contains Keystone and >> Horizon. This is a Kolla-based deployment, so Keystone is hidden >> behind an HAProxy, and has MariaDB and memcached as backend. At the >> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full >> OpenStack, except Keystone. We got something as follows at the end of >> the deployment: >> >> >> Admin Region (AR): >> - control: >> * Horizon >> * HAProxy >> * Keyston >> * MariaDB >> * memcached >> >> OpenStack Region x (OSRx): >> - control: >> * HAProxy >> * nova-api/conductor/scheduler >> * neutron-server/l3/dhcp/... >> * glance-api/registry >> * MariaDB >> * RabbitMQ >> >> - compute1: >> * nova-compute >> * neutron-agent >> >> - compute2: ... >> >> >> We do the deployment by running Kolla n+1 times. The first run deploys >> the Administrative Region (AR) and the other runs deploy OpenStack >> Regions (OSR). For each run, we fix the value of `openstack_region_name' >> variable to the name of the current region. >> >> In the context of multi-regions, Keystone (in the AR) should be >> available to all OSRs. This means, there are as many Keystone >> endpoints as regions. For instance, if we consider two OSRs, the >> result of listing endpoints at the end of the AR deployment looks like >> this: >> >> >> $ openstack endpoint list >> >> | Region | Serv Name | Serv Type | Interface | URL >> | >> >> |+---+---+---+--| >> | AR | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR1 | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | OSR1 | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | OSR1 | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR2 | keystone | identity | public| >> http://10.24.63.248:5000/v3 | >> | OSR2 | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | OSR2 | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> >> >> This requires patching the `keystone/tasks/register.yml' play[4] to >> re-execute the `Creating admin project, user, role, service, and >> endpoint' task for all regions we consider. An example of such a patch >> is given on our GitHub[5]. In this example, the `openstack_regions' >> variable is a list that contains the name of all regions (see [6]). As >> a drawback, the patch implies to know in advance all OSR. A better >> implementation would execute the `Creating admin project, user, role, >> service, and endpoint' task every time a new OSR is going to be >> deployed. But this requires to move the task somewhere else in the >> Kolla workflow and we have no idea where this should be. >> >> In the AR, we also have to change the Horizon configuration file to >> handle multi-regions[7]. The modification could be done easily and >> idiomatically by setting the `node_custome_config' variable to the >> `multi-regions' directory[8] and benefits from Kolla merging config >> system. >> >> Also, deploying OSRs requires patching the kolla-toolbox as it seems >> not region-aware. In particular, patch
Re: [openstack-dev] [kolla] Multi-Regions Support
Thanks Ronan, Great approach. I am always expecting multi-region support in Kolla. I hope what you did can be merged into Kolla. On Thu, Jan 5, 2017 at 10:12 PM, Ronan-Alexandre Cherrueau < ronan-alexandre.cherru...@inria.fr> wrote: > Hello, > > > TL;DR: We make a multi-regions deployment with Kolla. It requires to > patch the code a little bit, and you can find the diff on our > GitHub[1]. This patch is just a first attempt to support multi-regions > in Kolla and it raises questions. Some modifications are not done in > an idiomatic way and we do not expect this to be merged in Kolla. The > reminder of this mail explains our patch and states our questions. > > > At Inria/Discovery[2], we evaluate OpenStack at scale for the > Performance Working Group. So far, we focus on one single OpenStack > region deployment with hundreds of computes and we always go with > Kolla for our deployment. Over the last few days, we tried to achieve > a multi-regions OpenStack deployment with Kolla. We want to share with > you our current deployment workflow, patches we had to apply on Kolla > to support multi-regions, and also ask you if we do things correctly. > > First of all, our multi-regions deployment follows the one described > by the OpenStack documentation[3]. Concretely, the deployment > considers /one/ Administrative Region (AR) that contains Keystone and > Horizon. This is a Kolla-based deployment, so Keystone is hidden > behind an HAProxy, and has MariaDB and memcached as backend. At the > same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full > OpenStack, except Keystone. We got something as follows at the end of > the deployment: > > > Admin Region (AR): > - control: > * Horizon > * HAProxy > * Keyston > * MariaDB > * memcached > > OpenStack Region x (OSRx): > - control: > * HAProxy > * nova-api/conductor/scheduler > * neutron-server/l3/dhcp/... > * glance-api/registry > * MariaDB > * RabbitMQ > > - compute1: > * nova-compute > * neutron-agent > > - compute2: ... > > > We do the deployment by running Kolla n+1 times. The first run deploys > the Administrative Region (AR) and the other runs deploy OpenStack > Regions (OSR). For each run, we fix the value of `openstack_region_name' > variable to the name of the current region. > > In the context of multi-regions, Keystone (in the AR) should be > available to all OSRs. This means, there are as many Keystone > endpoints as regions. For instance, if we consider two OSRs, the > result of listing endpoints at the end of the AR deployment looks like > this: > > > $ openstack endpoint list > > | Region | Serv Name | Serv Type | Interface | URL > | > |+---+---+---+-- > | > | AR | keystone | identity | public| > http://10.24.63.248:5000/v3 | > | AR | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | AR | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > | OSR1 | keystone | identity | public| > http://10.24.63.248:5000/v3 | > | OSR1 | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | OSR1 | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > | OSR2 | keystone | identity | public| > http://10.24.63.248:5000/v3 | > | OSR2 | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | OSR2 | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > > > This requires patching the `keystone/tasks/register.yml' play[4] to > re-execute the `Creating admin project, user, role, service, and > endpoint' task for all regions we consider. An example of such a patch > is given on our GitHub[5]. In this example, the `openstack_regions' > variable is a list that contains the name of all regions (see [6]). As > a drawback, the patch implies to know in advance all OSR. A better > implementation would execute the `Creating admin project, user, role, > service, and endpoint' task every time a new OSR is going to be > deployed. But this requires to move the task somewhere else in the > Kolla workflow and we have no idea where this should be. > > In the AR, we also have to change the Horizon configuration file to > handle multi-regions[7]. The modification could be done easily and > idiomatically by setting the `node_custome_config' variable to the > `multi-regions' directory[8] and benefits from Kolla merging config > system. > > Also, deploying OSRs requires patching the kolla-toolbox as it seems > not region-aware. In particular, patch the `kolla_keystone_service.py' > module[9] that is responsible for contacting Keystone and creating a > new endpoint when we register a new OpenStack service. > > > 73 for _endpoint in cloud.keystone_client.endpoints.list(): > 74 if _endpoint.service_id == service.id and \ > 75 _endpoint.interface == interface: > 76endpoint = _endpoint > 77if
[openstack-dev] [kolla] Multi-Regions Support
Hello, TL;DR: We make a multi-regions deployment with Kolla. It requires to patch the code a little bit, and you can find the diff on our GitHub[1]. This patch is just a first attempt to support multi-regions in Kolla and it raises questions. Some modifications are not done in an idiomatic way and we do not expect this to be merged in Kolla. The reminder of this mail explains our patch and states our questions. At Inria/Discovery[2], we evaluate OpenStack at scale for the Performance Working Group. So far, we focus on one single OpenStack region deployment with hundreds of computes and we always go with Kolla for our deployment. Over the last few days, we tried to achieve a multi-regions OpenStack deployment with Kolla. We want to share with you our current deployment workflow, patches we had to apply on Kolla to support multi-regions, and also ask you if we do things correctly. First of all, our multi-regions deployment follows the one described by the OpenStack documentation[3]. Concretely, the deployment considers /one/ Administrative Region (AR) that contains Keystone and Horizon. This is a Kolla-based deployment, so Keystone is hidden behind an HAProxy, and has MariaDB and memcached as backend. At the same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full OpenStack, except Keystone. We got something as follows at the end of the deployment: Admin Region (AR): - control: * Horizon * HAProxy * Keyston * MariaDB * memcached OpenStack Region x (OSRx): - control: * HAProxy * nova-api/conductor/scheduler * neutron-server/l3/dhcp/... * glance-api/registry * MariaDB * RabbitMQ - compute1: * nova-compute * neutron-agent - compute2: ... We do the deployment by running Kolla n+1 times. The first run deploys the Administrative Region (AR) and the other runs deploy OpenStack Regions (OSR). For each run, we fix the value of `openstack_region_name' variable to the name of the current region. In the context of multi-regions, Keystone (in the AR) should be available to all OSRs. This means, there are as many Keystone endpoints as regions. For instance, if we consider two OSRs, the result of listing endpoints at the end of the AR deployment looks like this: $ openstack endpoint list | Region | Serv Name | Serv Type | Interface | URL | |+---+---+---+--| | AR | keystone | identity | public| http://10.24.63.248:5000/v3 | | AR | keystone | identity | internal | http://10.24.63.248:5000/v3 | | AR | keystone | identity | admin | http://10.24.63.248:35357/v3 | | OSR1 | keystone | identity | public| http://10.24.63.248:5000/v3 | | OSR1 | keystone | identity | internal | http://10.24.63.248:5000/v3 | | OSR1 | keystone | identity | admin | http://10.24.63.248:35357/v3 | | OSR2 | keystone | identity | public| http://10.24.63.248:5000/v3 | | OSR2 | keystone | identity | internal | http://10.24.63.248:5000/v3 | | OSR2 | keystone | identity | admin | http://10.24.63.248:35357/v3 | This requires patching the `keystone/tasks/register.yml' play[4] to re-execute the `Creating admin project, user, role, service, and endpoint' task for all regions we consider. An example of such a patch is given on our GitHub[5]. In this example, the `openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute the `Creating admin project, user, role, service, and endpoint' task every time a new OSR is going to be deployed. But this requires to move the task somewhere else in the Kolla workflow and we have no idea where this should be. In the AR, we also have to change the Horizon configuration file to handle multi-regions[7]. The modification could be done easily and idiomatically by setting the `node_custome_config' variable to the `multi-regions' directory[8] and benefits from Kolla merging config system. Also, deploying OSRs requires patching the kolla-toolbox as it seems not region-aware. In particular, patch the `kolla_keystone_service.py' module[9] that is responsible for contacting Keystone and creating a new endpoint when we register a new OpenStack service. 73 for _endpoint in cloud.keystone_client.endpoints.list(): 74 if _endpoint.service_id == service.id and \ 75 _endpoint.interface == interface: 76endpoint = _endpoint 77if endpoint.url != url: 78 changed = True 79 cloud.keystone_client.endpoints.update( 80endpoint, url=url) 81break 82 else: 83changed = True 84cloud.keystone_client.endpoints.create( 85 service=service.id, 86 url=url, 87 interface=interface, 88 region=endpoint_region) At some point, this module /create/ or /update/ a service endpoint. It first tests if the service