On Fri, Jan 6, 2017 at 8:01 PM, Jay Pipes <jaypi...@gmail.com> wrote:
> On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote: > >> Hello, >> >> TL;DR: We make a multi-regions deployment with Kolla. It requires to >> patch the code a little bit, and you can find the diff on our >> GitHub[1]. This patch is just a first attempt to support multi-regions >> in Kolla and it raises questions. Some modifications are not done in >> an idiomatic way and we do not expect this to be merged in Kolla. The >> reminder of this mail explains our patch and states our questions. >> >> At Inria/Discovery[2], we evaluate OpenStack at scale for the >> Performance Working Group. So far, we focus on one single OpenStack >> region deployment with hundreds of computes and we always go with >> Kolla for our deployment. Over the last few days, we tried to achieve >> a multi-regions OpenStack deployment with Kolla. We want to share with >> you our current deployment workflow, patches we had to apply on Kolla >> to support multi-regions, and also ask you if we do things correctly. >> >> First of all, our multi-regions deployment follows the one described >> by the OpenStack documentation[3]. >> > > I don't see an "Admin Region" as part of the OpenStack documentation for > multi-region deployment. I also see LDAP mentioned as the recommended > authentication/IdM store. > > > Concretely, the deployment > >> considers /one/ Administrative Region (AR) that contains Keystone and >> Horizon. >> > > That's not a region. Those should be shared resources *across* regions. > > > This is a Kolla-based deployment, so Keystone is hidden > >> behind an HAProxy, and has MariaDB and memcached as backend. >> > > I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL > system REDIS"? But here, you're using MariaDB -- a non-distributed database > -- for the Keystone component which is the very thing that is the most > highly distributed of all state storage in OpenStack. > This should be read as MariaDB+Galera for replication. It is a highly-available database. > > So, you are replacing the Nova DB (which doesn't need to be distributed at > all, since it's a centralized control plane piece) within the regions with > a "distributed" NoSQL store (and throwing away transactional safety I might > add) but you're going with a non-distributed traditional RDBMS for the very > piece that needs to be shared, distributed, and highly-available across > OpenStack. I don't understand that. > > At the > >> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full >> OpenStack, except Keystone. We got something as follows at the end of >> the deployment: >> >> Admin Region (AR): >> - control: >> * Horizon >> * HAProxy >> * Keyston >> * MariaDB >> * memcached >> > > Again, that's not a region. Those are merely shared services between > regions. > > > OpenStack Region x (OSRx): >> - control: >> * HAProxy >> * nova-api/conductor/scheduler >> * neutron-server/l3/dhcp/... >> * glance-api/registry >> * MariaDB >> * RabbitMQ >> >> - compute1: >> * nova-compute >> * neutron-agent >> >> - compute2: ... >> >> We do the deployment by running Kolla n+1 times. The first run deploys >> the Administrative Region (AR) and the other runs deploy OpenStack >> Regions (OSR). For each run, we fix the value of `openstack_region_name' >> variable to the name of the current region. >> >> In the context of multi-regions, Keystone (in the AR) should be >> available to all OSRs. This means, there are as many Keystone >> endpoints as regions. For instance, if we consider two OSRs, the >> result of listing endpoints at the end of the AR deployment looks like >> this: >> >> >> $ openstack endpoint list >> >> | Region | Serv Name | Serv Type | Interface | URL >> | >> |--------+-----------+-----------+-----------+------------- >> -----------------| >> | AR | keystone | identity | public | >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | AR | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR1 | keystone | identity | public | >> http://10.24.63.248:5000/v3 | >> | OSR1 | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | OSR1 | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> | OSR2 | keystone | identity | public | >> http://10.24.63.248:5000/v3 | >> | OSR2 | keystone | identity | internal | >> http://10.24.63.248:5000/v3 | >> | OSR2 | keystone | identity | admin | >> http://10.24.63.248:35357/v3 | >> > > There shouldn't be an AR region. If the Keystone authentication domain is > indeed shared between OpenStack regions, then an administrative user should > be able to hit any Keystone endpoint in any OpenStack region and add > users/projects/roles, etc. to the shared Keystone data store (or if using > LDAP, the admin should be able to add a user to ActiveDirectory/ApacheDS in > any OpenStack region and have that user information immediately show up in > any of the other regions). > > Jay I also picked up on a bit of terminology misuse. There were a few other bits that were not great. I can't speak for everyone else, but I do share your concerns about some of this, but I was going to save those comments for gerrit reviews once the code was rebased to master. I feel confident this can all work for proper Multi-Region support, though not as-is. Thanks, SamYaple > Best, > -jay > > > >> This requires patching the `keystone/tasks/register.yml' play[4] to >> re-execute the `Creating admin project, user, role, service, and >> endpoint' task for all regions we consider. An example of such a patch >> is given on our GitHub[5]. In this example, the `openstack_regions' >> variable is a list that contains the name of all regions (see [6]). As >> a drawback, the patch implies to know in advance all OSR. A better >> implementation would execute the `Creating admin project, user, role, >> service, and endpoint' task every time a new OSR is going to be >> deployed. But this requires to move the task somewhere else in the >> Kolla workflow and we have no idea where this should be. >> >> In the AR, we also have to change the Horizon configuration file to >> handle multi-regions[7]. The modification could be done easily and >> idiomatically by setting the `node_custome_config' variable to the >> `multi-regions' directory[8] and benefits from Kolla merging config >> system. >> >> Also, deploying OSRs requires patching the kolla-toolbox as it seems >> not region-aware. In particular, patch the `kolla_keystone_service.py' >> module[9] that is responsible for contacting Keystone and creating a >> new endpoint when we register a new OpenStack service. >> >> >> 73 for _endpoint in cloud.keystone_client.endpoints.list(): >> 74 if _endpoint.service_id == service.id and \ >> 75 _endpoint.interface == interface: >> 76 endpoint = _endpoint >> 77 if endpoint.url != url: >> 78 changed = True >> 79 cloud.keystone_client.endpoints.update( >> 80 endpoint, url=url) >> 81 break >> 82 else: >> 83 changed = True >> 84 cloud.keystone_client.endpoints.create( >> 85 service=service.id, >> 86 url=url, >> 87 interface=interface, >> 88 region=endpoint_region) >> >> >> At some point, this module /create/ or /update/ a service endpoint. It >> first tests if the service is already registered, and update the URL >> if so (l. 74 to 81), or create the new endpoint in other cases (l. >> 82). Unfortunately, the test (l. 74 to 75) only looks at the service >> (e.g., glance) and the interface (e.g., public). This makes the test >> wrong because we deploy the same service many times, but into >> different regions. We have to add an extra condition to not only test >> the service but also its region. >> >> >> 74 if _endpoint.service_id == service.id and \ >> 75 _endpoint.interface == interface and \ >> 76 _endpoint.region == endpoint_region: >> >> >> Finally, when we deployed OSRs, we override the value of >> `keystone_admin_url', `keystone_internal_url' and `keystone_public_url' >> to target Keystone in AR. We also have to change the >> `[keystone_authtoken]' section of nova/neutron/glance conf[10] so that >> it uses these variables instead of the canonical form with the >> `kolla_internal_fqdn' variable[11]. In this regards, is it due to some >> legacy code that configuration files use the `kolla_internal_fqdn' >> variable instead of `keystone_*_url'? >> >> That's almost all. As you can see, handle multi-regions implies a very >> small number of modifications if you call Kolla multiple times. >> >> So, thanks for the great job Kolla team! And waiting for your >> feedback. >> >> >> >> [1] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936 >> b7cc3136dfc082935e2995a65554 >> [2] https://beyondtheclouds.github.io/ >> [3] http://docs.openstack.org/arch-design/multi-site-architecture.html >> [4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac44 >> 1505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml >> [5] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936 >> b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1 >> [6] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936 >> b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215 >> [7] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936 >> b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188 >> [8] https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7 >> cc3136dfc082935e2995a65554/multi-regions >> [9] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936 >> b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72 >> [10] https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7 >> cc3136dfc082935e2995a65554/multi-regions/global.conf >> [11] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac44 >> 1505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155 >> >> >> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev