Thanks Ronan, Great approach.
I am always expecting multi-region support in Kolla. I hope what you did can be merged into Kolla. On Thu, Jan 5, 2017 at 10:12 PM, Ronan-Alexandre Cherrueau < ronan-alexandre.cherru...@inria.fr> wrote: > Hello, > > > TL;DR: We make a multi-regions deployment with Kolla. It requires to > patch the code a little bit, and you can find the diff on our > GitHub[1]. This patch is just a first attempt to support multi-regions > in Kolla and it raises questions. Some modifications are not done in > an idiomatic way and we do not expect this to be merged in Kolla. The > reminder of this mail explains our patch and states our questions. > > > At Inria/Discovery[2], we evaluate OpenStack at scale for the > Performance Working Group. So far, we focus on one single OpenStack > region deployment with hundreds of computes and we always go with > Kolla for our deployment. Over the last few days, we tried to achieve > a multi-regions OpenStack deployment with Kolla. We want to share with > you our current deployment workflow, patches we had to apply on Kolla > to support multi-regions, and also ask you if we do things correctly. > > First of all, our multi-regions deployment follows the one described > by the OpenStack documentation[3]. Concretely, the deployment > considers /one/ Administrative Region (AR) that contains Keystone and > Horizon. This is a Kolla-based deployment, so Keystone is hidden > behind an HAProxy, and has MariaDB and memcached as backend. At the > same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full > OpenStack, except Keystone. We got something as follows at the end of > the deployment: > > > Admin Region (AR): > - control: > * Horizon > * HAProxy > * Keyston > * MariaDB > * memcached > > OpenStack Region x (OSRx): > - control: > * HAProxy > * nova-api/conductor/scheduler > * neutron-server/l3/dhcp/... > * glance-api/registry > * MariaDB > * RabbitMQ > > - compute1: > * nova-compute > * neutron-agent > > - compute2: ... > > > We do the deployment by running Kolla n+1 times. The first run deploys > the Administrative Region (AR) and the other runs deploy OpenStack > Regions (OSR). For each run, we fix the value of `openstack_region_name' > variable to the name of the current region. > > In the context of multi-regions, Keystone (in the AR) should be > available to all OSRs. This means, there are as many Keystone > endpoints as regions. For instance, if we consider two OSRs, the > result of listing endpoints at the end of the AR deployment looks like > this: > > > $ openstack endpoint list > > | Region | Serv Name | Serv Type | Interface | URL > | > |--------+-----------+-----------+-----------+-------------- > ----------------| > | AR | keystone | identity | public | > http://10.24.63.248:5000/v3 | > | AR | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | AR | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > | OSR1 | keystone | identity | public | > http://10.24.63.248:5000/v3 | > | OSR1 | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | OSR1 | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > | OSR2 | keystone | identity | public | > http://10.24.63.248:5000/v3 | > | OSR2 | keystone | identity | internal | > http://10.24.63.248:5000/v3 | > | OSR2 | keystone | identity | admin | > http://10.24.63.248:35357/v3 | > > > This requires patching the `keystone/tasks/register.yml' play[4] to > re-execute the `Creating admin project, user, role, service, and > endpoint' task for all regions we consider. An example of such a patch > is given on our GitHub[5]. In this example, the `openstack_regions' > variable is a list that contains the name of all regions (see [6]). As > a drawback, the patch implies to know in advance all OSR. A better > implementation would execute the `Creating admin project, user, role, > service, and endpoint' task every time a new OSR is going to be > deployed. But this requires to move the task somewhere else in the > Kolla workflow and we have no idea where this should be. > > In the AR, we also have to change the Horizon configuration file to > handle multi-regions[7]. The modification could be done easily and > idiomatically by setting the `node_custome_config' variable to the > `multi-regions' directory[8] and benefits from Kolla merging config > system. > > Also, deploying OSRs requires patching the kolla-toolbox as it seems > not region-aware. In particular, patch the `kolla_keystone_service.py' > module[9] that is responsible for contacting Keystone and creating a > new endpoint when we register a new OpenStack service. > > > 73 for _endpoint in cloud.keystone_client.endpoints.list(): > 74 if _endpoint.service_id == service.id and \ > 75 _endpoint.interface == interface: > 76 endpoint = _endpoint > 77 if endpoint.url != url: > 78 changed = True > 79 cloud.keystone_client.endpoints.update( > 80 endpoint, url=url) > 81 break > 82 else: > 83 changed = True > 84 cloud.keystone_client.endpoints.create( > 85 service=service.id, > 86 url=url, > 87 interface=interface, > 88 region=endpoint_region) > > > At some point, this module /create/ or /update/ a service endpoint. It > first tests if the service is already registered, and update the URL > if so (l. 74 to 81), or create the new endpoint in other cases (l. > 82). Unfortunately, the test (l. 74 to 75) only looks at the service > (e.g., glance) and the interface (e.g., public). This makes the test > wrong because we deploy the same service many times, but into > different regions. We have to add an extra condition to not only test > the service but also its region. > > > 74 if _endpoint.service_id == service.id and \ > 75 _endpoint.interface == interface and \ > 76 _endpoint.region == endpoint_region: > > > Finally, when we deployed OSRs, we override the value of > `keystone_admin_url', `keystone_internal_url' and `keystone_public_url' > to target Keystone in AR. We also have to change the > `[keystone_authtoken]' section of nova/neutron/glance conf[10] so that > it uses these variables instead of the canonical form with the > `kolla_internal_fqdn' variable[11]. In this regards, is it due to some > legacy code that configuration files use the `kolla_internal_fqdn' > variable instead of `keystone_*_url'? > > That's almost all. As you can see, handle multi-regions implies a very > small number of modifications if you call Kolla multiple times. > > So, thanks for the great job Kolla team! And waiting for your > feedback. > > > > [1] https://github.com/BeyondTheClouds/kolla/commit/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554 > [2] https://beyondtheclouds.github.io/ > [3] http://docs.openstack.org/arch-design/multi-site-architecture.html > [4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5 > cf974216ae/ansible/roles/keystone/tasks/register.yml > [5] https://github.com/BeyondTheClouds/kolla/commit/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff- > 40be75c3a2237adfd1a05178f6f60006R1 > [6] https://github.com/BeyondTheClouds/kolla/commit/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff- > 607e6e5925d0031dafa79eb80d198640R215 > [7] https://github.com/BeyondTheClouds/kolla/commit/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff- > 6cbb63bb9c4843bcf8db4710e588475bR188 > [8] https://github.com/BeyondTheClouds/kolla/tree/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions > [9] https://github.com/BeyondTheClouds/kolla/commit/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff- > ba10dd4575f65e03d50d586febdccbadR72 > [10] https://github.com/BeyondTheClouds/kolla/blob/ > 7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf > [11] https://github.com/openstack/kolla/blob/ > 11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/ > templates/nova.conf.j2#L155 > > > -- > Ronan-A. Cherrueau > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Regards, Jeffrey Zhang Blog: http://xcodest.me
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev