Public bug reported: This is a very difficult to reproduce bug but occurs nevertheless and can be observed when we switch the backend drivers.
Steps to reproduce: 1. Switch the backened driver in keystone conf file from one driver to another . Restart keystone 2. Immediately (if you wait for few seconds, this cannot be reproduced) , make calls that in turn access the keystone methods /keystone/identity/core >> get_group and list_users_in_group. It doesn't have to be exactly these two. It can be any two similar methods in identity.core.py which uses the @domains_configured decorator. https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L100 Invoke two methods that use this decorators and these method invocations must be almost parallel. Both the methods hit the following flow where the race condition occurs: def domains_configured(f): """Wraps API calls to lazy load domain configs after init. """ @functools.wraps(f) def wrapper(self, *args, **kwargs): if (not self.domain_configs.configured and CONF.identity.domain_specific_drivers_enabled): self.domain_configs.setup_domain_drivers( self.driver, self.resource_api) else: LOG.error('domains will not be configured') return f(self, *args, **kwargs) return wrapper def setup_domain_drivers(self, standard_driver, resource_api): # This is called by the api call wrapper self.configured = True self.driver = standard_driver ..... When the first call is placed, it sets self.configured to True and then proceeds towards loading the driver that corresponds to the domain-..However, the second request call assumes the the driver load is already complete (purely based on the value set to self.configured - which is True even though driver is not really loaded). It thus ends up using the default driver (ie driver which is not domain specific ) and retrieves the values. There should be some synchronization logic added inside domains_configured (or one of the internal methods) so that incorrect backend driver is not used by a request. ** Affects: keystone Importance: Undecided Assignee: Divya K Konoor (dikonoor) Status: New ** Changed in: keystone Assignee: (unassigned) => Divya K Konoor (dikonoor) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1549726 Title: Race condition in keystone domain config Status in OpenStack Identity (keystone): New Bug description: This is a very difficult to reproduce bug but occurs nevertheless and can be observed when we switch the backend drivers. Steps to reproduce: 1. Switch the backened driver in keystone conf file from one driver to another . Restart keystone 2. Immediately (if you wait for few seconds, this cannot be reproduced) , make calls that in turn access the keystone methods /keystone/identity/core >> get_group and list_users_in_group. It doesn't have to be exactly these two. It can be any two similar methods in identity.core.py which uses the @domains_configured decorator. https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L100 Invoke two methods that use this decorators and these method invocations must be almost parallel. Both the methods hit the following flow where the race condition occurs: def domains_configured(f): """Wraps API calls to lazy load domain configs after init. """ @functools.wraps(f) def wrapper(self, *args, **kwargs): if (not self.domain_configs.configured and CONF.identity.domain_specific_drivers_enabled): self.domain_configs.setup_domain_drivers( self.driver, self.resource_api) else: LOG.error('domains will not be configured') return f(self, *args, **kwargs) return wrapper def setup_domain_drivers(self, standard_driver, resource_api): # This is called by the api call wrapper self.configured = True self.driver = standard_driver ..... When the first call is placed, it sets self.configured to True and then proceeds towards loading the driver that corresponds to the domain-..However, the second request call assumes the the driver load is already complete (purely based on the value set to self.configured - which is True even though driver is not really loaded). It thus ends up using the default driver (ie driver which is not domain specific ) and retrieves the values. There should be some synchronization logic added inside domains_configured (or one of the internal methods) so that incorrect backend driver is not used by a request. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1549726/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
