Reviewed: https://review.openstack.org/643670 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d6df1dff3e519a26c1a12b6c32f9799484be5966 Submitter: Zuul Branch: master
commit d6df1dff3e519a26c1a12b6c32f9799484be5966 Author: Corey Bryant <[email protected]> Date: Mon Mar 18 13:46:37 2019 -0400 PY3: Ensure LDAP searches use unicode attributes This is a bug fix that corresponds to changes missed in commit eca0829c4c65e6b64f08023ce2d5a55dc329248f. In Python 3, python-ldap no longer allows bytes for some fields (DNs, RDNs, attribute names, queries). Instead, text values are represented as str, the Unicode text type. Compatibility support is provided for Python 2 by setting bytes_mode=False [1]. This support was provided in commit eca0829c4c65e6b64f08023ce2d5a55dc329248f. In this patch we ensure that attribute names specified in searches are no longer encoded. [1] More details about byte/str usage in python-ldap can be found at: http://www.python-ldap.org/en/latest/bytes_mode.html#bytes-mode Change-Id: If3398e2d08ea14fa4b8c498b2a9a7c7edb47b9e5 Closes-Bug: #1820333 ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1820333 Title: [SRU] ldap search should not encode attributes Status in Ubuntu Cloud Archive: Fix Committed Status in OpenStack Identity (keystone): Fix Released Status in keystone package in Ubuntu: Fix Released Bug description: [Impact] Listing user fails with LDAP backend fails ------------------------------------------ $ openstack user list --debug --domain userdomain Request returned failure status: 400 ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98) Traceback (most recent call last): File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action group=group, File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator return wrapped(*args, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func return f(*args, **new_kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list list_resp = self._list(url_query, self.collection_key) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list resp, body = self.client.get(url, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get return self.request(url, 'GET', **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request return self.session.request(url, method, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98) clean_up ListUser: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98) Traceback (most recent call last): File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action group=group, File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator return wrapped(*args, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func return f(*args, **new_kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list list_resp = self._list(url_query, self.collection_key) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list resp, body = self.client.get(url, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get return self.request(url, 'GET', **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request return self.session.request(url, method, **kwargs) File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98) END return value: 1 /var/log/keystone/keystone.log ------------------------------ (keystone.common.wsgi): 2019-03-15 15:26:15,385 ERROR ('attrs_from_List(): expected string in list', b'mail') Traceback (most recent call last): File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__ result = method(req, **params) File "/usr/lib/python3/dist-packages/keystone/common/controller.py", line 103, in wrapper return f(self, request, filters, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/controllers.py", line 71, in list_users domain_scope=domain, hints=hints File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped __ret_val = __f(*args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 416, in wrapper return f(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 426, in wrapper return f(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in list_users ref_list = self._handle_shadow_and_local_users(driver, hints) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1044, in _handle_shadow_and_local_users return driver.list_users(hints) + fed_res File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users return self.user.get_all_filtered(hints) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered for user in self.get_all(query, hints)] File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all hints=hints) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1888, in get_all return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1590, in get_all for x in self._ldap_get_all(hints, ldap_filter)] File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper return f(self, hints, *args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1543, in _ldap_get_all attrs) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 976, in search_s attrlist_utf8, attrsonly) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 654, in wrapper return func(self, conn, *args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 803, in search_s attrsonly) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 858, in search_s return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1264, in search_ext_s return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1202, in _apply_method_s return func(self,*args,**kwargs) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 851, in search_ext_s msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 847, in search_ext timeout,sizelimit, File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 319, in _ldap_call result = func(*args,**kwargs) TypeError: ('attrs_from_List(): expected string in list', b'mail') --------------------------------------------- In search_s() we're still encoding attrlist (note similar behavior in paged_search_s): attrlist_utf8 = list(map(utf8_encode, attrlist)) Looking closer at the attribute list these all appear to be attribute names and that also appears to be how LDAP searches generally work; they specify attribute names they want to return, not values: [b'enabled', b'sn', b'userPassword', b'cn', b'description', b'mail'] In Python 3 (and Python2 with bytes_mode=False) python-ldap no longer allows bytes for some fields (DNs, RDNs, attribute names, queries). Instead, text values are represented as str, the Unicode text type. A prior patch to Keystone's LDAP backend (see commit eca0829c4c65e6b64f08023ce2d5a55dc329248f) enabled this support but missed the above lines of code. Changing the above line of code to not utf8 encode the attrlist fixes the problem for me. [Test Case] Run charm-keystone-ldap functional tests for OpenStack Rocky or above. Upstream unit tests are also run. [Regression Potential] The only regression potential would be for PY2 code paths. PY3 code paths never worked for keystone's LDAP backend. The approach to the patch have purposefully minimized amount of code required and therefore regression potential for PY2. Note that Rocky for Ubuntu supports PY2 but as of Stein Ubuntu has dropped PY2 support. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1820333/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
