[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

This bug was fixed in the package keystone - 2:9.3.0-0ubuntu3.2~cloud0
---

 keystone (2:9.3.0-0ubuntu3.2~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 keystone (2:9.3.0-0ubuntu3.2) xenial; urgency=medium
 .
   * LDAP backend performance improvements (LP: #1582585)
 - d/p/prevent-error-when-duplicate-mapping-is-created.patch:
   Handle races for creating id mappings.
 - d/p/added-cache-for-id-mapping-manager.patch: Add a cache to
   the id mapping manager to improve performance.
 - d/p/add-mapping_populate-command.patch: Add a keystone-manage
   command to populate id mappings between backend identity
   provider and keystone database.
 - d/p/faster-id-mapping-lookup.patch: Allow querying for all
   public ids in a domain at once instead of N queries (one per
   entity).
 - d/p/fallback-for-custom-id-map-drivers.patch: Add fallback
   path for faster-id-mapping lookup for any customer id mapping
   drivers that may be in use or existing deployments.


** Changed in: cloud-archive/mitaka
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

This bug was fixed in the package keystone - 2:10.0.3-0ubuntu1~cloud1
---

 keystone (2:10.0.3-0ubuntu1~cloud1) xenial-newton; urgency=medium
 .
   * LDAP backend performance improvements (LP: #1582585)
 - d/p/faster-id-mapping-lookup.patch: Allow querying for all
   public ids in a domain at once instead of N queries (one per
   entity).
 - d/p/fallback-for-custom-id-map-drivers.patch: Add fallback
   path for faster-id-mapping lookup for any customer id mapping
   drivers that may be in use or existing deployments.


** Changed in: cloud-archive/newton
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

Regression testing for newton and mitaka were successful:

xenial-newton-proposed:

Ran: 102 tests in 1448.5591 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 919.5493 sec.

xenial-mitaka-proposed:

Ran: 102 tests in 1031.4567 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 621.0117 sec.

trusty-mitaka-proposed:

Ran: 102 tests in 1036.5808 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 603.3084 sec.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Launchpad Bug Tracker]

This bug was fixed in the package keystone - 2:9.3.0-0ubuntu3.2

---
keystone (2:9.3.0-0ubuntu3.2) xenial; urgency=medium

  * LDAP backend performance improvements (LP: #1582585)
- d/p/prevent-error-when-duplicate-mapping-is-created.patch:
  Handle races for creating id mappings.
- d/p/added-cache-for-id-mapping-manager.patch: Add a cache to
  the id mapping manager to improve performance.
- d/p/add-mapping_populate-command.patch: Add a keystone-manage
  command to populate id mappings between backend identity
  provider and keystone database.
- d/p/faster-id-mapping-lookup.patch: Allow querying for all
  public ids in a domain at once instead of N queries (one per
  entity).
- d/p/fallback-for-custom-id-map-drivers.patch: Add fallback
  path for faster-id-mapping lookup for any customer id mapping
  drivers that may be in use or existing deployments.

 -- Billy Olsen   Wed, 10 Jan 2018 14:24:36
-0700

** Changed in: keystone (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

** Tags removed: verification-newton-done verification-xenial-done
** Tags added: verification-done-newton verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

Tests completed below with 12,500 users in ldap tree

=== Mitaka Results ===

$ dpkg -l | grep keystone
ii  keystone 2:9.3.0-0ubuntu3.1 
all  OpenStack identity service - Daemons

$ time openstack user list --domain userdomain
...
real2m3.608s
user0m6.848s
sys 0m0.548s


 Mitaka Proposed 
$ dpkg -l | grep keystone
ii  keystone 2:9.3.0-0ubuntu3.2 
all  OpenStack identity service - Daemons

$ time openstack user list --domain userdomain
...
real0m16.695s
user0m6.708s
sys 0m0.328s


=== Newton Results ===

$ dpkg -l | grep keystone
ii  keystone 2:10.0.3-0ubuntu1~cloud0   
all  OpenStack identity service - Daemons

$ time openstack user list --domain userdomain
...
real0m22.626s
user0m6.232s
sys 0m0.492s

=== Newton-Staging Results ===

$ dpkg -l | grep keystone
ii  keystone 2:10.0.3-0ubuntu1~cloud1   
all  OpenStack identity service - Daemons

$ time openstack user list --domain userdomain
...
real0m16.297s
user0m8.692s
sys 0m0.452s

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

Adding tar.gz of /etc/keystone for reference

** Attachment added: "keystone-config.tar.gz"
   
https://bugs.launchpad.net/keystone/+bug/1582585/+attachment/5051668/+files/keystone-config.tar.gz

** Tags removed: verification-mitaka-needed verification-needed 
verification-needed-xenial verification-newton-needed
** Tags added: verification-done-mitaka verification-newton-done 
verification-xenial-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

@gunph1ld - these changes are already in the ocata code you are running.
While 15-20 seconds is still not fast, it is generally faster than the
previous code which would issue a query per entry record in the
id_mapping table.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Jack Ivanov]

I have the same issue on Ocata (keystone-11.0.3), each request to keystone 
takes 15-20 sec.
There are more then 15 records in the id_mapping table. 
Does the patch actually fix the problem?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

@raja - can you share your keystone config (with sensitive parts
redacted)? How many users are in your ldap directory which is configured
as the backend domain?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[raja]

and for the mitaka-proposed, looks like the same.

the keystone version is:

```
root@ubuntu:~# apt-cache policy keystone
keystone:
  Installed: 2:9.3.0-0ubuntu3.1~cloud0
  Candidate: 2:9.3.0-0ubuntu3.1~cloud0
  Version table:
 *** 2:9.3.0-0ubuntu3.1~cloud0 0
500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ 
trusty-updates/mitaka/main amd64 Packages
100 /var/lib/dpkg/status
 1:2014.1.5-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 
Packages
 1:2014.1.3-0ubuntu2.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 
Packages
 1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
```

but listing users still doesn't take much time, about 10 seconds:

```
root@ubuntu:~# time openstack user list --domain default
real0m9.662s
user0m1.284s
sys 0m0.232s
``` 


so seems that I can't reproduce this problem, do you have any suggestions?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[raja]

the ubuntu release is: DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[raja]

for the newton-proposed:

As our ldap server has a limit exceed setting, so I configured

```
[ldap]
page_size = 2
```

before using the keystone in proposed repo, the keystone version is:
```
root@ubuntu:~# apt-cache policy keystone
keystone:
  Installed: 2:10.0.3-0ubuntu1~cloud0
  Candidate: 2:10.0.3-0ubuntu1~cloud1
  Version table:
 2:10.0.3-0ubuntu1~cloud1 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu 
xenial-proposed/newton/main amd64 Packages
500 http://ubuntu-cloud.archive.canonical.com/ubuntu 
xenial-proposed/newton/main i386 Packages
 *** 2:10.0.3-0ubuntu1~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu 
xenial-updates/newton/main amd64 Packages
500 http://ubuntu-cloud.archive.canonical.com/ubuntu 
xenial-updates/newton/main i386 Packages
100 /var/lib/dpkg/status
 2:9.3.0-0ubuntu3.1 500
500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
500 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
500 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages
500 http://archive.ubuntu.com/ubuntu xenial-security/main i386 Packages
 2:9.0.0-0ubuntu1 500
500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
500 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages

```

list users in ldap will take about 7-11 seconds:

```
root@ubuntu:~# time openstack user list --domain default
...
real0m8.522s
user0m1.476s
sys 0m0.108s

```

after using the package in proposed repo, it takes about 7-11 seconds:

```
root@ubuntu:~# time openstack user list --domain default
...
real0m8.637s
user0m1.484s
sys 0m0.108s

```

I execute above commands for several times, it seems that the patch
doesn't have a strong performance improvement

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Łukasz Zemczak]

Hello jackning, or anyone else affected,

Accepted keystone into xenial-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/keystone/2:9.3.0-0ubuntu3.2 in a
few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-xenial to verification-done-xenial. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-xenial. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: keystone (Ubuntu Xenial)
   Status: Triaged => Fix Committed

** Tags added: verification-needed verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Łukasz Zemczak]

** Description changed:

+ [Impact]
+ 
+  * When using an LDAP backend for Keystone, the performance can be slow if 
there are
+a large number of users using the cloud. This is due in large part to 
querying the
+SQL database for the identity mapping information of each user in a 
separate transaction.
+For example, an environment with 12,000 users will result in 12,000 sql 
queries to the
+backend database in order to fulfill a user list request. This causes some 
admin
+functions in Horizon UI to take several minutes, which often exceeds the 
WSGI and any
+haproxy timeouts configured.
+ 
+  * This is fixed by backporting a series of patches which caches previously 
fetched identity
+mapping information in a memcached instance and changes the logic to query 
all of the
+user id mapping by the domain the id mapping is in. Additionally, the 
keystone-manage
+command to sync the id mapping information with a backend database in an 
offline manner
+is included to allow offline syncing of the data.
+ 
+ [Test Case]
+ 
+  * Install keystone using an ldap backend w/ large number of users.
+  * List user information: openstack user list --domain 
+  * observe slow down
+ 
+ [Regression Potential]
+ 
+  * For Mitaka, the caching backends such as memcached or mongodb will likely 
see more
+usage and an increased footprint due to additional data being cached. 
Caching the
+identity mapping information is now standard since Newton and no major 
issues have
+been seen coming from this.
+ 
+  * This code affects the identity mapping between keystone user and the ldap 
user
+(essentially the bridge between the two). While it does not functionally 
alter the
+information that is mapped (e.g. no difference in how the identity mapping 
is calculated),
+it does alter a key code path for information regarding user identity 
mappings.
+ 
+ [Other Info]
+ 
+  * These patches have been run and tested in a staging environment to 
production and
+have had exposure in the Mitaka path for approximately one month to show 
their stability.
+ 
+ [Original Description]
+ 
  In our project, the speed of query user from ldap server is very
  slow,our ldap user number is 12,000,the query costs almost 45 seconds
  
  The reason is that keystone will generate the uuid for the ldap users one by 
one and insert db.And second query time later,it also goes to db,not use the 
cache.
  So adding the cache to improve the query speed
  
  After adding @MEMOIZE to the following function
  
https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L580.
  First query time almost costs 50 seconds,but second query time later it only 
costs 7 seconds.
  
  So it is very necessary to improve this feature

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

** Changed in: cloud-archive
   Status: New => Invalid

** Changed in: keystone (Ubuntu)
   Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

This is currently building in newton-staging, and will be promoted to
newton-proposed once built. I've also uploaded the new package version
for xenial to the unapproved queue where it is awaiting review by the
SRU team.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

** Tags added: sts

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

The last bullet above is in reference to faster-id-mapping-lookup.patch
and fallback-for-custom-id-map-drivers.patch.

And patch 1-3 are in reference to:
* Patch 1: prevent-error-when-duplicate-mapping-is-created.patch
* Patch 2: added-cache-for-id-mapping-manager.patch
* Patch 3: add-mapping_populate-command.patch

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

Thanks for the patches Billy. I've reviewed the patches and I think they
justify an SRU.

Some notes from my review:
* Patches 1 through 3 are fairly straight forward cherry-picks that are already 
in Newton and require a very minimal changes to apply to Mitaka.
* Patch 3 adds a new 'mapping_populate' subcommand to keystone-manage that 
enables creation of id mapping entries. This is a feature, and while we 
typically don't backport features, I think this case is warranted as it is an 
optional command that allows an admin to pre-create pubic IDs to prevent users 
from hitting costly CLI/API calls.
* Patch requires additional code to handle backends that don't support 
get_domain_mapping_list(). I want to make sure testing of the fallback path is 
performed in addition to the new path.

To avoid regression of existing consumers, the OpenStack team will run
their continuous integration tests against the packages that are in
-proposed.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Ubuntu Foundations Team Bug Bot]

The attachment "newton-lp1582585.debdiff" seems to be a debdiff.  The
ubuntu-sponsors team has been subscribed to the bug report so that they
can review and hopefully sponsor the debdiff.  If the attachment isn't a
patch, please remove the "patch" flag from the attachment, remove the
"patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe
the team.

[This is an automated message performed by a Launchpad user owned by
~brian-murray, for any issue please contact him.]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

[Impact]

 * When using an LDAP backend for Keystone, the performance can be slow if 
there are
   a large number of users using the cloud. This is due in large part to 
querying the
   SQL database for the identity mapping information of each user in a separate 
transaction.
   For example, an environment with 12,000 users will result in 12,000 sql 
queries to the
   backend database in order to fulfill a user list request. This causes some 
admin
   functions in Horizon UI to take several minutes, which often exceeds the 
WSGI and any
   haproxy timeouts configured.

 * This is fixed by backporting a series of patches which caches previously 
fetched identity
   mapping information in a memcached instance and changes the logic to query 
all of the
   user id mapping by the domain the id mapping is in. Additionally, the 
keystone-manage
   command to sync the id mapping information with a backend database in an 
offline manner
   is included to allow offline syncing of the data.

[Test Case]

 * Install keystone using an ldap backend w/ large number of users.
 * List user information: openstack user list --domain 
 * observe slow down

[Regression Potential]

 * For Mitaka, the caching backends such as memcached or mongodb will likely 
see more
   usage and an increased footprint due to additional data being cached. 
Caching the
   identity mapping information is now standard since Newton and no major 
issues have
   been seen coming from this.

 * This code affects the identity mapping between keystone user and the ldap 
user
   (essentially the bridge between the two). While it does not functionally 
alter the
   information that is mapped (e.g. no difference in how the identity mapping 
is calculated),
   it does alter a key code path for information regarding user identity 
mappings.

[Other Info]
 
 * These patches have been run and tested in a staging environment to 
production and
   have had exposure in the Mitaka path for approximately one month to show 
their stability.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

A patch for Mitaka version (xenial and the trusty-mitaka Ubuntu Cloud
Archive)

** Patch added: "mitaka-lp1582585.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/keystone/+bug/1582585/+attachment/5034743/+files/mitaka-lp1582585.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

A patch for xenial-newton Ubuntu Cloud Archive

** Patch added: "newton-lp1582585.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/keystone/+bug/1582585/+attachment/5034732/+files/newton-lp1582585.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

** Also affects: cloud-archive/newton
   Importance: Undecided
   Status: New

** Changed in: cloud-archive/newton
   Status: New => Triaged

** Changed in: cloud-archive/newton
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Corey Bryant]

** Also affects: cloud-archive/mitaka
   Importance: Undecided
   Status: New

** Also affects: keystone (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: cloud-archive/mitaka
   Status: New => Triaged

** Changed in: keystone (Ubuntu Xenial)
   Status: New => Triaged

** Changed in: cloud-archive/mitaka
   Importance: Undecided => High

** Changed in: keystone (Ubuntu Xenial)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1582585] Re: the speed of query user from ldap server is very slow

[Billy Olsen]

** Also affects: keystone (Ubuntu)
   Importance: Undecided
   Status: New

** Also affects: cloud-archive
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1582585

Title:
  the speed of query user from ldap server is very slow

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1582585/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs