[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
Since it's not already mentioned in this bug, the long term solution here is to simply not persist tokens at all: https://blueprints.launchpad.net/keystone/+spec/ephemeral-pki-tokens ** Also affects: openstack-manuals Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
Addressed by the following patch: https://review.openstack.org/#/c/79105/ ** Changed in: openstack-manuals Status: New = Fix Released ** Changed in: openstack-manuals Milestone: None = icehouse -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
A CLI command is an interesting stopgap, but on a heavily utilized OpenStack installation with automated tools operating against OpenStack, this has a high manual maintenance cost. Surely there is some better default that lies in the middle ground between keeping tokens for ever and ever and requiring a manual removal of tokens? As a reference point, I wasn't even aware this was an issue, until one of our test deployments of grizzly using a limited IO system started acting horribly (30 second response times). After tracing the problem from nova to keystone to mysql, I found a 442,000 row token table with 440,000 expired tokens. I went and checked our havana test on a somewhat beefier system and found 1M rows. This issue is a timebomb for any production OS install. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
On 04/05/2014 01:43 PM, Andrew Mann wrote: A CLI command is an interesting stopgap, but on a heavily utilized OpenStack installation with automated tools operating against OpenStack, this has a high manual maintenance cost. Surely there is some better default that lies in the middle ground between keeping tokens for ever and ever and requiring a manual removal of tokens? As a reference point, I wasn't even aware this was an issue, until one of our test deployments of grizzly using a limited IO system started acting horribly (30 second response times). After tracing the problem from nova to keystone to mysql, I found a 442,000 row token table with 440,000 expired tokens. I went and checked our havana test on a somewhat beefier system and found 1M rows. This issue is a timebomb for any production OS install. CRON the CLI job. There is no reason to try and integrate a scheduler into Keystone. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
That seems reasonable. I would suggest adding that as a step to the OpenStack setup documentation. No one has infinite database space, eventually all used OpenStack installations will suffer unless this is done. Regardless of what component (or person) has to complete the task, they'll need to be aware of it. I would prefer to see some kind of automatically expire old tokens configuration option so that the maintenance of keystone stays in keystone - whether that be implemented as a expire-on-access query or cron-derived mechanism, but re-implementing cron is not great and a purge tagged on to other operations could cause a slowdown on those operations. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone (Ubuntu) Status: Confirmed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone Milestone: havana-1 = 2013.2 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone Status: Fix Committed = Fix Released ** Changed in: keystone Milestone: None = havana-1 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
Reviewed: https://review.openstack.org/28133 Committed: http://github.com/openstack/keystone/commit/ff76a1b5cd3308cfb0ce936800364e27413ed946 Submitter: Jenkins Branch:master commit ff76a1b5cd3308cfb0ce936800364e27413ed946 Author: Jamie Lennox jlen...@redhat.com Date: Fri May 3 14:04:09 2013 +1000 Implement Token Flush via keystone-manage. Creates a cli entry 'token_flush' which removes all expired tokens. Fixes: bug 1032633 Implements: blueprint keystone-manage-token-flush Change-Id: I47eab99b577ff9e9ee74fee08e18fd07c4af5aad ** Changed in: keystone Status: In Progress = Fix Committed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
Fix proposed to branch: master Review: https://review.openstack.org/28133 ** Changed in: keystone Status: Invalid = In Progress ** Changed in: keystone Assignee: (unassigned) = Jamie Lennox (jamielennox) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone (Ubuntu) Assignee: (unassigned) = David Höppner (0xffea) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
Moving this issue to a BP: https://blueprints.launchpad.net/keystone/+spec/keystone-manage-token- flush ** Changed in: keystone Status: Confirmed = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone (Ubuntu) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Changed in: keystone Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
** Also affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally when using SQL backend.
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: keystone (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032633] Re: Keystone's token table grows unconditionally.
** Description changed: Keystone's `token` table grows unconditionally with expired tokens. Keystone should provide a backend-agnostic method to find and delete these tokens. This could be run via a periodic task or supplied as a script to run as a cron job. An example SQL statement (if you're using a SQL backend) to workaround this problem: sql DELETE FROM token WHERE expired = NOW(); It may be ideal to allow a date smear to allow older tokens to persist if needed. + + System Information: + + $ dpkg-query --show keystone + keystone2012.1+stable~20120608-aff45d6-0ubuntu1 + + $ cat /etc/lsb-release + DISTRIB_ID=Ubuntu + DISTRIB_RELEASE=12.04 + DISTRIB_CODENAME=precise + DISTRIB_DESCRIPTION=Ubuntu 12.04 LTS ** Summary changed: - Keystone's token table grows unconditionally. + Keystone's token table grows unconditionally when using SQL backend. ** Description changed: - Keystone's `token` table grows unconditionally with expired tokens. + Keystone's `token` table grows unconditionally with expired tokens when + using the SQL backend. Keystone should provide a backend-agnostic method to find and delete these tokens. This could be run via a periodic task or supplied as a script to run as a cron job. An example SQL statement (if you're using a SQL backend) to workaround this problem: - sql DELETE FROM token WHERE expired = NOW(); + sql DELETE FROM token WHERE expired = NOW(); It may be ideal to allow a date smear to allow older tokens to persist if needed. + + Choosing the `memcache` backend may workaround this issue, but SQL is + the package default. System Information: $ dpkg-query --show keystone keystone2012.1+stable~20120608-aff45d6-0ubuntu1 - $ cat /etc/lsb-release + $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04 LTS -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to keystone in Ubuntu. https://bugs.launchpad.net/bugs/1032633 Title: Keystone's token table grows unconditionally when using SQL backend. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keystone/+bug/1032633/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
