[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
newton is EOL ** Changed in: keystone/newton Status: In Progress => Won't Fix ** Changed in: keystone/ocata Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: puppet-keystone Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: keystone (Ubuntu) Status: In Progress => Invalid ** Changed in: cloud-archive Status: In Progress => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Tags removed: sts-sru-needed ** Tags added: sts-sru-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: tripleo Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: tripleo Milestone: pike-rc1 => queens-1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: tripleo Milestone: pike-3 => pike-rc1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This bug was fixed in the package keystone - 2:10.0.1-0ubuntu2~cloud0 --- keystone (2:10.0.1-0ubuntu2~cloud0) xenial-newton; urgency=medium . * New update for the Ubuntu Cloud Archive. . keystone (2:10.0.1-0ubuntu2) yakkety; urgency=high . * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes between batches in order to lower resource consumption and make flushing more robust for replication (LP: #1649616). ** Changed in: cloud-archive/newton Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This bug was fixed in the package keystone - 2:9.3.0-0ubuntu2~cloud0 --- keystone (2:9.3.0-0ubuntu2~cloud0) trusty-mitaka; urgency=medium . * New update for the Ubuntu Cloud Archive. . keystone (2:9.3.0-0ubuntu2) xenial; urgency=high . * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes between batches in order to lower resource consumption and make flushing more robust for replication (LP: #1649616). ** Changed in: cloud-archive/mitaka Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This bug was fixed in the package keystone - 2:11.0.2-0ubuntu1~cloud0 --- keystone (2:11.0.2-0ubuntu1~cloud0) xenial-ocata; urgency=medium . * New upstream release for the Ubuntu Cloud Archive. . keystone (2:11.0.2-0ubuntu1) zesty; urgency=medium . [ Jorge Niedbalski ] * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes between batches in order to lower resource consumption and make flushing more robust for replication (LP: #1649616). . [ James Page ] * New upstream stable release for OpenStack Ocata (LP: #1696139). ** Changed in: cloud-archive/ocata Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This bug was fixed in the package keystone - 2:9.3.0-0ubuntu2 --- keystone (2:9.3.0-0ubuntu2) xenial; urgency=high * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes between batches in order to lower resource consumption and make flushing more robust for replication (LP: #1649616). -- Jorge Niedbalski Wed, 07 Jun 2017 10:33:50 +0100 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This bug was fixed in the package keystone - 2:10.0.1-0ubuntu2 --- keystone (2:10.0.1-0ubuntu2) yakkety; urgency=high * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes between batches in order to lower resource consumption and make flushing more robust for replication (LP: #1649616). -- Jorge Niedbalski Wed, 07 Jun 2017 13:07:50 +0100 ** Changed in: keystone (Ubuntu Yakkety) Status: Fix Committed => Fix Released ** Changed in: keystone (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This bug was fixed in the package keystone - 2:11.0.2-0ubuntu1 --- keystone (2:11.0.2-0ubuntu1) zesty; urgency=medium [ Jorge Niedbalski ] * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes between batches in order to lower resource consumption and make flushing more robust for replication (LP: #1649616). [ James Page ] * New upstream stable release for OpenStack Ocata (LP: #1696139). -- James Page Wed, 07 Jun 2017 16:01:45 +0100 ** Changed in: keystone (Ubuntu Zesty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Same results for ocata (http://pastebin.ubuntu.com/25085019/) ** Tags removed: verification-needed verification-needed-zesty verification-ocata-needed ** Tags added: verification-done verification-done-zesty verification-ocata-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Same results for yakkety/newton for 5) http://pastebin.ubuntu.com/25084654/ Marking the verification as done for Y/N ** Tags removed: verification-needed-yakkety verification-newton-needed ** Tags added: verification-done-yakkety verification-newton-done ** Merge proposal unlinked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324871 ** Merge proposal unlinked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324870 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Tags added: verification-needed-zesty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
This is the verification that I ran through: 1) juju run --application keystone "sudo test -f /etc/cron.d/keystone- token-flush && echo 'Exists'" --format=yaml 2) Commented /etc/cron.d/keystone-token-flush 3) Killed all the mysql threads *) SELECT trx_mysql_thread_id as ID, trx_id, trx_query, trx_started, trx_state FROM information_schema.innodb_trx WHERE trx_state = "RUNNING" AND trx_query LIKE "DELETE FROM token%" ORDER BY trx_started; *) KILL thread_id; *) KILL CONNECTION thread_id; 4) Manually run the token_flush command *) sudo -H -u keystone bash -c "/usr/bin/keystone-manage token_flush > /tmp/token-flush.log" After running this, provide me the the following details as attachments into this case. (Please connect to the database using the before mentioned method). 5) Re-enable the cronjob, and perform the following queries on the database to check if there were any locks or stuck transactions. SHOW open tables WHERE In_use > 0; SELECT * FROM information_schema.innodb_locks; SELECT * FROM information_schema.innodb_trx ORDER BY trx_started; SHOW ENGINE innodb status; -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
For the case of xenial/mitaka the results for 5) http://pastebin.ubuntu.com/25082901/, which indeed shows the problem as solved Marking the verification as done on X/M. ** Tags removed: verification-mitaka-needed verification-needed-xenial ** Tags added: verification-done-xenial verification-mitaka-done ** Merge proposal unlinked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324873 ** Merge proposal unlinked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324872 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Reviewed: https://review.openstack.org/482552 Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=c1bda5f81e35ec597f8d684a35c2c1446c8b1527 Submitter: Jenkins Branch:stable/newton commit c1bda5f81e35ec597f8d684a35c2c1446c8b1527 Author: Juan Antonio Osorio Robles Date: Tue Apr 18 13:13:27 2017 +0300 Change keystone token flush to run hourly In a recent commit [1] the keystone token flush cron job was changed to run twice a day. However, this change was not enough for big deployments. After getting some customer feedback and looking at what other projects are doing [2] [3] [4]. It seems that running this job hourly is the way to go. [1] Ia0b0fb422318712f4b0f4d023cbb3a61d40bb85d [2] https://www.ibm.com/support/knowledgecenter/en/SSB27U_6.4.0/com.ibm.zvm.v640.hcpo4/exptoken.htm [3] https://review.openstack.org/#/c/88670/8 [4] https://github.com/openstack/charm-keystone/blob/master/templates/keystone-token-flush Conflicts: manifests/cron/token_flush.pp spec/acceptance/keystone_federation_identity_provider_spec.rb spec/acceptance/keystone_federation_shibboleth_spec.rb spec/acceptance/keystone_wsgi_apache_spec.rb spec/classes/keystone_cron_token_flush_spec.rb (cherry picked from commit f694b5551f896042df6aeb751c65986ef3342f54) Change-Id: I6ec7ec8111bd93e5638cfe96189e36f0e0691d65 Related-Bug: #1649616 (cherry picked from commit 90ffc7f6008370e9d9893ab69f31683352c854c3) ** Tags added: in-stable-newton -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Fix proposed to branch: stable/newton Review: https://review.openstack.org/482606 ** Changed in: keystone/newton Assignee: (unassigned) => Raildo Mascena de Sousa Filho (raildo) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Fix proposed to branch: stable/ocata Review: https://review.openstack.org/482601 ** Changed in: keystone/ocata Assignee: (unassigned) => Raildo Mascena de Sousa Filho (raildo) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: tripleo Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Is testing for this bug part of the standard test-suite run for the point-release stable update (https://bugs.launchpad.net/ubuntu/+source/swift/+bug/1696139)? If so, please mention it so that this can be released. If not, please test this so that the point-releases can be released :). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Hello Alex, or anyone else affected, Accepted keystone into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keystone/2:10.0.1-0ubuntu2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-yakkety to verification-done-yakkety. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-yakkety. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: keystone (Ubuntu Yakkety) Status: In Progress => Fix Committed ** Tags added: verification-needed-yakkety ** Changed in: keystone (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags added: verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Hello Alex, or anyone else affected, Accepted keystone into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keystone/2:11.0.2-0ubuntu1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: keystone (Ubuntu Zesty) Status: In Progress => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: tripleo Milestone: pike-2 => pike-3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
@slashd this sru is not finished yet so i'll re-instate the -needed tag (that we use for tracking purposes) ** Tags removed: sts-sru-done ** Tags added: sts-sru-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Tags removed: sts-sru-needed ** Tags added: sts-sru-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Uploaded to yakkety and zesty proposed for SRU team review. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
Uploaded to xenial-proposed for SRU team review. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Also affects: keystone/newton Importance: Undecided Status: New ** Also affects: keystone/ocata Importance: Undecided Status: New ** Changed in: keystone/newton Importance: Undecided => Medium ** Changed in: keystone/newton Status: New => In Progress ** Changed in: keystone/ocata Importance: Undecided => Medium ** Changed in: keystone/ocata Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Description changed: - The Keystone token flush job can get into a state where it will never + [Impact] + + * The Keystone token flush job can get into a state where it will never complete because the transaction size exceeds the mysql galara transaction size - wsrep_max_ws_size (1073741824). + [Test Case] - Steps to Reproduce: 1. Authenticate many times 2. Observe that keystone token flush job runs (should be a very long time depending on disk) >20 hours in my environment 3. Observe errors in mysql.log indicating a transaction that is too large - Actual results: Expired tokens are not actually flushed from the database without any errors in keystone.log. Only errors appear in mysql.log. - Expected results: Expired tokens to be removed from the database + [Additional info:] - Additional info: - It is likely that you can demonstrate this with less than 1 million tokens as the >1 million token table is larger than 13GiB and the max transaction size is 1GiB, my token bench-marking Browbeat job creates more than needed. + It is likely that you can demonstrate this with less than 1 million + tokens as the >1 million token table is larger than 13GiB and the max + transaction size is 1GiB, my token bench-marking Browbeat job creates + more than needed. Once the token flush job can not complete the token table will never decrease in size and eventually the cloud will run out of disk space. Furthermore the flush job will consume disk utilization resources. This was demonstrated on slow disks (Single 7.2K SATA disk). On faster disks you will have more capacity to generate tokens, you can then generate the number of tokens to exceed the transaction size even faster. Log evidence: [root@overcloud-controller-0 log]# grep " Total expired" /var/log/keystone/keystone.log 2016-12-08 01:33:40.530 21614 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1082434 2016-12-09 09:31:25.301 14120 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1084241 2016-12-11 01:35:39.082 4223 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1086504 2016-12-12 01:08:16.170 32575 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1087823 2016-12-13 01:22:18.121 28669 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1089202 - [root@overcloud-controller-0 log]# tail mysqld.log + [root@overcloud-controller-0 log]# tail mysqld.log 161208 1:33:41 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161208 1:33:41 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161209 9:31:26 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161209 9:31:26 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161211 1:35:39 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161211 1:35:40 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161212 1:08:16 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161212 1:08:17 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161213 1:22:18 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161213 1:22:19 [ERROR] WSREP: rbr write fail, data_len: 0, 2 + Disk utilization issue graph is attached. The entire job in that graph + takes from the first spike is disk util(~5:18UTC) and culminates in + about ~90 minutes of pegging the disk (between 1:09utc to 2:43utc). - Disk utilization issue graph is attached. The entire job in that graph takes from the first spike is disk util(~5:18UTC) and culminates in about ~90 minutes of pegging the disk (between 1:09utc to 2:43utc). + [Regression Potential] + * Not identified -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Tags added: sts-sru-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Patch added: "fix-lp-1649616-zesty.debdiff" https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886958/+files/fix-lp-1649616-zesty.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Patch added: "fix-lp-1649616-xenial.debdiff" https://bugs.launchpad.net/ubuntu/xenial/+source/keystone/+bug/1649616/+attachment/4886960/+files/fix-lp-1649616-xenial.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Patch added: "fix-lp-1649616-yakkety.debdiff" https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886959/+files/fix-lp-1649616-yakkety.debdiff ** Changed in: keystone (Ubuntu Xenial) Importance: Medium => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Patch added: "fix-lp-1649616-artful.debdiff" https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886957/+files/fix-lp-1649616-artful.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: cloud-archive Status: New => In Progress ** Changed in: cloud-archive Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: cloud-archive/mitaka Status: Triaged => In Progress ** Changed in: cloud-archive/mitaka Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: cloud-archive/newton Status: Triaged => In Progress ** Changed in: cloud-archive/newton Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: cloud-archive/ocata Status: Triaged => In Progress ** Changed in: cloud-archive/ocata Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: keystone (Ubuntu) Importance: Undecided => High ** Changed in: keystone (Ubuntu) Status: New => In Progress ** Changed in: keystone (Ubuntu) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: keystone (Ubuntu Xenial) Status: Triaged => In Progress ** Changed in: keystone (Ubuntu Xenial) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: keystone (Ubuntu Yakkety) Importance: Medium => High ** Changed in: keystone (Ubuntu Yakkety) Status: Triaged => In Progress ** Changed in: keystone (Ubuntu Yakkety) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: keystone (Ubuntu Zesty) Importance: Medium => High ** Changed in: keystone (Ubuntu Zesty) Status: Triaged => In Progress ** Changed in: keystone (Ubuntu Zesty) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Merge proposal linked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324870 ** Merge proposal linked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324871 ** Merge proposal linked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324872 ** Merge proposal linked: https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324873 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Changed in: keystone (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: keystone (Ubuntu Yakkety) Importance: Undecided => Medium ** Changed in: keystone (Ubuntu Zesty) Importance: Undecided => Medium ** Changed in: keystone (Ubuntu Xenial) Status: New => Triaged ** Changed in: keystone (Ubuntu Yakkety) Status: New => Triaged ** Changed in: keystone (Ubuntu Zesty) Status: New => Triaged ** Also affects: cloud-archive/mitaka Importance: Undecided Status: New ** Also affects: cloud-archive/newton Importance: Undecided Status: New ** Also affects: cloud-archive/ocata Importance: Undecided Status: New ** Changed in: cloud-archive/mitaka Importance: Undecided => Medium ** Changed in: cloud-archive/mitaka Status: New => Triaged ** Changed in: cloud-archive/newton Importance: Undecided => Medium ** Changed in: cloud-archive/newton Status: New => Triaged ** Changed in: cloud-archive/ocata Importance: Undecided => Medium ** Changed in: cloud-archive/ocata Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Also affects: keystone (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: keystone (Ubuntu Yakkety) Importance: Undecided Status: New ** Also affects: keystone (Ubuntu Zesty) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment
** Also affects: cloud-archive Importance: Undecided Status: New ** Tags added: sts ** Also affects: keystone (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1649616 Title: Keystone Token Flush job does not complete in HA deployed environment To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs