[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Morgan Fainberg]

newton is EOL

** Changed in: keystone/newton
   Status: In Progress => Won't Fix

** Changed in: keystone/ocata
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Tobias Urdin]

** Changed in: puppet-keystone
   Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Corey Bryant]

** Changed in: keystone (Ubuntu)
   Status: In Progress => Invalid

** Changed in: cloud-archive
   Status: In Progress => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Edward Hope-Morley]

** Tags removed: sts-sru-needed
** Tags added: sts-sru-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Juan Antonio Osorio Robles]

** Changed in: tripleo
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Emilien Macchi]

** Changed in: tripleo
Milestone: pike-rc1 => queens-1

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Emilien Macchi]

** Changed in: tripleo
Milestone: pike-3 => pike-rc1

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[James Page]

This bug was fixed in the package keystone - 2:10.0.1-0ubuntu2~cloud0
---

 keystone (2:10.0.1-0ubuntu2~cloud0) xenial-newton; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 keystone (2:10.0.1-0ubuntu2) yakkety; urgency=high
 .
   * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
 between batches in order to lower resource consumption and
 make flushing more robust for replication (LP: #1649616).


** Changed in: cloud-archive/newton
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[James Page]

This bug was fixed in the package keystone - 2:9.3.0-0ubuntu2~cloud0
---

 keystone (2:9.3.0-0ubuntu2~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 keystone (2:9.3.0-0ubuntu2) xenial; urgency=high
 .
   * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit
 token flushes between batches in order to lower resource
 consumption and make flushing more robust for replication
 (LP: #1649616).


** Changed in: cloud-archive/mitaka
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[James Page]

This bug was fixed in the package keystone - 2:11.0.2-0ubuntu1~cloud0
---

 keystone (2:11.0.2-0ubuntu1~cloud0) xenial-ocata; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 keystone (2:11.0.2-0ubuntu1) zesty; urgency=medium
 .
   [ Jorge Niedbalski ]
   * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
 between batches in order to lower resource consumption and
 make flushing more robust for replication (LP: #1649616).
 .
   [ James Page ]
   * New upstream stable release for OpenStack Ocata (LP: #1696139).


** Changed in: cloud-archive/ocata
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Launchpad Bug Tracker]

This bug was fixed in the package keystone - 2:9.3.0-0ubuntu2

---
keystone (2:9.3.0-0ubuntu2) xenial; urgency=high

  * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit
token flushes between batches in order to lower resource
consumption and make flushing more robust for replication
(LP: #1649616).

 -- Jorge Niedbalski   Wed, 07 Jun 2017
10:33:50 +0100

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Launchpad Bug Tracker]

This bug was fixed in the package keystone - 2:10.0.1-0ubuntu2

---
keystone (2:10.0.1-0ubuntu2) yakkety; urgency=high

  * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
between batches in order to lower resource consumption and
make flushing more robust for replication (LP: #1649616).

 -- Jorge Niedbalski   Wed, 07 Jun 2017
13:07:50 +0100

** Changed in: keystone (Ubuntu Yakkety)
   Status: Fix Committed => Fix Released

** Changed in: keystone (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Launchpad Bug Tracker]

This bug was fixed in the package keystone - 2:11.0.2-0ubuntu1

---
keystone (2:11.0.2-0ubuntu1) zesty; urgency=medium

  [ Jorge Niedbalski ]
  * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
between batches in order to lower resource consumption and
make flushing more robust for replication (LP: #1649616).

  [ James Page ]
  * New upstream stable release for OpenStack Ocata (LP: #1696139).

 -- James Page   Wed, 07 Jun 2017 16:01:45 +0100

** Changed in: keystone (Ubuntu Zesty)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

Same results for ocata (http://pastebin.ubuntu.com/25085019/)


** Tags removed: verification-needed verification-needed-zesty 
verification-ocata-needed
** Tags added: verification-done verification-done-zesty verification-ocata-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

Same results for yakkety/newton for 5)
http://pastebin.ubuntu.com/25084654/

Marking the verification as done for Y/N

** Tags removed: verification-needed-yakkety verification-newton-needed
** Tags added: verification-done-yakkety verification-newton-done

** Merge proposal unlinked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324871

** Merge proposal unlinked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324870

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Brian Murray]

** Tags added: verification-needed-zesty

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

This is the verification that I ran through:

1) juju run --application keystone "sudo test -f /etc/cron.d/keystone-
token-flush && echo 'Exists'" --format=yaml

2) Commented /etc/cron.d/keystone-token-flush

3) Killed all the mysql threads

  *) SELECT trx_mysql_thread_id as ID, trx_id, trx_query, trx_started,
trx_state FROM information_schema.innodb_trx WHERE trx_state = "RUNNING"
AND trx_query LIKE "DELETE FROM token%" ORDER BY trx_started;

  *) KILL thread_id; 
  *) KILL CONNECTION thread_id; 

4) Manually run the token_flush command

  *) sudo -H -u keystone bash -c "/usr/bin/keystone-manage token_flush >
/tmp/token-flush.log"

After running this, provide me the the following details as attachments
into this case. (Please connect to the database using the before
mentioned method).

5) Re-enable the cronjob, and perform the following queries on the database to 
check 
if there were any locks or stuck transactions.

SHOW open tables WHERE In_use > 0; 
SELECT * FROM information_schema.innodb_locks; 
SELECT * FROM information_schema.innodb_trx ORDER BY trx_started; 
SHOW ENGINE innodb status;

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

For the case of xenial/mitaka the results for 5) 
http://pastebin.ubuntu.com/25082901/, 
which indeed shows the problem as solved 

Marking the verification as done on X/M.

** Tags removed: verification-mitaka-needed verification-needed-xenial
** Tags added: verification-done-xenial verification-mitaka-done

** Merge proposal unlinked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324873

** Merge proposal unlinked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324872

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[OpenStack Infra]

Reviewed:  https://review.openstack.org/482552
Committed: 
https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=c1bda5f81e35ec597f8d684a35c2c1446c8b1527
Submitter: Jenkins
Branch:stable/newton

commit c1bda5f81e35ec597f8d684a35c2c1446c8b1527
Author: Juan Antonio Osorio Robles 
Date:   Tue Apr 18 13:13:27 2017 +0300

Change keystone token flush to run hourly

In a recent commit [1] the keystone token flush cron job was changed to
run twice a day. However, this change was not enough for big
deployments.

After getting some customer feedback and looking at what other projects
are doing [2] [3] [4]. It seems that running this job hourly is the way
to go.

[1] Ia0b0fb422318712f4b0f4d023cbb3a61d40bb85d
[2] 
https://www.ibm.com/support/knowledgecenter/en/SSB27U_6.4.0/com.ibm.zvm.v640.hcpo4/exptoken.htm
[3] https://review.openstack.org/#/c/88670/8
[4] 
https://github.com/openstack/charm-keystone/blob/master/templates/keystone-token-flush

Conflicts:
manifests/cron/token_flush.pp
spec/acceptance/keystone_federation_identity_provider_spec.rb
spec/acceptance/keystone_federation_shibboleth_spec.rb
spec/acceptance/keystone_wsgi_apache_spec.rb
spec/classes/keystone_cron_token_flush_spec.rb

(cherry picked from commit f694b5551f896042df6aeb751c65986ef3342f54)
Change-Id: I6ec7ec8111bd93e5638cfe96189e36f0e0691d65
Related-Bug: #1649616
(cherry picked from commit 90ffc7f6008370e9d9893ab69f31683352c854c3)


** Tags added: in-stable-newton

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[OpenStack Infra]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/482606

** Changed in: keystone/newton
 Assignee: (unassigned) => Raildo Mascena de Sousa Filho (raildo)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[OpenStack Infra]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/482601

** Changed in: keystone/ocata
 Assignee: (unassigned) => Raildo Mascena de Sousa Filho (raildo)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Emilien Macchi]

** Changed in: tripleo
   Status: Triaged => In Progress

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Chris Halse Rogers]

Is testing for this bug part of the standard test-suite run for the
point-release stable update
(https://bugs.launchpad.net/ubuntu/+source/swift/+bug/1696139)?

If so, please mention it so that this can be released.

If not, please test this so that the point-releases can be released :).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Andy Whitcroft]

Hello Alex, or anyone else affected,

Accepted keystone into yakkety-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/keystone/2:10.0.1-0ubuntu2 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-yakkety to verification-done-yakkety. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-yakkety. In either case, details of your
testing will help us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: keystone (Ubuntu Yakkety)
   Status: In Progress => Fix Committed

** Tags added: verification-needed-yakkety

** Changed in: keystone (Ubuntu Xenial)
   Status: In Progress => Fix Committed

** Tags added: verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Andy Whitcroft]

Hello Alex, or anyone else affected,

Accepted keystone into zesty-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/keystone/2:11.0.2-0ubuntu1 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: keystone (Ubuntu Zesty)
   Status: In Progress => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Emilien Macchi]

** Changed in: tripleo
Milestone: pike-2 => pike-3

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Edward Hope-Morley]

@slashd this sru is not finished yet so i'll re-instate the -needed tag
(that we use for tracking purposes)

** Tags removed: sts-sru-done
** Tags added: sts-sru-needed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Eric Desrochers]

** Tags removed: sts-sru-needed
** Tags added: sts-sru-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[James Page]

Uploaded to yakkety and zesty proposed for SRU team review.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[James Page]

Uploaded to xenial-proposed for SRU team review.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Lance Bragstad]

** Also affects: keystone/newton
   Importance: Undecided
   Status: New

** Also affects: keystone/ocata
   Importance: Undecided
   Status: New

** Changed in: keystone/newton
   Importance: Undecided => Medium

** Changed in: keystone/newton
   Status: New => In Progress

** Changed in: keystone/ocata
   Importance: Undecided => Medium

** Changed in: keystone/ocata
   Status: New => In Progress

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Description changed:

- The Keystone token flush job can get into a state where it will never
+ [Impact]
+ 
+  * The Keystone token flush job can get into a state where it will never
  complete because the transaction size exceeds the mysql galara
  transaction size - wsrep_max_ws_size (1073741824).
  
+ [Test Case]
  
- Steps to Reproduce:
  1. Authenticate many times
  2. Observe that keystone token flush job runs (should be a very long time 
depending on disk) >20 hours in my environment
  3. Observe errors in mysql.log indicating a transaction that is too large
  
- 
  Actual results:
  Expired tokens are not actually flushed from the database without any errors 
in keystone.log.  Only errors appear in mysql.log.
- 
  
  Expected results:
  Expired tokens to be removed from the database
  
+ [Additional info:]
  
- Additional info:
- It is likely that you can demonstrate this with less than 1 million tokens as 
the >1 million token table is larger than 13GiB and the max transaction size is 
1GiB, my token bench-marking Browbeat job creates more than needed.  
+ It is likely that you can demonstrate this with less than 1 million
+ tokens as the >1 million token table is larger than 13GiB and the max
+ transaction size is 1GiB, my token bench-marking Browbeat job creates
+ more than needed.
  
  Once the token flush job can not complete the token table will never
  decrease in size and eventually the cloud will run out of disk space.
  
  Furthermore the flush job will consume disk utilization resources.  This
  was demonstrated on slow disks (Single 7.2K SATA disk).  On faster disks
  you will have more capacity to generate tokens, you can then generate
  the number of tokens to exceed the transaction size even faster.
  
  Log evidence:
  [root@overcloud-controller-0 log]# grep " Total expired" 
/var/log/keystone/keystone.log
  2016-12-08 01:33:40.530 21614 INFO keystone.token.persistence.backends.sql 
[-] Total expired tokens removed: 1082434
  2016-12-09 09:31:25.301 14120 INFO keystone.token.persistence.backends.sql 
[-] Total expired tokens removed: 1084241
  2016-12-11 01:35:39.082 4223 INFO keystone.token.persistence.backends.sql [-] 
Total expired tokens removed: 1086504
  2016-12-12 01:08:16.170 32575 INFO keystone.token.persistence.backends.sql 
[-] Total expired tokens removed: 1087823
  2016-12-13 01:22:18.121 28669 INFO keystone.token.persistence.backends.sql 
[-] Total expired tokens removed: 1089202
- [root@overcloud-controller-0 log]# tail mysqld.log 
+ [root@overcloud-controller-0 log]# tail mysqld.log
  161208  1:33:41 [Warning] WSREP: transaction size limit (1073741824) 
exceeded: 1073774592
  161208  1:33:41 [ERROR] WSREP: rbr write fail, data_len: 0, 2
  161209  9:31:26 [Warning] WSREP: transaction size limit (1073741824) 
exceeded: 1073774592
  161209  9:31:26 [ERROR] WSREP: rbr write fail, data_len: 0, 2
  161211  1:35:39 [Warning] WSREP: transaction size limit (1073741824) 
exceeded: 1073774592
  161211  1:35:40 [ERROR] WSREP: rbr write fail, data_len: 0, 2
  161212  1:08:16 [Warning] WSREP: transaction size limit (1073741824) 
exceeded: 1073774592
  161212  1:08:17 [ERROR] WSREP: rbr write fail, data_len: 0, 2
  161213  1:22:18 [Warning] WSREP: transaction size limit (1073741824) 
exceeded: 1073774592
  161213  1:22:19 [ERROR] WSREP: rbr write fail, data_len: 0, 2
  
+ Disk utilization issue graph is attached.  The entire job in that graph
+ takes from the first spike is disk util(~5:18UTC) and culminates in
+ about ~90 minutes of pegging the disk (between 1:09utc to 2:43utc).
  
- Disk utilization issue graph is attached.  The entire job in that graph takes 
from the first spike is disk util(~5:18UTC) and culminates in about ~90 minutes 
of pegging the disk (between 1:09utc to 2:43utc).
+ [Regression Potential] 
+ * Not identified

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Eric Desrochers]

** Tags added: sts-sru-needed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Patch added: "fix-lp-1649616-zesty.debdiff"
   
https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886958/+files/fix-lp-1649616-zesty.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Patch added: "fix-lp-1649616-xenial.debdiff"
   
https://bugs.launchpad.net/ubuntu/xenial/+source/keystone/+bug/1649616/+attachment/4886960/+files/fix-lp-1649616-xenial.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Patch added: "fix-lp-1649616-yakkety.debdiff"
   
https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886959/+files/fix-lp-1649616-yakkety.debdiff

** Changed in: keystone (Ubuntu Xenial)
   Importance: Medium => High

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Patch added: "fix-lp-1649616-artful.debdiff"
   
https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886957/+files/fix-lp-1649616-artful.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Changed in: cloud-archive
   Status: New => In Progress

** Changed in: cloud-archive
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: cloud-archive/mitaka
   Status: Triaged => In Progress

** Changed in: cloud-archive/mitaka
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: cloud-archive/newton
   Status: Triaged => In Progress

** Changed in: cloud-archive/newton
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: cloud-archive/ocata
   Status: Triaged => In Progress

** Changed in: cloud-archive/ocata
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: keystone (Ubuntu)
   Importance: Undecided => High

** Changed in: keystone (Ubuntu)
   Status: New => In Progress

** Changed in: keystone (Ubuntu)
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: keystone (Ubuntu Xenial)
   Status: Triaged => In Progress

** Changed in: keystone (Ubuntu Xenial)
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: keystone (Ubuntu Yakkety)
   Importance: Medium => High

** Changed in: keystone (Ubuntu Yakkety)
   Status: Triaged => In Progress

** Changed in: keystone (Ubuntu Yakkety)
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

** Changed in: keystone (Ubuntu Zesty)
   Importance: Medium => High

** Changed in: keystone (Ubuntu Zesty)
   Status: Triaged => In Progress

** Changed in: keystone (Ubuntu Zesty)
 Assignee: (unassigned) => Jorge Niedbalski (niedbalski)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Launchpad Bug Tracker]

** Merge proposal linked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324870

** Merge proposal linked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324871

** Merge proposal linked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324872

** Merge proposal linked:
   
https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324873

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Corey Bryant]

** Changed in: keystone (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: keystone (Ubuntu Yakkety)
   Importance: Undecided => Medium

** Changed in: keystone (Ubuntu Zesty)
   Importance: Undecided => Medium

** Changed in: keystone (Ubuntu Xenial)
   Status: New => Triaged

** Changed in: keystone (Ubuntu Yakkety)
   Status: New => Triaged

** Changed in: keystone (Ubuntu Zesty)
   Status: New => Triaged

** Also affects: cloud-archive/mitaka
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/newton
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/ocata
   Importance: Undecided
   Status: New

** Changed in: cloud-archive/mitaka
   Importance: Undecided => Medium

** Changed in: cloud-archive/mitaka
   Status: New => Triaged

** Changed in: cloud-archive/newton
   Importance: Undecided => Medium

** Changed in: cloud-archive/newton
   Status: New => Triaged

** Changed in: cloud-archive/ocata
   Importance: Undecided => Medium

** Changed in: cloud-archive/ocata
   Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Eric Desrochers]

** Also affects: keystone (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Also affects: keystone (Ubuntu Yakkety)
   Importance: Undecided
   Status: New

** Also affects: keystone (Ubuntu Zesty)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1649616] Re: Keystone Token Flush job does not complete in HA deployed environment

[Jorge Niedbalski]

** Also affects: cloud-archive
   Importance: Undecided
   Status: New

** Tags added: sts

** Also affects: keystone (Ubuntu)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1649616

Title:
  Keystone Token Flush job does not complete in HA deployed environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1649616/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs