[Yahoo-eng-team] [Bug 1592043] Re: os-brick 1.4.0 increases volume setup failure rates
The oslo.privsep part of this bug was fixed in https://review.openstack.org/#/c/329766/ I'm not sure why that didn't show up as it does appear to have a bug reference. ** Changed in: oslo.privsep Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1592043 Title: os-brick 1.4.0 increases volume setup failure rates Status in OpenStack Compute (nova): Fix Released Status in os-brick: Invalid Status in oslo.privsep: Fix Released Bug description: Since merging upper constraints 1.4.0 into upper-constraints, the multinode grenade jobs are hitting a nearly 1/3 failure rate on boot from volume scenarios around volume setup. This would be on Newton code using Mitaka configs. Representative failures are of the following form: http://logs.openstack.org/71/327971/5/gate/gate-grenade-dsvm-neutron- multinode/f2690e3/logs/new/screen-n-cpu.txt.gz?level=WARNING#_2016-06-13_15_22_59_095 The 1/3 failure rate is suspicious, and in the past has often hinted towards a race condition interacting between parallel API requests. The failure rate increase can be seen here - http://tinyurl.com/zrq35e8 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1592043/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1592043] Re: os-brick 1.4.0 increases volume setup failure rates
Cinder related change: https://review.openstack.org/#/c/331973/ ** Changed in: os-brick Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1592043 Title: os-brick 1.4.0 increases volume setup failure rates Status in OpenStack Compute (nova): Fix Released Status in os-brick: Invalid Status in oslo.privsep: New Bug description: Since merging upper constraints 1.4.0 into upper-constraints, the multinode grenade jobs are hitting a nearly 1/3 failure rate on boot from volume scenarios around volume setup. This would be on Newton code using Mitaka configs. Representative failures are of the following form: http://logs.openstack.org/71/327971/5/gate/gate-grenade-dsvm-neutron- multinode/f2690e3/logs/new/screen-n-cpu.txt.gz?level=WARNING#_2016-06-13_15_22_59_095 The 1/3 failure rate is suspicious, and in the past has often hinted towards a race condition interacting between parallel API requests. The failure rate increase can be seen here - http://tinyurl.com/zrq35e8 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1592043/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1592043] Re: os-brick 1.4.0 increases volume setup failure rates
Reviewed: https://review.openstack.org/329769 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4a8f2b0d44ee10dfac2d3d828cd9dc574d5ddbb2 Submitter: Jenkins Branch:master commit 4a8f2b0d44ee10dfac2d3d828cd9dc574d5ddbb2 Author: Angus Lees Date: Wed Jun 15 15:46:38 2016 +1000 Initialise oslo.privsep early in main Any process using oslo.privsep should now initialise the library before first use with things like the rootwrap command to use. This should be done near the top of main() in any command that expects to make privileged calls via oslo.privsep (eg: nova-compute, and not nova-api). See I3ea73e16b07a870629e7d69e897f2524d7068ae8 for the corresponding change in oslo.privsep. Change-Id: I3a52f762deb176fe9201b2a0f0da363057f8aaec Depends-On: I52259e2023e277e8fd62be5df4fd7f799e9b36d7 Closes-Bug: #1592043 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1592043 Title: os-brick 1.4.0 increases volume setup failure rates Status in OpenStack Compute (nova): Fix Released Status in os-brick: In Progress Status in oslo.privsep: New Bug description: Since merging upper constraints 1.4.0 into upper-constraints, the multinode grenade jobs are hitting a nearly 1/3 failure rate on boot from volume scenarios around volume setup. This would be on Newton code using Mitaka configs. Representative failures are of the following form: http://logs.openstack.org/71/327971/5/gate/gate-grenade-dsvm-neutron- multinode/f2690e3/logs/new/screen-n-cpu.txt.gz?level=WARNING#_2016-06-13_15_22_59_095 The 1/3 failure rate is suspicious, and in the past has often hinted towards a race condition interacting between parallel API requests. The failure rate increase can be seen here - http://tinyurl.com/zrq35e8 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1592043/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1592043] Re: os-brick 1.4.0 increases volume setup failure rates
** Also affects: oslo.privsep Importance: Undecided Status: New ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1592043 Title: os-brick 1.4.0 increases volume setup failure rates Status in OpenStack Compute (nova): New Status in os-brick: In Progress Status in oslo.privsep: New Bug description: Since merging upper constraints 1.4.0 into upper-constraints, the multinode grenade jobs are hitting a nearly 1/3 failure rate on boot from volume scenarios around volume setup. This would be on Newton code using Mitaka configs. Representative failures are of the following form: http://logs.openstack.org/71/327971/5/gate/gate-grenade-dsvm-neutron- multinode/f2690e3/logs/new/screen-n-cpu.txt.gz?level=WARNING#_2016-06-13_15_22_59_095 The 1/3 failure rate is suspicious, and in the past has often hinted towards a race condition interacting between parallel API requests. The failure rate increase can be seen here - http://tinyurl.com/zrq35e8 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1592043/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
