[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend

2017-06-16 Thread Sean Dague
** Changed in: nova/mitaka
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1580625

Title:
  volume-backed resize intermittently fails with rbd imagebackend

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  Fix Released
Status in OpenStack Compute (nova) mitaka series:
  Fix Released

Bug description:
  After this Tempest change landed to test volume-backed resize:

  https://review.openstack.org/#/c/314816/

  The ceph plugin job has been intermittently failing the test:

  http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full-
  devstack-plugin-
  ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874

  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
[req-1eef0755-362f-4136-ab7c-695b0bb4f0bb 
tempest-TestServerAdvancedOps-1194878159 
tempest-TestServerAdvancedOps-1912438683] Exception during handling message
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 
133, in _process_incoming
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = 
self.dispatcher.dispatch(message)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
153, in dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
self._do_dispatch(endpoint, method, ctxt, args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
122, in _do_dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = 
func(ctxt, **new_args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
f(self, context, *args, **kw)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
kwargs['instance'], e, sys.exc_info())
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
do_confirm_resize(context, instance, migration.id)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 
271, in inner
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
f(*args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", 

[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend

2017-06-16 Thread Sean Dague
** Changed in: nova/liberty
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1580625

Title:
  volume-backed resize intermittently fails with rbd imagebackend

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  Fix Released
Status in OpenStack Compute (nova) mitaka series:
  Fix Committed

Bug description:
  After this Tempest change landed to test volume-backed resize:

  https://review.openstack.org/#/c/314816/

  The ceph plugin job has been intermittently failing the test:

  http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full-
  devstack-plugin-
  ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874

  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
[req-1eef0755-362f-4136-ab7c-695b0bb4f0bb 
tempest-TestServerAdvancedOps-1194878159 
tempest-TestServerAdvancedOps-1912438683] Exception during handling message
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 
133, in _process_incoming
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = 
self.dispatcher.dispatch(message)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
153, in dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
self._do_dispatch(endpoint, method, ctxt, args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
122, in _do_dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = 
func(ctxt, **new_args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
f(self, context, *args, **kw)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
kwargs['instance'], e, sys.exc_info())
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
do_confirm_resize(context, instance, migration.id)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 
271, in inner
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
f(*args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 

[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend

2016-05-12 Thread OpenStack Infra
Reviewed:  https://review.openstack.org/315187
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=671bb2651d9d1a07947690468825581d30482dcb
Submitter: Jenkins
Branch:master

commit 671bb2651d9d1a07947690468825581d30482dcb
Author: Nicolas Simonds 
Date:   Wed May 11 10:52:52 2016 -0700

imagebackend: Check that the RBD image exists before trying to cleanup

In volume-backed setups, there is no image to clean up, so any
attempts to cleanup the resize snapshots will fail by definition.
Make sure the image exists first.

Change-Id: I25f65bcc76b83f31a8fce77c2b751d2d167ffc7e
Closes-Bug: 1580625


** Changed in: nova
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1580625

Title:
  volume-backed resize intermittently fails with rbd imagebackend

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  In Progress
Status in OpenStack Compute (nova) mitaka series:
  In Progress

Bug description:
  After this Tempest change landed to test volume-backed resize:

  https://review.openstack.org/#/c/314816/

  The ceph plugin job has been intermittently failing the test:

  http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full-
  devstack-plugin-
  ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874

  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
[req-1eef0755-362f-4136-ab7c-695b0bb4f0bb 
tempest-TestServerAdvancedOps-1194878159 
tempest-TestServerAdvancedOps-1912438683] Exception during handling message
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 
133, in _process_incoming
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = 
self.dispatcher.dispatch(message)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
153, in dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
self._do_dispatch(endpoint, method, ctxt, args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
122, in _do_dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = 
func(ctxt, **new_args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
f(self, context, *args, **kw)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
kwargs['instance'], e, sys.exc_info())
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, 

[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend

2016-05-11 Thread Matt Riedemann
** Also affects: nova/liberty
   Importance: Undecided
   Status: New

** Also affects: nova/mitaka
   Importance: Undecided
   Status: New

** Changed in: nova/liberty
   Status: New => In Progress

** Changed in: nova/mitaka
   Status: New => Confirmed

** Changed in: nova/liberty
   Status: In Progress => Confirmed

** Changed in: nova/mitaka
   Importance: Undecided => High

** Changed in: nova/liberty
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1580625

Title:
  volume-backed resize intermittently fails with rbd imagebackend

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) liberty series:
  Confirmed
Status in OpenStack Compute (nova) mitaka series:
  Confirmed

Bug description:
  After this Tempest change landed to test volume-backed resize:

  https://review.openstack.org/#/c/314816/

  The ceph plugin job has been intermittently failing the test:

  http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full-
  devstack-plugin-
  ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874

  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
[req-1eef0755-362f-4136-ab7c-695b0bb4f0bb 
tempest-TestServerAdvancedOps-1194878159 
tempest-TestServerAdvancedOps-1912438683] Exception during handling message
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 
133, in _process_incoming
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = 
self.dispatcher.dispatch(message)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
153, in dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
self._do_dispatch(endpoint, method, ctxt, args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
122, in _do_dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = 
func(ctxt, **new_args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
f(self, context, *args, **kw)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
kwargs['instance'], e, sys.exc_info())
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in 
__exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in 
force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return 
function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File 
"/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 
do_confirm_resize(context, instance,