[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend
** Changed in: nova/mitaka Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1580625 Title: volume-backed resize intermittently fails with rbd imagebackend Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) liberty series: Fix Released Status in OpenStack Compute (nova) mitaka series: Fix Released Bug description: After this Tempest change landed to test volume-backed resize: https://review.openstack.org/#/c/314816/ The ceph plugin job has been intermittently failing the test: http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full- devstack-plugin- ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server do_confirm_resize(context, instance, migration.id) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py",
[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend
** Changed in: nova/liberty Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1580625 Title: volume-backed resize intermittently fails with rbd imagebackend Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) liberty series: Fix Released Status in OpenStack Compute (nova) mitaka series: Fix Committed Bug description: After this Tempest change landed to test volume-backed resize: https://review.openstack.org/#/c/314816/ The ceph plugin job has been intermittently failing the test: http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full- devstack-plugin- ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server do_confirm_resize(context, instance, migration.id) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File
[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend
Reviewed: https://review.openstack.org/315187 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=671bb2651d9d1a07947690468825581d30482dcb Submitter: Jenkins Branch:master commit 671bb2651d9d1a07947690468825581d30482dcb Author: Nicolas SimondsDate: Wed May 11 10:52:52 2016 -0700 imagebackend: Check that the RBD image exists before trying to cleanup In volume-backed setups, there is no image to clean up, so any attempts to cleanup the resize snapshots will fail by definition. Make sure the image exists first. Change-Id: I25f65bcc76b83f31a8fce77c2b751d2d167ffc7e Closes-Bug: 1580625 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1580625 Title: volume-backed resize intermittently fails with rbd imagebackend Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) liberty series: In Progress Status in OpenStack Compute (nova) mitaka series: In Progress Bug description: After this Tempest change landed to test volume-backed resize: https://review.openstack.org/#/c/314816/ The ceph plugin job has been intermittently failing the test: http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full- devstack-plugin- ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context,
[Yahoo-eng-team] [Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend
** Also affects: nova/liberty Importance: Undecided Status: New ** Also affects: nova/mitaka Importance: Undecided Status: New ** Changed in: nova/liberty Status: New => In Progress ** Changed in: nova/mitaka Status: New => Confirmed ** Changed in: nova/liberty Status: In Progress => Confirmed ** Changed in: nova/mitaka Importance: Undecided => High ** Changed in: nova/liberty Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1580625 Title: volume-backed resize intermittently fails with rbd imagebackend Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) liberty series: Confirmed Status in OpenStack Compute (nova) mitaka series: Confirmed Bug description: After this Tempest change landed to test volume-backed resize: https://review.openstack.org/#/c/314816/ The ceph plugin job has been intermittently failing the test: http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full- devstack-plugin- ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server do_confirm_resize(context, instance,