Public bug reported:
With Ceph as cinder backend, detaching a volume doesn't work after the
following steps:
# boot volume, create instance
nova boot --poll --key-name testkey --flavor 2 --image
c848067a-a8bb-4631-a6a6-ed0404d444a9 test01
cinder create 1
export INSTANCEID=adcfcda6-5a27-4c85-a770-210a6b08894b
export VOLID=fad11ab9-3a1f-43d7-83c3-55bded6280e4
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state available $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
sleep 1
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state in-use $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
After the last step, the volume hangs in "detaching" state. nova-compute.log
shows:
2015-03-17 08:06:06.456 1342 AUDIT nova.compute.manager
[req-68662304-4140-4e19-b4dc-ec65cb43ec2b 04030aff6a0a4416bc532863109343ff
362d7cfe3e7b491290f7d1684db0a48f] [instance:
adcfcda6-5a27-4c85-a770-210a6b08894b] Detach volume
fad11ab9-3a1f-43d7-83c3-55bded6280e4 from mountpoint /dev/vdb
2015-03-17 08:06:06.574 1342 ERROR oslo.messaging.rpc.dispatcher [-] Exception
during message handling: expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
133, in _dispatch_and_reply
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
176, in _dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
122, in _do_dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher payload)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher pass
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in
detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
self._detach_volume(context, instance, bdm)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in
_detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
connection_info = jsonutils.loads(bdm.connection_info)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line
164, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
json.loads(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/json/__init__.py", line 307, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
_default_decoder.decode(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/json/decoder.py", line 319, in decode
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher obj, end =
self.raw_decode(s, idx=_w(s, 0).end())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher TypeError:
expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] Returning
exception expected string or buffer to caller
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-]
['Traceback (most recent call last):\n', ' File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
133, in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', ' File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n', ' File "/usr/lib64/python2.6/site-packages/nova/exception.py",
line 88, in wrapped\n payload)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped\n
return f(self, context, *args, **kw)\n', ' File "/usr/lib64/python2
.6/site-packages/nova/compute/manager.py", line 282, in decorated_function\n
pass\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in
decorated_function\n return function(self, context, *args, **kwargs)\n', '
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in
decorated_function\n e, sys.exc_info())\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in
decorated_function\n return function(self, context, *args, **kwargs)\n', '
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267,
in detach_volume\n self._detach_volume(context, instance, bdm)\n',
' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line
4212, in _detach_volume\n connection_info =
jsonutils.loads(bdm.connection_info)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line
164, in loads\n return json.loads(s)\n', ' File
"/usr/lib64/python2.6/json/__init__.py", line 307, in loads\n return
_default_decoder.decode(s)\n', ' File "/usr/lib64/python2.6/json/decoder.py",
line 319, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n',
'TypeError: expected string or buffer\n']
The problem seems that the entry in the block_device_mapping table is
removed for the attached volume which leads to this error.
** Affects: nova
Importance: Undecided
Status: New
** Tags: ceph volumes
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1432984
Title:
volume hangs in "detaching" state after cinder "reset-state" was used
Status in OpenStack Compute (Nova):
New
Bug description:
With Ceph as cinder backend, detaching a volume doesn't work after the
following steps:
# boot volume, create instance
nova boot --poll --key-name testkey --flavor 2 --image
c848067a-a8bb-4631-a6a6-ed0404d444a9 test01
cinder create 1
export INSTANCEID=adcfcda6-5a27-4c85-a770-210a6b08894b
export VOLID=fad11ab9-3a1f-43d7-83c3-55bded6280e4
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state available $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
sleep 1
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state in-use $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
After the last step, the volume hangs in "detaching" state. nova-compute.log
shows:
2015-03-17 08:06:06.456 1342 AUDIT nova.compute.manager
[req-68662304-4140-4e19-b4dc-ec65cb43ec2b 04030aff6a0a4416bc532863109343ff
362d7cfe3e7b491290f7d1684db0a48f] [instance:
adcfcda6-5a27-4c85-a770-210a6b08894b] Detach volume
fad11ab9-3a1f-43d7-83c3-55bded6280e4 from mountpoint /dev/vdb
2015-03-17 08:06:06.574 1342 ERROR oslo.messaging.rpc.dispatcher [-]
Exception during message handling: expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
133, in _dispatch_and_reply
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
176, in _dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
122, in _do_dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher payload)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher pass
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in
decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in
detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
self._detach_volume(context, instance, bdm)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in
_detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
connection_info = jsonutils.loads(bdm.connection_info)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line
164, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
json.loads(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/json/__init__.py", line 307, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return
_default_decoder.decode(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/json/decoder.py", line 319, in decode
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher obj, end
= self.raw_decode(s, idx=_w(s, 0).end())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher TypeError:
expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-]
Returning exception expected string or buffer to caller
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-]
['Traceback (most recent call last):\n', ' File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
133, in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', ' File
"/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n', ' File "/usr/lib64/python2.6/site-packages/nova/exception.py",
line 88, in wrapped\n payload)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped\n
return f(self, context, *args, **kw)\n', ' File "/usr/lib64/pytho
n2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function\n
pass\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in
decorated_function\n return function(self, context, *args, **kwargs)\n', '
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in
decorated_function\n e, sys.exc_info())\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in
decorated_function\n return function(self, context, *args, **kwargs)\n', '
File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267,
in detach_volume\n self._detach_volume(context, instance, bdm)\n
', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line
4212, in _detach_volume\n connection_info =
jsonutils.loads(bdm.connection_info)\n', ' File
"/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line
164, in loads\n return json.loads(s)\n', ' File
"/usr/lib64/python2.6/json/__init__.py", line 307, in loads\n return
_default_decoder.decode(s)\n', ' File "/usr/lib64/python2.6/json/decoder.py",
line 319, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n',
'TypeError: expected string or buffer\n']
The problem seems that the entry in the block_device_mapping table is
removed for the attached volume which leads to this error.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1432984/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
