Nova has the same issue with local lvm block storage:
http://logs.openstack.org/80/237480/8/check/gate-tempest-dsvm-
full/96d1bff/logs/screen-n-cpu.txt.gz?level=TRACE
** Also affects: nova
Importance: Undecided
Status: New
** Tags added: volumes
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1270192
Title:
cinder volume hanging on removing snapshots
Status in Cinder:
Fix Released
Status in Cinder icehouse series:
Fix Released
Status in OpenStack Compute (nova):
New
Bug description:
On any lvm2 version without lvmetad running, I can get cinder-volume
to hang on issuing lvm related commands after deleting snapshots (that
are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device
mapper entries, and at some point cinder-volume does a lvm related command and
hangs on that. a setting of ignore_suspended_devices = 1 in lvm.conf helps with
that, as lvremove hangs on scanning the device state (which it
needs to do because it doesn't have current information available via
lvmetad).
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of
space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow.
The script then also never finishes and any other lvm command hangs as
well.
apparently in real-life this is either udev or the dd command still
having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1270192/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
