Reviewed: https://review.opendev.org/741712 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=806575cfd5327f96e62462f484118d06d17cbe8d Submitter: Zuul Branch: master
commit 806575cfd5327f96e62462f484118d06d17cbe8d Author: Alex Deiter <[email protected]> Date: Fri Jul 17 20:38:55 2020 +0000 Detach is broken for multi-attached fs-based volumes Fixed an issue with detaching multi-attached fs-based volumes. Volume drivers using _HostMountStateManager are special case. _HostMountStateManager ensures that the compute node only attempts to mount a single mountpoint in use by multiple attachments once, and that it is not unmounted until it is no longer in use by any attachments. So we can skip the multiattach check for volume drivers that based on LibvirtMountedFileSystemVolumeDriver. Closes-Bug: 1888022 Change-Id: Ia91b63c0676f42ad8a7a0d16e6870bafc2ee7675 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1888022 Title: Detach is broken for multi-attached fs-based volumes Status in OpenStack Compute (nova): Fix Released Bug description: Description: Detach is broken for multi-attached LibvirtMountedFileSystemVolumeDriver-based volumes. Steps to reproduce: 1. Deploy Devstack Master 2. Configure Nova to use KVM/Libvirt 3. Configure Cinder to use any LibvirtMountedFileSystemVolumeDriver- based volume driver - for example - NFS: cinder.conf: [nfs] volume_driver = cinder.volume.drivers.nfs.NfsDriver volume_backend_name = nfs nas_secure_file_operations = False nfs_snapshot_support = True nas_host = 10.3.35.41 nas_share_path = /volumes/pool1/nas 4. Create a volume type with enabled multi-attach feature: $ cinder type-create multiattach $ cinder type-key multiattach set multiattach="<is> True" $ cinder type-key multiattach set volume_backend_name=nfs 5. Create a volume: $ cinder create --volume-type nfs 1 6. Boot two Nova virtual machines: $ nova boot --flavor m1.nano --image linux --nic none a $ nova boot --flavor m1.nano --image linux --nic none b 7. Attach the volume to both VM's: $ nova volume-attach ac100d66-e92d-40da-a765-fea72ae0af3c 31b702b9-423b-4402-8a6e-1c3dcf84f956 $ nova volume-attach 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb 31b702b9-423b-4402-8a6e-1c3dcf84f956 8. Check Nova CPU service log file: Jul 16 22:40:36 openstack-master-lustre7 nova-compute[74494]: INFO nova.compute.manager [None req-bc029573-7eea-4f56-ba89-060c158f2f75 admin admin] [instance: ac100d66-e92d-40da-a765-fea72ae0af3c] Attaching volume 31b702b9-423b-4402-8a6e-1c3dcf84f956 to /dev/sdb Jul 16 22:40:38 openstack-master-lustre7 nova-compute[74494]: DEBUG nova.virt.libvirt.guest [None req-bc029573-7eea-4f56-ba89-060c158f2f75 admin admin] attach device xml: <disk type="file" device="disk"> <driver name="qemu" type="raw" cache="none" io="native"/> <source file="/opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c/volume-31b702b9-423b-4402-8a6e-1c3dcf84f956"/> <target dev="sdb" bus="scsi"/> <serial>31b702b9-423b-4402-8a6e-1c3dcf84f956</serial> <shareable/> <address type="drive" controller="0" unit="1"/> </disk> Jul 16 22:40:47 openstack-master-lustre7 nova-compute[74494]: INFO nova.compute.manager [None req-d0d5238d-a617-48d4-a439-4c998eea21b5 admin admin] [instance: 0843c96e-2cfe-49ca-a8eb-0d25f806ffeb] Attaching volume 31b702b9-423b-4402-8a6e-1c3dcf84f956 to /dev/sdb Jul 16 22:40:48 openstack-master-lustre7 nova-compute[74494]: DEBUG nova.virt.libvirt.guest [None req-d0d5238d-a617-48d4-a439-4c998eea21b5 admin admin] attach device xml: <disk type="file" device="disk"> <driver name="qemu" type="raw" cache="none" io="native"/> <source file="/opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c/volume-31b702b9-423b-4402-8a6e-1c3dcf84f956"/> <target dev="sdb" bus="scsi"/> <serial>31b702b9-423b-4402-8a6e-1c3dcf84f956</serial> <shareable/> <address type="drive" controller="0" unit="1"/> </disk> 9. Check the mountpoint: $ mount -t nfs4 10.3.35.41:/volumes/pool1/nas on /opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.3.196.7,local_lock=none,addr=10.3.35.41) Looks good and works as expected. 10. Detach the volume at the VM a 11. Detach the volume at the VM b 12. Check the mountpoint: $ mount -t nfs4 10.3.35.41:/volumes/pool1/nas on /opt/stack/data/nova/mnt/0abe5ba79045d7dd179ddc8a4ff1991c type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.3.196.7,local_lock=none,addr=10.3.35.41) it still mounted - but should be unmounted, when we detach the volume at all VM's! 13. Check Nova CPU service log file: Jul 16 22:46:43 openstack-master-lustre7 nova-compute[74494]: INFO nova.virt.libvirt.driver [None req-ec46e195-645e- 4ad7-a495-3b5fd9578935 admin admin] [instance: ac100d66-e92d- 40da-a765-fea72ae0af3c] Detected multiple connections on this host for volume: 31b702b9-423b-4402-8a6e-1c3dcf84f956, skipping target disconnect. Jul 16 22:46:43 openstack-master-lustre7 nova-compute[74494]: INFO nova.virt.libvirt.driver [None req-048bb510-7012-405e-af7b- 2b39772f5ad0 admin admin] [instance: 0843c96e-2cfe-49ca-a8eb- 0d25f806ffeb] Detected multiple connections on this host for volume: 31b702b9-423b-4402-8a6e-1c3dcf84f956, skipping target disconnect. Looks like an error in detach logic. Root cause: nova/compute/manager.py calls: self.driver.destroy(context, instance, network_info, block_device_info) from nova/virt/libvirt/driver.py and destroy(self, context, instance, network_info, block_device_info=None) calls: self.cleanup(context, instance, network_info, block_device_info) and self._cleanup(context, instance, network_info, block_device_info=block_device_info) _disconnect_volume((self, context, connection_info, instance, ncryption=None) and _disconnect_volume checks: if self._should_disconnect_target(context, connection_info, instance) and _should_disconnect_target(context, connection_info, instance) function checks the volume multi-attach property and number of attachments. And if len(attachments) > 1 and VM is running on the current host => it skip volume disconnect. And this logic is correct for true block device drivers (iSCSI/FC), but it don't works as expected for LibvirtMountedFileSystemVolumeDriver-based drivers. All LibvirtMountedFileSystemVolumeDriver-based drivers uses _HostMountStateManager from nova/virt/libvirt/volume/mount.py and this class manages mounts/unmounts and itself keeps track of the number of mounts for a particular volume. So if we exclude the _should_disconnect_target() logic for LibvirtMountedFileSystemVolumeDriver-based volumes, then this class will work correctly for multi-attached volumes. Please take a look my patch. Thank you! To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1888022/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
