** Description changed: - Description - =========== - The following commit (1) is causing volume attachments which fail due to libvirt device attach erros to be silently ignored and Nova report the attachment as successful. + [Impact] + + The following commit (1) is causing volume attachments which fail due to + libvirt device attach erros to be silently ignored and Nova report the + attachment as successful. It seems that the original intention of the commit was to log a condition and re-raise the exeption, but if the exception is of type libvirt.libvirtError and does not contain the searched pattern, the exception is ignored. If you unindent the raise statement, errors are reported again. In our case we had ceph/apparmor configuration problems in compute nodes which prevented virsh attaching the device; volumes appeared as successfully attached but the corresponding block device missing in guests VMs. Other libvirt attach error conditions are ignored also, as when you have already occuppied device names (i.e. 'Target vdb already exists', device is busy, etc.) (1) https://github.com/openstack/nova/commit/78891c2305bff6e16706339a9c5eca99a84e409c - Steps to reproduce - ================== - This is somewhat hacky, but is a quick way to provoke a virsh attach error: - - virsh detach-disk <domain> vdb - - update nova & cinder DB as if volume is detached - - re-attach volume - - volume is marked as attached, but VM block device is missing + [Test Case] - Expected result - =============== - - Error 'libvirtError: Requested operation is not valid: target vdb already exists' should be raised, and volume not attached + * Deploy any OpenStack version up to Pike , which includes ceph backed cinder + * Create a guest VM (openstack server ...) + * Create a test cinder volume - Actual result - ============= - - Attach successful but virsh block device not created + $ openstack volume create test --size 10 - Environment - =========== - - Openstack version Queens + * Force a drop on ceph traffic. Run the following command on the nova + hypervisor on which the server runs. + + $ iptables -A OUTPUT -d ceph-mon-addr -p tcp --dport 6800 -j DROP + + * Attach the volume to a running instance. + + $ openstack server add volume 7151f507-a6b7-4f6d-a4cc-fd223d9feb5d + 742ff117-21ae-4d1b-a52b-5b37955716ff + + * This should cause the volume attachment to fail + + $ virsh domblklist instance-xxxxx + Target Source + ------------------------------------------------ + vda nova/7151f507-a6b7-4f6d-a4cc-fd223d9feb5d_disk + + No volume should attached after this step. + + * If the behavior is fixed: + + * Check that openstack server show , doesn't displays the displays the volume as attached. + * Check that proper log entries states the libvirt exception and error. + + * If the behavior isn't fixed: + + * openstack server show <ID> , will display the volume in the + volumes_attached property. + + [Expected result] + + * Volume attach fails and a proper exception is logged. + + [Actual result] + + * Volume attach fails but remains connected to the host and no further + exception gets logged. + + [Regression Potential] + + * We haven't identified any regression potential on this SRU. + + [Other Info] + + * N/A + + Description
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1825882 Title: Virsh disk attach errors silently ignored To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1825882/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
