** Description changed:

- Server has dependency of original base image used to Create Server or
- Rebuild Server. If original image is deleted from glance, nova has
- server.image.id reference to non-existent image. Rescue in stable device
- rescue mode leads to driver error and Server in ERROR state.
+ [ Impact ]
+ 
+ When rescuing an instance using stable device rescue mode (i.e. rescue
+ image contains `hw_rescue_bus` or `hw_rescue_device` properties), Nova's
+ libvirt driver attempts to look up the original base image metadata from
+ Glance via `instance.image_ref`.
+ 
+ If the original base image (or snapshot/backup image after a rebuild)
+ has been deleted from Glance, the lookup raises
+ `nova.exception.ImageNotFound`. This leads to a driver error and puts
+ the instance into the unrecoverable `ERROR` state.
+ 
+ The fix catches `nova.exception.ImageNotFound` during image metadata
+ retrieval in `nova/virt/libvirt/driver.py` and falls back to using the
+ stashed `instance.image_meta` already saved on the instance.
+ 
+ [ Test Plan ]
+ 
+ 1. Prepare test images:
+    $ openstack image save --file /tmp/cirros.qcow2 cirros
+    $ openstack image create --disk-format qcow2 --container-format bare 
--file /tmp/cirros.qcow2 test-base-img
+    $ openstack image create --disk-format qcow2 --container-format bare 
--file /tmp/cirros.qcow2 \
+        --property hw_rescue_bus=scsi --property hw_rescue_device=cdrom 
test-rescue-img
+ 
+ 2. Boot an instance from the base image and wait until it is ACTIVE:
+    $ openstack server create --image test-base-img --flavor m1.tiny --network 
private test-server
+    $ openstack server show test-server -f value -c status
+ 
+ 3. Delete the base image from Glance:
+    $ openstack image delete test-base-img
+ 
+ 4. Rescue the server with the rescue image:
+    $ openstack server rescue --image test-rescue-img test-server
+ 
+ Verification:
+ - Without fix: The server rescue fails with "Driver Error: Image <UUID> could 
not be found." and the server transitions to ERROR state.
+ - With fix: The server successfully transitions to RESCUE state. Running 
`openstack server unrescue test-server` successfully restores the server to 
ACTIVE state.
+ 
+ [ Where problems could occur ]
+ 
+ - The modification is strictly localized to the libvirt driver's stable 
rescue path (`nova/virt/libvirt/driver.py`) inside the 
`hardware.check_hw_rescue_props()` block.
+ - It only triggers when `ImageNotFound` is encountered; regular rescue 
operations with existing Glance images follow the existing code path.
+ - Potential regressions could occur if the stashed `instance.image_meta` was 
malformed or missing attributes required for disk mapping generation, which 
could cause rescue failures. However, `instance.image_meta` is saved upon 
instance create/rebuild and contains the necessary properties.
+ - Standard rescue (non-stable rescue mode) and boot-from-volume instances 
without `image_ref` are unaffected.
+ 
+ [ Other Info ]
+ 
+ - Upstream commit: 714894a7ff81e23f5ac7545a6b91ea484447a988
+ - Fixed upstream in OpenStack Nova 30.0.0 (Dalmatian).
+ 
+ Original Bug Description Below
+ ===========
+ Server has dependency of original base image used to Create Server or Rebuild 
Server. If original image is deleted from glance, nova has server.image.id 
reference to non-existent image. Rescue in stable device rescue mode leads to 
driver error and Server in ERROR state.
  
  Step to reproduce:
  1. Create server from glance image.
  2. Delete original glance image.
  3. Run rescue server with input parameter of rescue image CD with 
hw_rescue_device and hw_rescue_bus image properties defined.
  Second use case:
  1. Restore Server from backup (with server Rebuild)
  2. Delete backup image.
  3. Run rescue in stable device mode.
  
  Result: Instance 73706a9a-e976-4024-8068-c439404ec953 cannot be rescued: 
Driver Error: Image be0e302d-79bd-44e1-93a5-68661e93b43a could not be found.
  Result2: VM in ERROR state.
  
  Expected: VM returned to original state, or in shutdown state.
  Expected2: I see no reason to keep dependency from original image for any 
operation with server. I tested all other operations with server, and they all 
were succesful, even if base image deleted. Libvirt driver code shows that 
rescue mode use only 1 property from image_meta to determine bus type for 
device attached. So I see no reason to raise exception if image_meta data if 
empty.
  
  Stack trace for openstack ZED:
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [None 
req-6611c9f1-63f1-497a-a1bc-51e770d2df09 a27cbf048299483592d84f13e9fe9797 
b7df78883aad4ef7ba0a2778c7741462 - - 8a096fed8060480f8f906f57fca19780 
8a096fed8060480f8f906f57fca19780] [instance: 
73706a9a-e976-4024-8068-c439404ec953] Error trying to Rescue Instance: 
nova.exception.ImageNotFound: Image be0e302d-79bd-44e1-93a5-68661e93b43a could 
not be found.
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] Traceback (most recent call last):
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 285, in show
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     image = self._client.call(context, 2, 
'get', args=(image_id,))
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 191, in call
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     result = getattr(controller, 
method)(*args, **kwargs)
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     return self._get(image_id)
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in 
inner
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     return RequestIdProxy(wrapped(*args, 
**kwargs))
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     resp, body = 
self.http_client.get(url, headers=header)
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     return self.request(url, 'GET', 
**kwargs)
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 380, in 
request
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     return self._handle_response(resp)
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 120, in 
_handle_response
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     raise exc.from_response(resp, 
resp.content)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 285, in show
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] image = self._client.call(context, 2, 
'get', args=(image_id,))
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 191, in call
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] result = getattr(controller, 
method)(*args, **kwargs)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] return self._get(image_id)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in 
inner
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] return RequestIdProxy(wrapped(*args, 
**kwargs))
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] resp, body = self.http_client.get(url, 
headers=header)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] return self.request(url, 'GET', **kwargs)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 380, in 
request
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] return self._handle_response(resp)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 120, in 
_handle_response
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] raise exc.from_response(resp, 
resp.content)
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] glanceclient.exc.HTTPNotFound: HTTP 404 
Not Found: No image found with ID be0e302d-79bd-44e1-93a5-68661e93b43a
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] During handling of the above exception, 
another exception occurred:
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]
  2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] Traceback (most recent call last):
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4485, in 
rescue_instance
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     self.driver.rescue(context, instance, 
network_info,
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4249, in 
rescue
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     image_meta = 
objects.ImageMeta.from_image_ref(
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/objects/image_meta.py", line 151, in 
from_image_ref
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     image_meta = image_api.get(context, 
image_ref)
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 1205, in get
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     return session.show(context, image_id,
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]   File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 287, in show
- 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953]     
_reraise_translated_image_exception(image_id)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4485, in 
rescue_instance
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] self.driver.rescue(context, instance, 
network_info,
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4249, in 
rescue
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] image_meta = 
objects.ImageMeta.from_image_ref(
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/objects/image_meta.py", line 151, in 
from_image_ref
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] image_meta = image_api.get(context, 
image_ref)
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 1205, in get
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] return session.show(context, image_id,
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] File 
"/usr/lib/python3/dist-packages/nova/image/glance.py", line 287, in show
+ 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 
73706a9a-e976-4024-8068-c439404ec953] 
_reraise_translated_image_exception(image_id)

** Summary changed:

- Server Rescue leads to VM ERROR state if VM base image is deleted
+ [SRU] Server Rescue leads to VM ERROR state if VM base image is deleted

** Tags added: sts

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2002606

Title:
  [SRU] Server Rescue leads to VM ERROR state if VM base image is
  deleted

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2002606/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to