I haven't seen this issue in nova CI results in the last 7 days so I'm
closing this as fixed.
❯ logsearch storedsearch --days 7 bug-2109428-glance-cannot-delete-image-in-use
Running stored search:
bug-2109428-glance-cannot-delete-image-in-use:
files:
- controller/logs/screen-n-api.txt
job-groups:
- nova-devstack
project: openstack/nova
regex:
- The image cannot be deleted because it is in use through the backend
store outside
of Glance.
result: FAILURE
Found builds:
+----------------------------------+---------------------+-----------------+-----------------------------------+---------------------+--------------------------------------------+
| uuid | finished | pipeline |
review | branch | job
|
+----------------------------------+---------------------+-----------------+-----------------------------------+---------------------+--------------------------------------------+
| d8eb68d0def24d0f858f0a7696308395 | 2025-06-03T14:27:54 | check |
https://review.opendev.org/948450 | master |
nova-grenade-multinode |
| a92b9d4ac7af432ea8e2554791bd1944 | 2025-05-31T08:53:49 | periodic-weekly |
None | stable/2025.1 |
tempest-integrated-compute-centos-9-stream |
| 5dc534073c364921a2a0c039b67ed343 | 2025-05-31T08:43:54 | periodic-weekly |
None | unmaintained/zed |
tempest-integrated-compute-centos-9-stream |
| 8e96f5d1817d4a2998982e2f9fded2be | 2025-05-31T08:42:38 | periodic-weekly |
None | unmaintained/2023.1 |
tempest-integrated-compute-centos-9-stream |
| 4774df90113d4d7681c8963264eb67cd | 2025-05-31T08:26:44 | periodic-weekly |
None | stable/2024.1 |
tempest-integrated-compute-centos-9-stream |
| 29277ea6287d4b8c9df09429e2c2d46f | 2025-05-31T08:15:58 | periodic-weekly |
None | stable/2024.2 |
tempest-integrated-compute-centos-9-stream |
| a43b0392f2954e4693c00169aef3a35a | 2025-05-30T02:33:53 | check |
https://review.opendev.org/951226 | master |
nova-grenade-multinode |
| 7a13ba355c27400faa7b79aafd3cc7d2 | 2025-05-29T09:57:11 | check |
https://review.opendev.org/950623 | master | nova-next
|
| ab6f4741f911428ebc208a26823ebacc | 2025-05-29T08:23:24 | check |
https://review.opendev.org/949754 | master |
grenade-skip-level-always |
| 20301661f37941ceb1a4a2a588296291 | 2025-05-29T08:08:32 | check |
https://review.opendev.org/949754 | master |
tempest-integrated-compute |
| 4cd8dfae211443cb8f640afddc88896a | 2025-05-29T09:09:42 | check |
https://review.opendev.org/949754 | master | nova-ceph-multistore
|
| e9f742f169d24b84a85ac9a10107de87 | 2025-05-29T08:12:06 | check |
https://review.opendev.org/949754 | master |
nova-grenade-multinode |
| f74784093a8946dab2a15f95ab342bec | 2025-05-29T07:55:15 | check |
https://review.opendev.org/949754 | master | nova-lvm
|
| 7bf0c69b97d5417da026346dda011200 | 2025-05-29T07:50:27 | check |
https://review.opendev.org/949754 | master | nova-multi-cell
|
| 662427e2ea804c44a248596f56f4a5c9 | 2025-05-29T07:21:28 | check |
https://review.opendev.org/949754 | master | tempest-ipv6-only
|
| aaf0fb9d955f4943adb49493b1fe74b8 | 2025-05-28T13:26:55 | check |
https://review.opendev.org/950623 | master |
nova-grenade-multinode |
| ab3ce94011634332a7f1ca8187fbba98 | 2025-05-28T07:09:08 | check |
https://review.opendev.org/951084 | master | nova-next
|
| 0e3046fc797c4a74a1ec5930ecf7e581 | 2025-05-27T23:11:28 | check |
https://review.opendev.org/942973 | master | nova-multi-cell
|
+----------------------------------+---------------------+-----------------+-----------------------------------+---------------------+--------------------------------------------+
Downloading logs:
Builds with matching logs 0/18:
+------+----------+----------+--------+--------+-----+
| uuid | finished | pipeline | review | branch | job |
+------+----------+----------+--------+--------+-----+
+------+----------+----------+--------+--------+-----+
** Changed in: glance
Status: New => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2109428
Title:
Image delete fails in tempest nova-ceph-multistore job with: "The
image cannot be deleted because it is in use through the backend store
outside of Glance"
Status in Glance:
Fix Released
Bug description:
Example hits from the last 14 days:
❯ logsearch log --project openstack/nova --job nova-ceph-multistore
--result FAILURE --days 14 "The image cannot be deleted because it is
in use through the backend store outside of Glance"
Builds with matching logs 5/15:
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| uuid | finished | pipeline | review
| branch |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
| 5f7dccf493cf4b9d87d87a8f0a052192 | 2025-04-27T12:37:50 | check |
https://review.opendev.org/947212 | master |
| c7f28ff023b747849862c96817ffccdb | 2025-04-26T12:36:32 | check |
https://review.opendev.org/947212 | master |
| f94c9e6667874556849f625a01964367 | 2025-04-25T14:16:05 | check |
https://review.opendev.org/922496 | master |
| f169e50cabbf4e65a89a922b6ac9b29e | 2025-04-25T13:08:02 | check |
https://review.opendev.org/947966 | master |
| 0e53881b2e694ff8996b55bf7c477e79 | 2025-04-25T08:27:00 | check |
https://review.opendev.org/947212 | master |
+----------------------------------+---------------------+----------+-----------------------------------+--------+
I only see it in the nova-ceph-multistore job but in various test cases:
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f7/openstack/5f7dccf493cf4b9d87d87a8f0a052192/testr_results.html
*
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f94/openstack/f94c9e6667874556849f625a01964367/testr_results.html
*
https://40d1580bb656fd0ed240-3f272db0dacf207a646e9867f60c7e03.ssl.cf1.rackcdn.com/openstack/f169e50cabbf4e65a89a922b6ac9b29e/testr_results.html
I looked a bit into the occurrence
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/logs
below.
glance API logs:
https://zuul.opendev.org/t/openstack/build/5f7dccf493cf4b9d87d87a8f0a052192/log/controller/logs/screen-g-api.txt#12123-12132
Apr 27 11:52:19.736399 np0040547480 glance-api[98085]: DEBUG
oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="context_is_admin" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"auth_token": "***", "domain":
null, "global_request_id": null, "is_admin": false, "is_admin_project": true,
"project": "495c6985d5064e17970e2b26b58e56c6", "project_domain": "default",
"project_id": "495c6985d5064e17970e2b26b58e56c6", "read_only": false,
"request_id": "req-20c620eb-2315-4620-b145-98df5a0ed217", "resource_uuid":
null, "roles": ["reader", "member"], "service_catalog": [{"endpoints":
[{"publicURL": "https://158.69.64.188/identity";, "region": "RegionOne"}],
"name": "keystone", "type": "identity"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2/495c6985d5064e17970e2b26b58e56c6";, "region":
"RegionOne"}], "name": "nova_legacy", "type": "compute_legacy"}, {"endpoints":
[{"publicURL": "https://158.69.64.188/volume/v3";, "region": "RegionOne"}],
"name": "cinder", "type": "block-storage"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/placement";, "region": "RegionOne"}], "name":
"placement", "type": "placement"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/image";, "region": "RegionOne"}], "name": "glance",
"type": "image"}, {"endpoints": [{"publicURL":
"https://158.69.64.188/compute/v2.1";, "region": "RegionOne"}], "name": "nova",
"type": "compute"}, {"endpoints": [{"publicURL":
"https://158.69.64.188:9696/networking";, "region": "RegionOne"}], "name":
"neutron", "type": "network"}], "show_deleted": false, "system_scope": null,
"user": "ec6739598d2540019167fc2432e4d312", "user_domain": "default",
"user_identity": "ec6739598d2540019167fc2432e4d312
495c6985d5064e17970e2b26b58e56c6 - default default"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.748380 np0040547480 glance-api[98085]: DEBUG
glance.common.utils [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Sorted locations:
[{'store_name': 'robust'}] {{(pid=98085) sort_image_locations
/opt/stack/glance/glance/common/utils.py:735}}
Apr 27 11:52:19.750448 np0040547480 glance-api[98085]: DEBUG
oslo_policy.policy [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] enforce:
rule="delete_image" creds={"domain_id": null, "is_admin_project": true,
"project_domain_id": "default", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "roles": ["reader", "member"],
"service_project_domain_id": null, "service_project_id": null, "service_roles":
[], "service_user_domain_id": null, "service_user_id": null, "system_scope":
null, "user_domain_id": "default", "user_id":
"ec6739598d2540019167fc2432e4d312"} target={"backup_type": "daily",
"base_image_ref": "08f6a6f3-85eb-46c1-a2ab-e260218b0429", "boot_roles":
"reader,member", "checksum": null, "container_format": "bare", "created_at":
"2025-04-27T11:52:11.000000", "disk_format": "raw", "extra_properties": {},
"hw_cdrom_bus": "ide", "hw_disk_bus": "virtio", "hw_machine_type": "pc",
"hw_video_model": "virtio", "hw_vif_model": "virtio", "image_id":
"345b1922-5189-4d6f-a7af-bf28c7226051", "image_location": "snapshot",
"image_state": "available", "image_type": "backup", "instance_uuid":
"79f599ec-27d2-47cf-9072-be2ab65bee1b", "member": null, "min_disk": 1,
"min_ram": 0, "name": "tempest-backup-3-804999709", "os_glance_import_task":
"a39f3c07-8a30-48c3-ba47-5464019146e0", "os_hash_algo": "sha512",
"os_hash_value": null, "os_hidden": false, "owner":
"495c6985d5064e17970e2b26b58e56c6", "owner_id":
"495c6985d5064e17970e2b26b58e56c6", "owner_project_name":
"tempest-ServerActionsTestOtherB-1525757365", "owner_specified.openstack.md5":
"", "owner_specified.openstack.object": "images/cirros-raw",
"owner_specified.openstack.sha256": "", "owner_user_name":
"tempest-ServerActionsTestOtherB-1525757365-project-member", "project_id":
"495c6985d5064e17970e2b26b58e56c6", "protected": false, "size": 1073741824,
"status": "active", "tags": [], "updated_at": "2025-04-27T11:52:18.000000",
"user_id": "ec6739598d2540019167fc2432e4d312", "virtual_size": null,
"visibility": "private"} {{(pid=98085) enforce
/opt/stack/data/venv/lib/python3.12/site-packages/oslo_policy/policy.py:1040}}
Apr 27 11:52:19.751720 np0040547480 glance-api[98085]: DEBUG
glance_store.multi_backend [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Attempting to import
store rbd {{(pid=98085) _load_multi_store
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/multi_backend.py:170}}
Apr 27 11:52:19.773854 np0040547480 glance-api[98085]: DEBUG
glance_store.capabilities [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Store
glance_store._drivers.rbd.Store doesn't support updating dynamic storage
capabilities. Please overwrite 'update_capabilities' method of the store to
implement updating logics if needed. {{(pid=98085) update_capabilities
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/capabilities.py:91}}
Apr 27 11:52:19.774138 np0040547480 glance-api[98085]: DEBUG
glance_store.driver [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Late loading
location class glance_store._drivers.rbd.StoreLocation {{(pid=98085)
get_store_location_class
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/driver.py:133}}
Apr 27 11:52:19.774268 np0040547480 glance-api[98085]: DEBUG
glance_store.location [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Registering scheme
rbd with {'robust': {'store': <glance_store._drivers.rbd.Store object at
0x7687f29b94c0>, 'location_class': <class
'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'rbd'}}
{{(pid=98085) register_scheme_backend_map
/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/location.py:132}}
Apr 27 11:52:20.682141 np0040547480 glance-api[98085]: WARNING
glance_store._drivers.rbd [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Remove image
345b1922-5189-4d6f-a7af-bf28c7226051 failed. It is in use.: rbd.ImageBusy:
[errno 16] RBD image is busy (error removing image)
Apr 27 11:52:20.687798 np0040547480 glance-api[98085]: WARNING
glance.api.v2.images [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member] Image
345b1922-5189-4d6f-a7af-bf28c7226051 could not be deleted because it is in use:
The image cannot be deleted because it is in use through the backend store
outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be
deleted because it is in use through the backend store outside of Glance.
Apr 27 11:52:20.689146 np0040547480 glance-api[98085]: INFO
eventlet.wsgi.server [None req-20c620eb-2315-4620-b145-98df5a0ed217
tempest-ServerActionsTestOtherB-1525757365
tempest-ServerActionsTestOtherB-1525757365-project-member]
158.69.64.188,158.69.64.188,158.69.64.188 - - [27/Apr/2025 11:52:20] "DELETE
/v2/images/345b1922-5189-4d6f-a7af-bf28c7226051 HTTP/1.1" 409 462 0.956102
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2109428/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
