Hi, We have plenty of space, I have checked with both df -h and df -i.
Here are the vdsm logs from one node. 2021-03-08 12:09:15,550+0100 INFO (jsonrpc/7) [vdsm.api] START deleteImage(sdUUID=u'fb14e013-15f1-49b6-b129-210517aca0da', spUUID=u'aec485fa-cdbb-4979-8dc9-a2376559b2a4', imgUUID=u'459872a1-8783-463a-98ae-9e56d8883abb', postZero=u'false', force=u'false', discard=False) from=::ffff:172.16.122.253,44650, flow_id=39b2eddb, task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:48) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume symlink u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/46bf0806-8d16-40db-92df-8436f43628a4' (blockSD:722) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Unlink volume symlink u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a' (blockSD:722) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] removed: [u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c/27638d78-e935-44f4-b0cc-5ee2459e204a'] (blockSD:729) 2021-03-08 12:09:15,719+0100 INFO (tasks/9) [storage.StorageDomainManifest] Removing image directory u'/rhev/data-center/mnt/blockSD/fb14e013-15f1-49b6-b129-210517aca0da/images/01af322e-ff54-4981-9842-0f0a608da17c' (blockSD:735) 2021-03-08 12:09:15,729+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] FINISH task 9a57dcf5-85b2-4c7c-8ffb-f11eecc2313f (threadPool:210) 2021-03-08 12:09:16,671+0100 WARN (jsonrpc/7) [storage.LVM] Removing stale lv: fb14e013-15f1-49b6-b129-210517aca0da/46bf0806-8d16-40db-92df-8436f43628a4 (lvm:564) 2021-03-08 12:09:17,920+0100 WARN (tasks/7) [storage.LVM] lv: 46bf0806-8d16-40db-92df-8436f43628a4 not found in lvs vg: fb14e013-15f1-49b6-b129-210517aca0da response (lvm:736) 2021-03-08 12:09:17,921+0100 ERROR (tasks/7) [storage.TaskManager.Task] (Task='d4aa81a1-9adc-45c9-be46-05bc63862f4e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1966, in deleteVolume vol = dom.produceVolume(imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 913, in produceVolume volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 814, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 62, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 151, in validate raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'46bf0806-8d16-40db-92df-8436f43628a4',) 2021-03-08 12:09:17,946+0100 INFO (tasks/7) [storage.ThreadPool.WorkerThread] FINISH task d4aa81a1-9adc-45c9-be46-05bc63862f4e (threadPool:210) 2021-03-08 12:09:18,273+0100 WARN (jsonrpc/7) [storage.LVM] Command ['/usr/sbin/lvm', 'lvchange', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/2181ad2edfa7b485c6c9ce90045609718$|^/dev/mapper/2763fdda70ca458906c9ce90045609718$|^/dev/mapper/28ba7d1d12c7099ef6c9ce90045609718$|^/dev/mapper/2e9d013564898d36f6c9ce90045609718$|", "r|.*|"] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--autobackup', 'n', '-a', 'y', '--deltag', u'IU_459872a1-8783-463a-98ae-9e56d8883abb', '--addtag', u'IU__remove_me_459872a1-8783-463a-98ae-9e56d8883abb', 'fb14e013-15f1-49b6-b129-210517aca0da/5d51973d-4198-4cec-994a-727317640ccb'] succeeded with warnings: [' WARNING: Combining activation change with other commands is not advised.', ' Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca 458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.', ' Scan of VG fb14e013-15f1-49b6-b129-210517aca0da from /dev/mapper/2763fdda70ca458906c9ce90045609718 found metadata seqno 26448 vs previous 26447.'] (lvm:385) 2021-03-08 12:09:18,281+0100 INFO (jsonrpc/7) [vdsm.api] FINISH deleteImage return=None from=::ffff:172.16.122.253,44650, flow_id=39b2eddb, task_id=969cdd60-9b4c-43d9-960a-2a3b467b227c (api:54) 2021-03-08 12:09:18,293+0100 INFO (tasks/9) [storage.ThreadPool.WorkerThread] START task 969cdd60-9b4c-43d9-960a-2a3b467b227c (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x7f7b606c27e8>>, args=None) (threadPool:208) 2021-03-08 12:09:18,295+0100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Image.delete succeeded in 2.75 seconds (__init__:312) 2021-03-08 12:09:18,299+0100 INFO (jsonrpc/0) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:172.16.122.253,44628, task_id=f6411446-8bed-4766-a038-892ebcaab65e (api:48) We have a SAN storage, with plenty of space, plus some NFS storage. But, again, there is plenty of space on all of those. From what I could find, the sanlock issue is not specifically related to space on the storage devices. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IPEV2PSURWQJYZYLK6TQXPQ2BWYD3TD5/