Hi Umit, Thanks for the detailed analysis and the patch. Your finding that lvchange -aey on the surviving host gets the VM started is useful. I feel this is better handled on the management server side. It already knows which host last held the lock, and ClvmPoolManager.queryCurrentLockHolder (called from VolumeOrchestrator.prepare() on VM start) is already present to reconcile such cases. It checks the other hosts when the last holder is down and tries to activate the LV exclusively. So I'm curious why that didn't kick in here. If you still have the management-server logs from when this happened, could you share the relevant section? Anything around the lock-holder query or activation for that volume (lines like Fast path skip, Recovery activation ... failed, or an lvchange failure) would help us see what happened on that path.
Thanks, Pearl On Tue, Oct 6, 2026 at 8:05 AM Umit Eyigun <[email protected]> wrote: > Hello, > > Follow-up to the previous note. The patch for > ClvmStorageAdaptor.handleMissingDeviceNode is below and attached. > > Today, if the device node is missing, the method activates only volumes > named template-* (lvchange -asy). Every other volume throws "volume exists > in LVM but device node not accessible" and the start path refreshes the > libvirt pool until StartCommand times out. > > The change calls LibvirtComputingResource.activateClvmVolumeExclusive, > which is lvchange -aey with a 300 second timeout. The sanlock lock is not > wiped. The surviving host waits for the dead host's lease to expire, then > the device node appears and the VM can start. > > --- > a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/ClvmStorageAdaptor.java > +++ > b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/ClvmStorageAdaptor.java > @@ -1,6 +1,7 @@ > private String handleMissingDeviceNode(String volumeUuid, String > vgName, KVMStoragePool pool) { > if (Storage.StoragePoolType.CLVM_NG.equals(pool.getType()) && > volumeUuid.startsWith("template-")) { > return activateTemplateAndGetPath(volumeUuid, vgName); > } > - throw new CloudRuntimeException(String.format( > - "Could not find volume %s in VG %s - volume exists in LVM > but device node not accessible", > - volumeUuid, vgName)); > + String path = "/dev/" + vgName + "/" + volumeUuid; > + logger.info("Volume {} device node missing. Taking exclusive > lock.", volumeUuid); > + // Sanlock lease is not wiped. lvchange -aey waits until the dead > host's lease expires. > + LibvirtComputingResource.activateClvmVolumeExclusive(path); > + if (new File(path).exists()) { > + return path; > + } > + throw new CloudRuntimeException(String.format( > + "Could not find volume %s in VG %s - volume exists in LVM > but device node not accessible", > + volumeUuid, vgName)); > } > > Regards, > Umit Eyigun > Trtek Yazilim A.S. > [email protected] > -- Pearl Dsilva
