rp- commented on issue #12362: URL: https://github.com/apache/cloudstack/issues/12362#issuecomment-3722519657
@bilalinamdar I tried to find the first failuers, but it seems the logs only contain, were you recreated the cluster with nfs-primary first, because in the logs I only see that linstor was added later. And then the first error is already this: ``` 2026-01-01 21:51:13,639 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Attempting to create storage pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262 (NetworkFilesystem) in libvirt 2026-01-01 21:51:13,661 WARN [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Storage pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262 was not found running in libvirt. Need to create it. 2026-01-01 21:51:13,661 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Didn't find an existing storage pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262 by UUID, checking for pools with duplicate paths 2026-01-01 21:51:13,674 WARN [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:d379d526) Execution of process [47221] for command [/bin/bash -c mountpoint -q /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262 ] failed. 2026-01-01 21:51:13,675 WARN [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:d379d526) Process [47221] for command [/bin/bash -c mountpoint -q /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262 ] encountered the error: [32]. 2026-01-01 21:51:13,791 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Trying to fetch storage pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262 from libvirt 2026-01-01 21:51:14,237 INFO [kvm.storage.LinstorStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Linstor: Created drbd device: /dev/drbd1000 2026-01-01 21:51:14,395 WARN [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:d379d526) Execution of process [47237] for command [qemu-img convert -n --target-is-zero -W -S 1M -O raw -t none -U /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262/4d1fb221-a2c3-32c6-976a-b6e637819567.raw /dev/drbd1000 ] failed. 2026-01-01 21:51:14,396 WARN [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:d379d526) Process [47237] for command [qemu-img convert -n --target-is-zero -W -S 1M -O raw -t none -U /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262/4d1fb221-a2c3-32c6-976a-b6e637819567.raw /dev/drbd1000 ] encountered the error: [qemu-img: output file is smaller than input file]. 2026-01-01 21:51:14,396 ERROR [kvm.storage.LinstorStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: output file is smaller than input file 2026-01-01 21:51:15,478 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Attempting to remove storage pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262 from libvirt 2026-01-01 21:51:15,497 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Storage pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262 has no corresponding secret. Not removing any secret. 2026-01-01 21:51:15,515 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) deleteStoragePool removed pool from libvirt, but libvirt had trouble unmounting the pool. Trying umount location /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262 again in a few seconds 2026-01-01 21:51:20,525 WARN [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:d379d526) Execution of process [47340] for command [/bin/bash -c sleep 5 && umount /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262 ] failed. 2026-01-01 21:51:20,525 WARN [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:d379d526) Process [47340] for command [/bin/bash -c sleep 5 && umount /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262 ] encountered the error: [umount.nfs4: /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262: device is busy]. 2026-01-01 21:51:20,526 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Succeeded in unmounting /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262 2026-01-01 21:51:20,540 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:d379d526) Failed to destroy libvirt pool 8f125dcf-e274-37ee-b6cd-a9312d3b1262: org.libvirt.LibvirtException: internal error: Child process (/usr/bin/umount /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262) unexpected exit status 16: umount.nfs4: /mnt/8f125dcf-e274-37ee-b6cd-a9312d3b1262: device is busy ``` But now as I look at it, I found it more interessting that there is already an error mounting the nfs secondary share and I'm a bit surprised that it then even tried to `qemu-img convert` it, so maybe we are just looking at the wrong error, and the failing nfs mount is the real problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
