Hello team,
I am having the following problem, The Libvirt is unable to convert from
Secondary Storage to CEPH RBD.
CEPH is purely IPV6, The RBD was created, There are no error on the CEPH side.
The Initial VM Creation starts and then stops, they never come enabled,
the initial VMs are deleted and the process start in a loop.
Libvirt is up and running but throwing the following errors:
root@kvmvcompatl2001:~# systemctl status libvertd
Unit libvertd.service could not be found.
root@kvmvcompatl2001:~# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor
preset: enabled)
Active: active (running) since Tue 2025-04-01 01:19:22 UTC; 8h ago
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 2271 (libvirtd)
Tasks: 19 (limit: 32768)
Memory: 98.5M
CPU: 9.064s
CGroup: /system.slice/libvirtd.service
└─2271 /usr/sbin/libvirtd --listen
Apr 01 02:32:57 kvmvcompatl2001 libvirtd[2271]: invalid argument: Connections
from inside daemon must be direct
Apr 01 02:32:57 kvmvcompatl2001 libvirtd[2271]: End of file while reading data:
Input/output error
Apr 01 02:33:02 kvmvcompatl2001 libvirtd[2271]: invalid argument: Connections
from inside daemon must be direct
Apr 01 02:33:02 kvmvcompatl2001 libvirtd[2271]: End of file while reading data:
Input/output error
Apr 01 02:33:27 kvmvcompatl2001 libvirtd[2271]: invalid argument: Connections
from inside daemon must be direct
Apr 01 02:33:27 kvmvcompatl2001 libvirtd[2271]: End of file while reading data:
Input/output error
Apr 01 02:33:32 kvmvcompatl2001 libvirtd[2271]: invalid argument: Connections
from inside daemon must be direct
the agent.log is showing the following in a loop:
2025-04-01 02:33:54,480 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:[]) (logid:d64e452c) Starting copy from source image
/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
to RBD image 3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0
2025-04-01 02:33:54,480 DEBUG [utils.script.Script] (agentRequest-Handler-4:[])
(logid:d64e452c) Executing command [qemu-img convert -O raw -U --image-opts
driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30
].
2025-04-01 02:33:54,532 WARN [utils.script.Script] (agentRequest-Handler-4:[])
(logid:d64e452c) Execution of process [7500] for command [qemu-img convert -O
raw -U --image-opts
driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30
] failed.
2025-04-01 02:33:54,532 DEBUG [utils.script.Script] (agentRequest-Handler-4:[])
(logid:d64e452c) Exit value of process [7500] for command [qemu-img convert -O
raw -U --image-opts
driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30
] is [1].
2025-04-01 02:33:54,532 WARN [utils.script.Script] (agentRequest-Handler-4:[])
(logid:d64e452c) Process [7500] for command [qemu-img convert -O raw -U
--image-opts
driver=qcow2,file.filename=/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30
] encountered the error: [qemu-img:
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30:
error while converting raw: invalid conf option
550:5607:fff0::22:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported:
No such file or directory].
2025-04-01 02:33:54,532 ERROR [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:[]) (logid:d64e452c) Failed to convert from
/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
to
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30
the error was: qemu-img:
rbd:3cephUserandPool/f32a0f81-5661-41ac-832f-f5dfffa8b1e0:mon_host=20XX:YYYY:ZZZZ:LLLL::OO:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported=cephx:id=3cephUserandPool:key=KEYGENERATEDBYCEPH:rbd_default_format=2:client_mount_timeout=30:
error while converting raw: invalid conf option
550:5607:fff0::22:24;20XX:YYYY:ZZZZ:LLLL::OO:26;20XX:YYYY:ZZZZ:LLLL::OO:auth_supported:
No such file or directory
2025-04-01 02:33:54,532 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:[]) (logid:d64e452c) Attempting to remove storage pool
a1aa3257-b554-3896-a168-a593ebde9994 from libvirt
2025-04-01 02:33:54,532 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:[]) (logid:d64e452c) Looking for libvirtd connection
at: qemu:///system
2025-04-01 02:33:54,534 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:[]) (logid:d64e452c) Storage pool
a1aa3257-b554-3896-a168-a593ebde9994 has no corresponding secret. Not removing
any secret.
2025-04-01 02:33:54,577 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:[]) (logid:d64e452c) Storage pool
a1aa3257-b554-3896-a168-a593ebde9994 was successfully removed from libvirt.
2025-04-01 02:33:54,579 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:[])
(logid:d64e452c) Seq 3-8729383452727574659: { Ans: , MgmtId: 250977680725600,
via: 3, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
Failed to copy
/mnt/a1aa3257-b554-3896-a168-a593ebde9994/5ccb81a1-26ec-4d57-a02a-37f81e09be08.qcow2
to
f32a0f81-5661-41ac-832f-f5dfffa8b1e0","wait":"0","bypassHostMaintenance":"false"}}]
}
2025-04-01 02:33:54,621 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:[])
(logid:d64e452c) Request:Seq 3-8729383452727574660: { Cmd , MgmtId:
250977680725600, via: 3, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.138.135","forceStop":"false","volumesToDisconnect":[],"vmName":"v-172-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2025-04-01 02:33:54,621 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:[])
(logid:d64e452c) Processing command: com.cloud.agent.api.StopCommand
2025-04-01 02:33:54,621 DEBUG [resource.wrapper.LibvirtStopCommandWrapper]
(agentRequest-Handler-3:[]) (logid:d64e452c) backing up the cmdline
2025-04-01 02:33:57,691 DEBUG [resource.wrapper.LibvirtStopCommandWrapper]
(agentRequest-Handler-3:[]) (logid:d64e452c) Failed to backup cmdline file due
to There was a problem while connecting to 169.254.138.135:3922
Error logs showing:
libvirt: Remote Driver error : invalid argument: Connections from inside daemon
must be direct
libvirt: QEMU Driver error : Domain not found: no domain with matching name
'v-172-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name
'v-172-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name
'v-172-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name
'v-172-VM'
libvirt: QEMU Driver error : Domain not found: no domain with matching name
'v-172-VM'
libvirt: Storage Driver error : Storage pool not found: no storage pool with
matching uuid 'a1aa3257-b554-3896-a168-a593ebde9994'
libvirt: Secrets Driver error : Secret not found: no secret with matching uuid
'a1aa3257-b554-3896-a168-a593ebde9994'
Please kindly guide us.
Thank you
Tata Y.
