Hey everyone, I had to rework my network and reinstall cloudstack. While I am getting the hosts to appear, they will not start any system VMs. They keep failing and iterating. It looks like there is an error in libvirtd (I am also using systemd for my networking), but I can't quite seem to figure out what is causing this issue. I am using KVM as my hypervisor base and am running everything on Ubuntu 20.04. (The controller is in a VM if that makes a difference.)
The main part that I have not seen in my error before is this: 2021-10-07 08:59:37,558 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:e381c0f1) Can not find a connection for Instance s-1539-VM. Assuming the default connection. 2021-10-07 08:59:37,727 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:e381c0f1) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied. 2021-10-07 08:59:37,727 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-4:null) (logid:e381c0f1) Groovy scripting engine is not initialized. Data transformation skipped. It also appears that libvirt is trying to connect to my primary storage (which is Ceph), but cannot access the cloudstack key. I haven't seen this error before and cannot find a solution for it. Storage is showing up in the cloudstack management interface. Oct 07 09:31:25 gd-cs01-clst01-compute01 libvirtd[47830]: 2021-10-07T09:31:25.780-0400 7f2836766700 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> Oct 07 09:31:25 gd-cs01-clst01-compute01 libvirtd[47830]: 2021-10-07T09:31:25.780-0400 7f2836766700 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> Oct 07 09:31:31 gd-cs01-clst01-compute01 libvirtd[47830]: Failed to connect socket to '/var/run/libvirt/virtlxcd-sock': No such file or directory Oct 07 09:31:31 gd-cs01-clst01-compute01 libvirtd[47830]: End of file while reading data: Input/output error Oct 07 09:31:33 gd-cs01-clst01-compute01 libvirtd[47830]: Failed to connect socket to '/var/run/libvirt/virtlxcd-sock': No such file or directory Oct 07 09:31:33 gd-cs01-clst01-compute01 libvirtd[47830]: End of file while reading data: Input/output error Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: 2021-10-07T09:32:26.062-0400 7f2835764700 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: 2021-10-07T09:32:26.062-0400 7f2835764700 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: 2021-10-07T09:32:26.062-0400 7f2835764700 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: 2021-10-07T09:32:26.066-0400 7f280dffb700 -1 auth: unable to find a keyring on /etc/ceph/.client.cloudstack.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/> Here is the log from the agent; https://gist.github.com/PeterS-gd/2d930b3317ea7b25ddf950e0e1cb167c Here is the log from the management server: https://gist.github.com/PeterS-gd/75c0f507c96803a053df37caf3b25181
