The error you are seeing in you agent.log seems for me like an "old friend". Take a look into this: https://github.com/apache/cloudstack/issues/5491#issue-1003559121
You can check it quiet fast. Connect to one of the hosts and try to mount the nfs-share manually to an temp-folder. suggest to use the -v option of mount. if it fails you could try to pass the option manually. Should be all in the linked issue above Am Do., 7. Okt. 2021 um 22:33 Uhr schrieb Peter Stine < petrus.st...@gmail.com>: > It appears that it has the capability to use nfs4 (It's using Ceph > Octopus), but it is not clear whether it is using nfs3 or nfs4. I *think* > it is using nfs4 > > On 2021/10/07 16:14:36, "vas...@gmx.de" <vas...@gmx.de> wrote: > > reading the agent.log you have trouble mounting the storage properly. > > Does your nfs-server for providing secondary storage provide nfs4 ? > > > > > > > > Am Do., 7. Okt. 2021 um 15:33 Uhr schrieb Peter Stine < > > petrus.st...@gmail.com>: > > > > > Hey everyone, > > > > > > I had to rework my network and reinstall cloudstack. While I am getting > > > the hosts to appear, they will not start any system VMs. They keep > failing > > > and iterating. It looks like there is an error in libvirtd (I am also > using > > > systemd for my networking), but I can't quite seem to figure out what > is > > > causing this issue. > > > I am using KVM as my hypervisor base and am running everything on > Ubuntu > > > 20.04. (The controller is in a VM if that makes a difference.) > > > > > > The main part that I have not seen in my error before is this: > 2021-10-07 > > > 08:59:37,558 WARN [kvm.resource.LibvirtConnection] > > > (agentRequest-Handler-4:null) (logid:e381c0f1) Can not find a > connection > > > for Instance s-1539-VM. Assuming the default connection. > > > 2021-10-07 08:59:37,727 WARN [kvm.resource.LibvirtKvmAgentHook] > > > (agentRequest-Handler-4:null) (logid:e381c0f1) Groovy script > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not > > > available. Transformations will not be applied. > > > 2021-10-07 08:59:37,727 WARN [kvm.resource.LibvirtKvmAgentHook] > > > (agentRequest-Handler-4:null) (logid:e381c0f1) Groovy scripting engine > is > > > not initialized. Data transformation skipped. > > > > > > > > > It also appears that libvirt is trying to connect to my primary storage > > > (which is Ceph), but cannot access the cloudstack key. I haven't seen > this > > > error before and cannot find a solution for it. Storage is showing up > in > > > the cloudstack management interface. > > > > > > Oct 07 09:31:25 gd-cs01-clst01-compute01 libvirtd[47830]: > > > 2021-10-07T09:31:25.780-0400 7f2836766700 -1 auth: unable to find a > keyring > > > on > > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> > > > Oct 07 09:31:25 gd-cs01-clst01-compute01 libvirtd[47830]: > > > 2021-10-07T09:31:25.780-0400 7f2836766700 -1 auth: unable to find a > keyring > > > on > > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> > > > Oct 07 09:31:31 gd-cs01-clst01-compute01 libvirtd[47830]: Failed to > > > connect socket to '/var/run/libvirt/virtlxcd-sock': No such file or > > > directory > > > Oct 07 09:31:31 gd-cs01-clst01-compute01 libvirtd[47830]: End of file > > > while reading data: Input/output error > > > Oct 07 09:31:33 gd-cs01-clst01-compute01 libvirtd[47830]: Failed to > > > connect socket to '/var/run/libvirt/virtlxcd-sock': No such file or > > > directory > > > Oct 07 09:31:33 gd-cs01-clst01-compute01 libvirtd[47830]: End of file > > > while reading data: Input/output error > > > Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: > > > 2021-10-07T09:32:26.062-0400 7f2835764700 -1 auth: unable to find a > keyring > > > on > > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> > > > Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: > > > 2021-10-07T09:32:26.062-0400 7f2835764700 -1 auth: unable to find a > keyring > > > on > > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> > > > Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: > > > 2021-10-07T09:32:26.062-0400 7f2835764700 -1 auth: unable to find a > keyring > > > on > > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.> > > > Oct 07 09:32:26 gd-cs01-clst01-compute01 libvirtd[47830]: > > > 2021-10-07T09:32:26.066-0400 7f280dffb700 -1 auth: unable to find a > keyring > > > on > > > > /etc/ceph/.client.cloudstack.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/> > > > > > > > > > Here is the log from the agent; > > > https://gist.github.com/PeterS-gd/2d930b3317ea7b25ddf950e0e1cb167c > > > > > > Here is the log from the management server: > > > https://gist.github.com/PeterS-gd/75c0f507c96803a053df37caf3b25181 > > > > > >
