This turned out to be a storage issue, not sure why exactly, I'm using NFS and the share was mounted properly and was in RW mode, I did notice that there was some "junk" on it from previous install, once it was cleaned up and re-mounted everything started working OK, here is a sample of the error in agent.log
2013-02-09 20:37:53,186 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Exception com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: cannot create path '/mnt/78fa3e1e-b56c-3bd4-90d6-2c0fb33a52f4/v-2-VM-patchdisk': Operation not permitted at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createPhysicalDisk(LibvirtStorageAdaptor.java:556) at com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.createPhysicalDisk(LibvirtStoragePool.java:101) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createPatchVbd(LibvirtComputingResource.java:2980) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:2943) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2808) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1035) at com.cloud.agent.Agent.processRequest(Agent.java:518) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831) at com.cloud.utils.nio.Task.run(Task.java:83) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-02-09 20:37:53,188 WARN [cloud.agent.Agent] (agentRequest-Handler-1:null) Caught: -Oded -----Original Message----- From: Marcus Sorensen [mailto:shadow...@gmail.com] Sent: Thursday, February 14, 2013 6:59 PM To: aemne...@gmail.com Cc: Cloudstack users; Edison Su Subject: Re: ACS 4.0.1 - errors creating system VMs - KVM on Ubuntu 12.04 Unfortunately that log only tells us that the consoleproxy failed to start. It could even have started, but the agent was unable to ssh into it for some reason (didn't boot, no operating system, patchdisk didn't work, network config). The best way to troubleshoot this is to first do a 'virsh list' to see if the consoleproxy VM starts, then to use a VNC client to connect to it and see what state it is in. They can use 'virsh vncdisplay v-1-VM' for example to see what VNC port to connect to. Is there a bug for this already, where I can correspond with the user? Or are we still at the stage of determining whether it's a config issue? On Thu, Feb 14, 2013 at 10:31 AM, Ahmad Emneina <aemne...@gmail.com> wrote: > that looks pretty nasty, I'm going to cc Marcus and Edison, not sure > if they explicitly follow the user mailing list. They seem to be the > most active in terms of kvm activity. > > > On Thu, Feb 14, 2013 at 7:35 AM, Oded Nahum <oded.na...@citrix.com> wrote: >> >> Hi >> >> I'm getting the error below when CS 4.0.1 is trying to create the >> System VMs on Ubuntu 12.04 KVM host >> >> ved: false,moveToReserveredfalse >> 2013-02-14 16:29:57,548 WARN >> [cloud.consoleproxy.ConsoleProxyManagerImpl] >> (consoleproxy-1:null) Exception while trying to start console proxy >> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is >> unreachable: Host 1: Unable to start instance due to Unable to get >> answer that is of class com.cloud.agent.api.StartAnswer >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:847) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465) >> at >> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:627) >> at >> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:1164) >> at >> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1981) >> at >> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:173) >> at >> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:113) >> at >> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34) >> at >> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83) >> at >> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) >> at >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:679) >> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to >> get answer that is of class com.cloud.agent.api.StartAnswer >> at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80) >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMan >> agerImpl.java:783) >> >> no errors were seen during Zone creation, the UI shows the host is up >> and running. >> >> Any ideas ? >> >> -Oded > >