Re: 4.15.1 Snapshot UEFI Legacy: Unable to start VM due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type
Hi Slavka, This ISO is just the iPXE template because we deploy our VM automatically. Since we deploy Windows and Linux i have to upload it twice? Once categorized with Linux, the other time with Windows? But since i want to have the same behaviour as with our old cloudstack, which is /dev/sdX naming for the root device, Windows is the way to go, correct? I will try it out. Thx for your help! But cloudstack has to be changed in order to get a vm created with UEFI boot and an other OS than the named ones. At least an error must be shown when using UEFI boot with a template categorized Other and deployed on KVM. Gruß Udo Müller > Am 01.06.2021 um 19:01 schrieb Slavka Peleva : > > Hi Udo, > > I saw where is the problem, the OS type of your ISO is "Other (32 bit)" on > which CloudStack sets "ide" for bus. > 'sata' is set on Windows OS types; > 'virtio' on these - Ubuntu, Fedora ,CentOS,Red Hat Enterprise Linux,Debian > GNU/Linux,FreeBSD,Oracle,Other PV; > 'scsi' on "Other PV Virtio-SCSI" OS types; > the rest gets 'ide' for bus > > Regards, > Slavka > >> On Tue, Jun 1, 2021 at 6:24 PM Udo Müller wrote: >> >> Hi Slavka, >> >> Deployment is done via WebUI. In this case i choose the target host >> (since only this one has debug enabled), then choose service, disk and >> network offering, activate advanced boot options and check UEFI Legacy. >> >> After setting the name the vm is started. >> >> The XML of the VM is attached. >> >> There you see >> >> >> >> But the VM will work with sata instead of ide. I tested it by defining a >> VM via virt-manager and choosing sata for the bus type. >> >> See also: >> >> https://bugs.launchpad.net/nova/+bug/1831538 >> >> Regards Udo >> >> >>> Am 01.06.21 um 16:16 schrieb Slavka Peleva: >>> Hi Udo, >>> >>> Could you please share more information about the deployment of a VM? >>> Are you creating a new instance from ISO or a template? Also, can you set >>> the DEBUG level at the agent, and is it possible to share the XML that >>> CloudStack sends to libvirt (it's in the >>> /var/log/cloudstack/agent/agent.log)? >>> >>> One more question did you update and the management with the 4.15.1.0 >>> version? >>> >>> Kind regards, >>> Slavka >>> >>>> On Tue, Jun 1, 2021 at 4:33 PM Udo Müller wrote: >>> >>>> Hi guys, >>>> >>>> >>>> I played around with cloudstack 4.15 and UEFI boot until someone here >>>> pointed me to a PR which fixes the problem with the missing q35 chipset. >>>> >>>> Thats the reason why I just installed the 4.15.1 SNAPSHOT. >>>> >>>> Now I get the following problem: >>>> >>>> Unable to start VM on Host... due to Unsupported Configuration: IDE >>>> controllers >>>> are unsupported for this QEMU binary or machine type >>>> >>>> I am using cloudstack on CentOS 7.9 with >>>> >>>> # rpm -qa | egrep "(qemu|libvirtd|OVMF|cloudstack)" >>>> *cloudstack*-agent-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 >>>> *qemu*-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 >>>> ipxe-roms-*qemu*-20180825-3.git133f4c.el7.noarch >>>> libvirt-daemon-driver-*qemu*-4.5.0-36.el7.x86_64 >>>> *OVMF*-20180508-6.gitee3198e672e2.el7.noarch >>>> *qemu*-img-ev-2.12.0-44.1.el7_8.1.x86_64 >>>> *cloudstack*-common-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 >>>> *qemu*-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64 >>>> >>>> 1) Is the PR regarding the q35 chipset already included in the SNAPSHOT? >>>> I guess yes, because with 4.15.0 and i440 chipset the VM could be >>>> created and was running. >>>> >>>> 2) What do I have to change inside my settings to switch from IDE to >>>> SCSI or SATA? >>>> >>>> Regards Udo >>>> >>
Re: 4.15.1 Snapshot UEFI Legacy: Unable to start VM due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type
Hi Udo, I saw where is the problem, the OS type of your ISO is "Other (32 bit)" on which CloudStack sets "ide" for bus. 'sata' is set on Windows OS types; 'virtio' on these - Ubuntu, Fedora ,CentOS,Red Hat Enterprise Linux,Debian GNU/Linux,FreeBSD,Oracle,Other PV; 'scsi' on "Other PV Virtio-SCSI" OS types; the rest gets 'ide' for bus Regards, Slavka On Tue, Jun 1, 2021 at 6:24 PM Udo Müller wrote: > Hi Slavka, > > Deployment is done via WebUI. In this case i choose the target host > (since only this one has debug enabled), then choose service, disk and > network offering, activate advanced boot options and check UEFI Legacy. > > After setting the name the vm is started. > > The XML of the VM is attached. > > There you see > > > > But the VM will work with sata instead of ide. I tested it by defining a > VM via virt-manager and choosing sata for the bus type. > > See also: > > https://bugs.launchpad.net/nova/+bug/1831538 > > Regards Udo > > > Am 01.06.21 um 16:16 schrieb Slavka Peleva: > > Hi Udo, > > > > Could you please share more information about the deployment of a VM? > > Are you creating a new instance from ISO or a template? Also, can you set > > the DEBUG level at the agent, and is it possible to share the XML that > > CloudStack sends to libvirt (it's in the > > /var/log/cloudstack/agent/agent.log)? > > > > One more question did you update and the management with the 4.15.1.0 > > version? > > > > Kind regards, > > Slavka > > > > On Tue, Jun 1, 2021 at 4:33 PM Udo Müller wrote: > > > >> Hi guys, > >> > >> > >> I played around with cloudstack 4.15 and UEFI boot until someone here > >> pointed me to a PR which fixes the problem with the missing q35 chipset. > >> > >> Thats the reason why I just installed the 4.15.1 SNAPSHOT. > >> > >> Now I get the following problem: > >> > >> Unable to start VM on Host... due to Unsupported Configuration: IDE > >> controllers > >> are unsupported for this QEMU binary or machine type > >> > >> I am using cloudstack on CentOS 7.9 with > >> > >> # rpm -qa | egrep "(qemu|libvirtd|OVMF|cloudstack)" > >> *cloudstack*-agent-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 > >> *qemu*-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 > >> ipxe-roms-*qemu*-20180825-3.git133f4c.el7.noarch > >> libvirt-daemon-driver-*qemu*-4.5.0-36.el7.x86_64 > >> *OVMF*-20180508-6.gitee3198e672e2.el7.noarch > >> *qemu*-img-ev-2.12.0-44.1.el7_8.1.x86_64 > >> *cloudstack*-common-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 > >> *qemu*-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64 > >> > >> 1) Is the PR regarding the q35 chipset already included in the SNAPSHOT? > >> I guess yes, because with 4.15.0 and i440 chipset the VM could be > >> created and was running. > >> > >> 2) What do I have to change inside my settings to switch from IDE to > >> SCSI or SATA? > >> > >> Regards Udo > >> >
Re: 4.15.1 Snapshot UEFI Legacy: Unable to start VM due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type
Hi Slavka, Deployment is done via WebUI. In this case i choose the target host (since only this one has debug enabled), then choose service, disk and network offering, activate advanced boot options and check UEFI Legacy. After setting the name the vm is started. The XML of the VM is attached. There you see But the VM will work with sata instead of ide. I tested it by defining a VM via virt-manager and choosing sata for the bus type. See also: https://bugs.launchpad.net/nova/+bug/1831538 Regards Udo Am 01.06.21 um 16:16 schrieb Slavka Peleva: Hi Udo, Could you please share more information about the deployment of a VM? Are you creating a new instance from ISO or a template? Also, can you set the DEBUG level at the agent, and is it possible to share the XML that CloudStack sends to libvirt (it's in the /var/log/cloudstack/agent/agent.log)? One more question did you update and the management with the 4.15.1.0 version? Kind regards, Slavka On Tue, Jun 1, 2021 at 4:33 PM Udo Müller wrote: Hi guys, I played around with cloudstack 4.15 and UEFI boot until someone here pointed me to a PR which fixes the problem with the missing q35 chipset. Thats the reason why I just installed the 4.15.1 SNAPSHOT. Now I get the following problem: Unable to start VM on Host... due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type I am using cloudstack on CentOS 7.9 with # rpm -qa | egrep "(qemu|libvirtd|OVMF|cloudstack)" *cloudstack*-agent-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 *qemu*-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 ipxe-roms-*qemu*-20180825-3.git133f4c.el7.noarch libvirt-daemon-driver-*qemu*-4.5.0-36.el7.x86_64 *OVMF*-20180508-6.gitee3198e672e2.el7.noarch *qemu*-img-ev-2.12.0-44.1.el7_8.1.x86_64 *cloudstack*-common-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 *qemu*-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64 1) Is the PR regarding the q35 chipset already included in the SNAPSHOT? I guess yes, because with 4.15.0 and i440 chipset the VM could be created and was running. 2) What do I have to change inside my settings to switch from IDE to SCSI or SATA? Regards Udo [175/1177] i-4-10-VM 95a92efb-d5c5-4b27-ac52-8c9cb7bdba3e Other (32-bit) Apache Software Foundation CloudStack KVM Hypervisor 95a92efb-d5c5-4b27-ac52-8c9cb7bdba3e hvm /usr/share/OVMF/OVMF_CODE.secboot.fd /var/lib/libvirt/qemu/nvram/95a92efb-d5c5-4b27-ac52-8c9cb7bdba3e.fd restart destroy destroy 524288 1 [148/1177] 500 /usr/libexec/qemu-kvm dd1a64aecc3c455eac6f
Re: 4.15.1 Snapshot UEFI Legacy: Unable to start VM due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type
Hi Udo, Could you please share more information about the deployment of a VM? Are you creating a new instance from ISO or a template? Also, can you set the DEBUG level at the agent, and is it possible to share the XML that CloudStack sends to libvirt (it's in the /var/log/cloudstack/agent/agent.log)? One more question did you update and the management with the 4.15.1.0 version? Kind regards, Slavka On Tue, Jun 1, 2021 at 4:33 PM Udo Müller wrote: > Hi guys, > > > I played around with cloudstack 4.15 and UEFI boot until someone here > pointed me to a PR which fixes the problem with the missing q35 chipset. > > Thats the reason why I just installed the 4.15.1 SNAPSHOT. > > Now I get the following problem: > > Unable to start VM on Host... due to Unsupported Configuration: IDE > controllers > are unsupported for this QEMU binary or machine type > > I am using cloudstack on CentOS 7.9 with > > # rpm -qa | egrep "(qemu|libvirtd|OVMF|cloudstack)" > *cloudstack*-agent-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 > *qemu*-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 > ipxe-roms-*qemu*-20180825-3.git133f4c.el7.noarch > libvirt-daemon-driver-*qemu*-4.5.0-36.el7.x86_64 > *OVMF*-20180508-6.gitee3198e672e2.el7.noarch > *qemu*-img-ev-2.12.0-44.1.el7_8.1.x86_64 > *cloudstack*-common-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 > *qemu*-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64 > > 1) Is the PR regarding the q35 chipset already included in the SNAPSHOT? > I guess yes, because with 4.15.0 and i440 chipset the VM could be > created and was running. > > 2) What do I have to change inside my settings to switch from IDE to > SCSI or SATA? > > Regards Udo >
4.15.1 Snapshot UEFI Legacy: Unable to start VM due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type
Hi guys, I played around with cloudstack 4.15 and UEFI boot until someone here pointed me to a PR which fixes the problem with the missing q35 chipset. Thats the reason why I just installed the 4.15.1 SNAPSHOT. Now I get the following problem: Unable to start VM on Host... due to Unsupported Configuration: IDE controllers are unsupported for this QEMU binary or machine type I am using cloudstack on CentOS 7.9 with # rpm -qa | egrep "(qemu|libvirtd|OVMF|cloudstack)" *cloudstack*-agent-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 *qemu*-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 ipxe-roms-*qemu*-20180825-3.git133f4c.el7.noarch libvirt-daemon-driver-*qemu*-4.5.0-36.el7.x86_64 *OVMF*-20180508-6.gitee3198e672e2.el7.noarch *qemu*-img-ev-2.12.0-44.1.el7_8.1.x86_64 *cloudstack*-common-4.15.1.0-SNAPSHOT.20210601.el7.x86_64 *qemu*-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64 1) Is the PR regarding the q35 chipset already included in the SNAPSHOT? I guess yes, because with 4.15.0 and i440 chipset the VM could be created and was running. 2) What do I have to change inside my settings to switch from IDE to SCSI or SATA? Regards Udo
Re: Unable to start VM due to insufficient capacity
Hi Fariborz, I didn't have chance to read earlier and now the link is expired. Unfortunately the log lines pasted are not helpful, can you please upload the logs somewhere else or paste the error stack trace? Regards, Nicolas Vazquez De: Fariborz Navidan Enviado: domingo, 9 de mayo de 2021 16:17 Para: users@cloudstack.apache.org Asunto: Unable to start VM due to insufficient capacity Hello, I just stopped two VMs residing on one of primary storages and they just failed to start. VMs residing on a different primary storage are starting well. The NFS storage is mounted on the host (1 host available) and has access to it. The volume is ready and the file exists on the storage. However, the following line is read in the management-server.log: 2021-05-09 19:02:34,085 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-1:ctx-e51b68a0 job-44995/job-44996 ctx-3c928305) (logid:ee9f1fc5) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage()] Full log file can be accessed at https://ctxt.io/2/AACgneNCEg It expires within 1 hour. Please help[ me. It is a critical issue for us! Thanks
Unable to start VM due to insufficient capacity
Hello, I just stopped two VMs residing on one of primary storages and they just failed to start. VMs residing on a different primary storage are starting well. The NFS storage is mounted on the host (1 host available) and has access to it. The volume is ready and the file exists on the storage. However, the following line is read in the management-server.log: 2021-05-09 19:02:34,085 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-1:ctx-e51b68a0 job-44995/job-44996 ctx-3c928305) (logid:ee9f1fc5) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage()] Full log file can be accessed at https://ctxt.io/2/AACgneNCEg It expires within 1 hour. Please help[ me. It is a critical issue for us! Thanks
Re: Unable to start VM
Unable to start VM on Host[-1-Routing] due to internal error: guest failed to start: cannot find init path '/sbin/init' relative to container root: No such file or directory You can google a bit - seems related to LXC stuff On Mon, 17 Jun 2019 at 16:23, Alejandro Ruiz Bermejo < arbermejo0...@gmail.com> wrote: > Hi, > Well this is awkward a few days ago i managed, with help from here, to > create an LXC cluster and launch an LXC instance. Then suddenly now i can't > start any VM inside the cluster and i think is due to a network error. I > checked the instance logs and the creation of it was successful but it > crash when trying to start it. Here it the logs: > https://pastebin.com/t1A4Z6TM > > This other log can also be helpful with the error > https://pastebin.com/zgRtWSg0 > -- Andrija Panić
Unable to start VM
Hi, Well this is awkward a few days ago i managed, with help from here, to create an LXC cluster and launch an LXC instance. Then suddenly now i can't start any VM inside the cluster and i think is due to a network error. I checked the instance logs and the creation of it was successful but it crash when trying to start it. Here it the logs: https://pastebin.com/t1A4Z6TM This other log can also be helpful with the error https://pastebin.com/zgRtWSg0
Re: Troubleshooting the Exception "Unable to start VM instance"
Hi Peter, +1 to Paul's comment. We need full logs to get an accurate assessment. Also, while you are at it could you check if your hosts and agents are up and running in infra tab? Regards, Anurag On Mon, Apr 29, 2019 at 6:58 PM Paul Angus wrote: > Hi Peter, > > We'd be able to help best if you post a whole unedited chunk of the > management log from the time you attempt to start this VM. You need to use > something like pastebin.com as you can't add attachments on the mailing > lists. > > paul.an...@shapeblue.com > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > > > -Original Message- > From: peter.murysh...@zv.fraunhofer.de > Sent: 29 April 2019 14:00 > To: users@cloudstack.apache.org > Subject: AW: Troubleshooting the Exception "Unable to start VM instance" > > Hi, Rakesh, > > thanks! > > After further digging the logs so far I can find: > > Destroying vm VM[User|i-291-77209-VM] as it failed to create on Host with > Id:null VM state transitted from :Stopped to Error with event: > OperationFailedToErrorvm's original host id: null new host id: null host id > before state transition: null > > Additionally, via the log ID I've found an error which contains some > base64 encoding but does not decode well i.e. base64 exits with an error > and outputs something lengthy (2972 chars) looking like a Java stack trace > but with random binary characters. > > Error: [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-92:ctx-0c7bb167 job-1027206/job-1027210 ctx-74dde55e) > (logid:2477aa66) Complete async job-1027210, jobStatus: FAILED, resultCode: > 0, result: rO0ABXNyAC9jb20u. > > What is readable in this "base64" output is (special characters replaced > with hyphen or new line): > > com.cloud.utils.exception.CloudRuntimeExceptionVMOI > csErrorCodexr-java.lang.RuntimeException-detailMessaget > Ljava/lang/String;[ > xr-java.lang.Throwable-Lcauset-Ljava/lang/Throwable;L > stackTracet-[Ljava/lang/StackTraceElement; > L-suppressedExceptionst-Ljava/util/Lis > > kind regards > Peter > > > ________ > Von: Rakesh Venkatesh [www.rakeshv@gmail.com] > Gesendet: Montag, 29. April 2019 13:17 > An: users@cloudstack.apache.org > Betreff: Re: Troubleshooting the Exception "Unable to start VM instance" > > Hello Peter > > If you dig into logs a little bit more, you can see the actual reason for > the error. The usual errors are insufficient server capacity exception like > not enough cpu, memory, ip address or mismatch between template tag and > host tag or some other mismatch in the tags. These are the common errors I > have seen. > Search for "startVirtualMachine" api and trace its jobid which is usually > like "context-" and look at the logs > > On Mon, Apr 29, 2019 at 12:58 PM wrote: > > > Hi, all, > > > > sometimes the system can't start a VM instance (a 530 error). > > In the logs there is the following trace - how to troubleshoot it and > > understand which circumstances lead to it? > > > > > > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM > > instance > > at > > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4889) > > at sun.reflect.GeneratedMethodAccessor433.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > > at > > > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5047) > > at > > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > > at > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:615) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContex
RE: Troubleshooting the Exception "Unable to start VM instance"
Hi Peter, We'd be able to help best if you post a whole unedited chunk of the management log from the time you attempt to start this VM. You need to use something like pastebin.com as you can't add attachments on the mailing lists. paul.an...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue -Original Message- From: peter.murysh...@zv.fraunhofer.de Sent: 29 April 2019 14:00 To: users@cloudstack.apache.org Subject: AW: Troubleshooting the Exception "Unable to start VM instance" Hi, Rakesh, thanks! After further digging the logs so far I can find: Destroying vm VM[User|i-291-77209-VM] as it failed to create on Host with Id:null VM state transitted from :Stopped to Error with event: OperationFailedToErrorvm's original host id: null new host id: null host id before state transition: null Additionally, via the log ID I've found an error which contains some base64 encoding but does not decode well i.e. base64 exits with an error and outputs something lengthy (2972 chars) looking like a Java stack trace but with random binary characters. Error: [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-92:ctx-0c7bb167 job-1027206/job-1027210 ctx-74dde55e) (logid:2477aa66) Complete async job-1027210, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20u. What is readable in this "base64" output is (special characters replaced with hyphen or new line): com.cloud.utils.exception.CloudRuntimeExceptionVMOI csErrorCodexr-java.lang.RuntimeException-detailMessaget Ljava/lang/String;[ xr-java.lang.Throwable-Lcauset-Ljava/lang/Throwable;L stackTracet-[Ljava/lang/StackTraceElement; L-suppressedExceptionst-Ljava/util/Lis kind regards Peter Von: Rakesh Venkatesh [www.rakeshv@gmail.com] Gesendet: Montag, 29. April 2019 13:17 An: users@cloudstack.apache.org Betreff: Re: Troubleshooting the Exception "Unable to start VM instance" Hello Peter If you dig into logs a little bit more, you can see the actual reason for the error. The usual errors are insufficient server capacity exception like not enough cpu, memory, ip address or mismatch between template tag and host tag or some other mismatch in the tags. These are the common errors I have seen. Search for "startVirtualMachine" api and trace its jobid which is usually like "context-" and look at the logs On Mon, Apr 29, 2019 at 12:58 PM wrote: > Hi, all, > > sometimes the system can't start a VM instance (a 530 error). > In the logs there is the following trace - how to troubleshoot it and > understand which circumstances lead to it? > > > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM > instance > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4889) > at sun.reflect.GeneratedMethodAccessor433.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5047) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:615) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:563) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > kind regards > Peter > > -- Thanks and regards Rakesh venkatesh
AW: Troubleshooting the Exception "Unable to start VM instance"
Hi, Rakesh, thanks! After further digging the logs so far I can find: Destroying vm VM[User|i-291-77209-VM] as it failed to create on Host with Id:null VM state transitted from :Stopped to Error with event: OperationFailedToErrorvm's original host id: null new host id: null host id before state transition: null Additionally, via the log ID I've found an error which contains some base64 encoding but does not decode well i.e. base64 exits with an error and outputs something lengthy (2972 chars) looking like a Java stack trace but with random binary characters. Error: [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-92:ctx-0c7bb167 job-1027206/job-1027210 ctx-74dde55e) (logid:2477aa66) Complete async job-1027210, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20u. What is readable in this "base64" output is (special characters replaced with hyphen or new line): com.cloud.utils.exception.CloudRuntimeExceptionVMOI csErrorCodexr-java.lang.RuntimeException-detailMessaget Ljava/lang/String;[ xr-java.lang.Throwable-Lcauset-Ljava/lang/Throwable;L stackTracet-[Ljava/lang/StackTraceElement; L-suppressedExceptionst-Ljava/util/Lis kind regards Peter Von: Rakesh Venkatesh [www.rakeshv@gmail.com] Gesendet: Montag, 29. April 2019 13:17 An: users@cloudstack.apache.org Betreff: Re: Troubleshooting the Exception "Unable to start VM instance" Hello Peter If you dig into logs a little bit more, you can see the actual reason for the error. The usual errors are insufficient server capacity exception like not enough cpu, memory, ip address or mismatch between template tag and host tag or some other mismatch in the tags. These are the common errors I have seen. Search for "startVirtualMachine" api and trace its jobid which is usually like "context-" and look at the logs On Mon, Apr 29, 2019 at 12:58 PM wrote: > Hi, all, > > sometimes the system can't start a VM instance (a 530 error). > In the logs there is the following trace - how to troubleshoot it and > understand which circumstances lead to it? > > > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM > instance > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4889) > at sun.reflect.GeneratedMethodAccessor433.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5047) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:615) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:563) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > kind regards > Peter > > -- Thanks and regards Rakesh venkatesh
Re: Troubleshooting the Exception "Unable to start VM instance"
Hello Peter If you dig into logs a little bit more, you can see the actual reason for the error. The usual errors are insufficient server capacity exception like not enough cpu, memory, ip address or mismatch between template tag and host tag or some other mismatch in the tags. These are the common errors I have seen. Search for "startVirtualMachine" api and trace its jobid which is usually like "context-" and look at the logs On Mon, Apr 29, 2019 at 12:58 PM wrote: > Hi, all, > > sometimes the system can't start a VM instance (a 530 error). > In the logs there is the following trace - how to troubleshoot it and > understand which circumstances lead to it? > > > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM > instance > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4889) > at sun.reflect.GeneratedMethodAccessor433.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5047) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:615) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:563) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > kind regards > Peter > > -- Thanks and regards Rakesh venkatesh
Troubleshooting the Exception "Unable to start VM instance"
Hi, all, sometimes the system can't start a VM instance (a 530 error). In the logs there is the following trace - how to troubleshoot it and understand which circumstances lead to it? com.cloud.utils.exception.CloudRuntimeException: Unable to start VM instance at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4889) at sun.reflect.GeneratedMethodAccessor433.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5047) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:615) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:563) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) kind regards Peter
Re: Unable to Start VM SR failure
Hi Jeremy, You need to find the uuid of the vdi, I couldn't find it in the logs. The VDI Not Available error appears when a virtual machine (VM) is shutdown inappropriately or if there is a concern with the Storage Repository (SR) where the virtual machine exists There are a number of logs that differ slightly but all relate to the problem of the VDI locked. Steps to fix the issue : When a VM is started on a XenServer host, domain 0, needs to ensure that all resources required by the guest VM are available for it to start. If a VM has had an unexpected shutdown, power failure, etc., we could have a dirty flag on the VDI, which is normally a logical volume in a volume group (Storage Repository). To clear this flag, we need to do the following actions: 1. Via the command line, type xe vm-start vm= and you will get a better error message than noted in the XenCenter, namely the UUID of the VDI that cannot be used exclusively by the guest VM. 2. Make a note of the UUID of the VDI and proceed to remove it from the pool database via the following xe vdi-forget uuid=. You will then note that the VM no longer is attached to the VDI in XenCenter. 3. On the SR that the VDI is located on, proceed to the Storage tab and select Rescan. You should see the VDI with no name label on it. 4. To ensure we have the correct VDI, if there are many VDIs with no name, use the following command xe vdi-list uuid=. You should see it with no name label or description. We can then label this VDI via xe vdi-param-set uuid= name-label=. You should see the labeled VDI in the XenCenter. 5. On the VMs storage tab, select Attach and proceed to select the VDI on the SR. 6. VM should start without issue. Example: On Pool Master [root@tl01 ~]# xe vdi-list uuid=741ef409-2c15-43f3-a10e-50f1467ee668 uuid ( RO) : 741ef409-2c15-43f3-a10e-50f1467ee668 name-label ( RW): cloud-df8bdcdc-ff00-43cd-a6a4-e79ccd6cd8a1 name-description ( RW): sr-uuid ( RO): dc0e0a15-a520-9861-edb2-a808fb095df5 virtual-size ( RO): 10737418240 sharable ( RO): false read-only ( RO): false [root@tl01 ~]# xe vdi-forget uuid=741ef409-2c15-43f3-a10e-50f1467ee668 [root@tl01 ~]# xe sr-scan uuid=dc0e0a15-a520-9861-edb2-a808fb095df5 [root@tl01 ~]# xe vdi-param-set uuid=741ef409-2c15-43f3-a10e-50f1467ee668 name-label=cloud-df8bdcdc-ff00-43cd-a6a4-e79ccd6cd8a1 Regards, Adrian Sender -- Original Message --- From: Jeremy Peterson <jpeter...@acentek.net> To: "users@cloudstack.apache.org" <users@cloudstack.apache.org> Sent: Thu, 13 Jul 2017 21:26:21 +0000 Subject: Unable to Start VM SR failure > Ok so I am back at the users again. > > I had 1 VM lock up today and was forced to issue a restart as the OS > was non responsive. > > After powering off the VM bounced around between hosts and > ultimately stopped and failed with an insufficient resources error. > > I looked around in XenServer and found the ROOT volume on one of the > SCSI LUNS's 100GB great. > > I tried to download the volume in CS failed. I tried to create > template from volume failed. > > Went to start the Volume again and this is the error log. > > https://pastebin.com/VAR4KRrL > > What is jumping out at me is > > errorInfo: [SR_BACKEND_FAILURE, non-zero exit, , Traceback (most > recent call last): > > After that error I get a ton of /opt/xensource/sm failure messages. > > 1. 2017-07-13 16:01:59,983 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-380:ctx-0f865755) Unable to clean up VBD due to > 2. You gave an invalid object reference. The object may have > recently been deleted. The class parameter gives the type of > reference given, and the handle parameter echoes the bad value given. > > http://prntscr.com/fvdv41 > > You can see ROOT-277 is on my storage. > > http://prntscr.com/fvdvi1 > > The "dev" server has the Volumes of ROOT-277 > > Ideas ? > > Jeremy --- End of Original Message ---
Unable to Start VM SR failure
Ok so I am back at the users again. I had 1 VM lock up today and was forced to issue a restart as the OS was non responsive. After powering off the VM bounced around between hosts and ultimately stopped and failed with an insufficient resources error. I looked around in XenServer and found the ROOT volume on one of the SCSI LUNS's 100GB great. I tried to download the volume in CS failed. I tried to create template from volume failed. Went to start the Volume again and this is the error log. https://pastebin.com/VAR4KRrL What is jumping out at me is errorInfo: [SR_BACKEND_FAILURE, non-zero exit, , Traceback (most recent call last): After that error I get a ton of /opt/xensource/sm failure messages. 1. 2017-07-13 16:01:59,983 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-380:ctx-0f865755) Unable to clean up VBD due to 2. You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. http://prntscr.com/fvdv41 You can see ROOT-277 is on my storage. http://prntscr.com/fvdvi1 The "dev" server has the Volumes of ROOT-277 Ideas ? Jeremy
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
On Fri, Sep 18, 2015 at 12:20 PM, Anshul Gangwar
<anshul.gang...@citrix.com> wrote:
> From where you are using Hyper-V Agent?
>
This is the Citrix CloudPlatform hyperv agent 4.5.0.0-1
> Regards,
> Anshul
>
> On 18-Sep-2015, at 2:40 PM, Traiano Welcome
> <trai...@gmail.com<mailto:trai...@gmail.com>> wrote:
>
> Hi Anshul
>
>
> On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar
> <anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com>> wrote:
> Attach some Hyper-V agent log. They will give better idea on what is
> happening on host.
>
>
> Thanks for the feedback. Attached is the paste of the windows event
> manager error we're seeing that relates to the agent.
> I'm not sure if this is the correct log but I can't find any other
> logs written by the hyper-v agent (is there a way to set this up?)
>
> On the controller side, I see an agent directory, but it is empty ...
> How to enable agent logging on the controller side?
>
>
> Regards,
> Anshul
>
> On 17-Sep-2015, at 3:27 PM, Traiano Welcome
> <trai...@gmail.com<mailto:trai...@gmail.com>> wrote:
>
> Hi All
>
> I am trying to configure a zone based on a hyper-v cluster (ms windows
> 2012 hyper-v cluster of two nodes, using SMBv3).
>
> The zone configuration completes successfully, including adding
> primary and secondary SMBv3 storage and adding hyper-v cluster node.
> However, I see that CloudStack is continuously building the system
> VMs, and destroying them in an endless loop, without ever running them
> successfully
>
> What I'm seeing in the hyper-v management console:
>
> -The Cloudstack controller is successfully is launching and
> destroying the system vms:
> - System VMs are created and destroyed continuously (Console Proxy nd
> Secondary Storage VM) (confirmed by the cloudstack manager logs)
> - System VMs appear as stopped in the CloudStack web interface, and
> periodically vanish.
>
> Looking through the logs the VM build process fails with this message:
>
> ---
> .
> .
>
> 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230)
> (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to
> com.cloud.agent.api.StartCommand fail on exceptionObject reference not
> set to an instance of an object.
> .
> .
> .
> ---
>
> Details of my setup:
>
> 1. Hyper-V Version: Hypervisor version 6.2
> 2. CloudStack version/distribution: 4.5
> 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual
> 10Gibps NIC (teamed)
> 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 node
> 5. Zone Network Details: Basic.
> 6. The CloudStack controller is successfully managing VMs running on
> XenServer 6.2.
>
> Attached is a log dump (addresses sanitized) of the whole vm build cycle.
>
> Has anyone seen this kind of issue before when integrating Hyper-V
> with CloudStack ?
> Is there a set of additional things I could try to troubleshoot this,
> or configurations I could try?
>
> Here are the basic troubleshooting tests I'v etried:
>
> - Seeding the hyper-v image to secondary storage: success
> - Mounting the secondary and primary storage from the Windows 2012
> storage node to the CloudStck controller VM: success
> - Browsing to the agent port via https from the controller to the
> Hyper-V nodes: success
> - Creating VMs manually on the Hyper-V nodes: success
>
> I'm not seeing any network communications, physical hardware or
> hyper-v clusters. I suppose that the fact system VMs are being
> created on Hyper-V means that essentially CloudStack is communicating
> with hyper-V., and controlling it correctly ... The only question is
> why the System VM deployment fails.
>
> I suspect physical network addressing configuration, but I'm not sure
> how to confirm this ...
>
> Many thanks in advance for any assistance!
> Traiano
>
>
> P.S: Some key debug messages in the logs (leading up to the system vms
> being destroyed):
>
>
> -- ---
>
> 2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
> (secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm
> created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null,
> private IP: null
> .
> .
> 2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq
> 83-346478: { Ans: , MgmtId: 117586639591181, via: 83, Ver: v1,
> Flags:100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
> }
> .
> .
> 2015-09-17 04:45:07,692 DEBUG [c.c.h
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
CloudStack doesn’t support basic networking for Hyper-V. You have to try with advanced networking without security group. Regards, Anshul On 18-Sep-2015, at 6:23 pm, Traiano Welcome <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: Hi Prapul On Fri, Sep 18, 2015 at 2:32 PM, prapul cool <prapul_cool2...@yahoo.com.invalid<mailto:prapul_cool2...@yahoo.com.invalid>> wrote: Hi Traiano, Curious to know which network your using? I selected "Basic" when creating the zone. Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.com<http://Psiclouds.com>. On Friday, 18 September 2015 2:56 PM, Traiano Welcome <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: On Fri, Sep 18, 2015 at 12:20 PM, Anshul Gangwar <anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com>> wrote: From where you are using Hyper-V Agent? This is the Citrix CloudPlatform hyperv agent 4.5.0.0-1 Regards, Anshul On 18-Sep-2015, at 2:40 PM, Traiano Welcome <trai...@gmail.com<mailto:trai...@gmail.com><mailto:trai...@gmail.com>> wrote: Hi Anshul On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar <anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com><mailto:anshul.gang...@citrix.com>> wrote: Attach some Hyper-V agent log. They will give better idea on what is happening on host. Thanks for the feedback. Attached is the paste of the windows event manager error we're seeing that relates to the agent. I'm not sure if this is the correct log but I can't find any other logs written by the hyper-v agent (is there a way to set this up?) On the controller side, I see an agent directory, but it is empty ... How to enable agent logging on the controller side? Regards, Anshul On 17-Sep-2015, at 3:27 PM, Traiano Welcome <trai...@gmail.com<mailto:trai...@gmail.com><mailto:trai...@gmail.com>> wrote: Hi All I am trying to configure a zone based on a hyper-v cluster (ms windows 2012 hyper-v cluster of two nodes, using SMBv3). The zone configuration completes successfully, including adding primary and secondary SMBv3 storage and adding hyper-v cluster node. However, I see that CloudStack is continuously building the system VMs, and destroying them in an endless loop, without ever running them successfully What I'm seeing in the hyper-v management console: -The Cloudstack controller is successfully is launching and destroying the system vms: - System VMs are created and destroyed continuously (Console Proxy nd Secondary Storage VM) (confirmed by the cloudstack manager logs) - System VMs appear as stopped in the CloudStack web interface, and periodically vanish. Looking through the logs the VM build process fails with this message: --- . . 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230) (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail on exceptionObject reference not set to an instance of an object. . . . --- Details of my setup: 1. Hyper-V Version: Hypervisor version 6.2 2. CloudStack version/distribution: 4.5 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual 10Gibps NIC (teamed) 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 node 5. Zone Network Details: Basic. 6. The CloudStack controller is successfully managing VMs running on XenServer 6.2. Attached is a log dump (addresses sanitized) of the whole vm build cycle. Has anyone seen this kind of issue before when integrating Hyper-V with CloudStack ? Is there a set of additional things I could try to troubleshoot this, or configurations I could try? Here are the basic troubleshooting tests I'v etried: - Seeding the hyper-v image to secondary storage: success - Mounting the secondary and primary storage from the Windows 2012 storage node to the CloudStck controller VM: success - Browsing to the agent port via https from the controller to the Hyper-V nodes: success - Creating VMs manually on the Hyper-V nodes: success I'm not seeing any network communications, physical hardware or hyper-v clusters. I suppose that the fact system VMs are being created on Hyper-V means that essentially CloudStack is communicating with hyper-V., and controlling it correctly ... The only question is why the System VM deployment fails. I suspect physical network addressing configuration, but I'm not sure how to confirm this ... Many thanks in advance for any assistance! Traiano P.S: Some key debug messages in the logs (leading up to the system vms being destroyed): -- --- 2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null, private IP: null . . 2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
Hi Prapul On Fri, Sep 18, 2015 at 2:32 PM, prapul cool <prapul_cool2...@yahoo.com.invalid> wrote: > Hi Traiano, > Curious to know which network your using? I selected "Basic" when creating the zone. > Regards, > > Prapul sriram, > > Cloud Architect, > > PSI Cloud Solutions, > > Psiclouds.com. > > > On Friday, 18 September 2015 2:56 PM, Traiano Welcome > <trai...@gmail.com> wrote: > > > On Fri, Sep 18, 2015 at 12:20 PM, Anshul Gangwar > <anshul.gang...@citrix.com> wrote: >> From where you are using Hyper-V Agent? >> > > > This is the Citrix CloudPlatform hyperv agent 4.5.0.0-1 > > > >> Regards, >> Anshul >> >> On 18-Sep-2015, at 2:40 PM, Traiano Welcome >> <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: >> >> Hi Anshul >> >> >> On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar >> <anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com>> wrote: >> Attach some Hyper-V agent log. They will give better idea on what is >> happening on host. >> >> >> Thanks for the feedback. Attached is the paste of the windows event >> manager error we're seeing that relates to the agent. >> I'm not sure if this is the correct log but I can't find any other >> logs written by the hyper-v agent (is there a way to set this up?) >> >> On the controller side, I see an agent directory, but it is empty ... >> How to enable agent logging on the controller side? >> >> >> Regards, >> Anshul >> >> On 17-Sep-2015, at 3:27 PM, Traiano Welcome >> <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: >> >> Hi All >> >> I am trying to configure a zone based on a hyper-v cluster (ms windows >> 2012 hyper-v cluster of two nodes, using SMBv3). >> >> The zone configuration completes successfully, including adding >> primary and secondary SMBv3 storage and adding hyper-v cluster node. >> However, I see that CloudStack is continuously building the system >> VMs, and destroying them in an endless loop, without ever running them >> successfully >> >> What I'm seeing in the hyper-v management console: >> >> -The Cloudstack controller is successfully is launching and >> destroying the system vms: >> - System VMs are created and destroyed continuously (Console Proxy nd >> Secondary Storage VM) (confirmed by the cloudstack manager logs) >> - System VMs appear as stopped in the CloudStack web interface, and >> periodically vanish. >> >> Looking through the logs the VM build process fails with this message: >> >> --- >> . >> . >> >> 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230) >> (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to >> com.cloud.agent.api.StartCommand fail on exceptionObject reference not >> set to an instance of an object. >> . >> . >> . >> --- >> >> Details of my setup: >> >> 1. Hyper-V Version: Hypervisor version 6.2 >> 2. CloudStack version/distribution: 4.5 >> 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual >> 10Gibps NIC (teamed) >> 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 >> node >> 5. Zone Network Details: Basic. >> 6. The CloudStack controller is successfully managing VMs running on >> XenServer 6.2. >> >> Attached is a log dump (addresses sanitized) of the whole vm build cycle. >> >> Has anyone seen this kind of issue before when integrating Hyper-V >> with CloudStack ? >> Is there a set of additional things I could try to troubleshoot this, >> or configurations I could try? >> >> Here are the basic troubleshooting tests I'v etried: >> >> - Seeding the hyper-v image to secondary storage: success >> - Mounting the secondary and primary storage from the Windows 2012 >> storage node to the CloudStck controller VM: success >> - Browsing to the agent port via https from the controller to the >> Hyper-V nodes: success >> - Creating VMs manually on the Hyper-V nodes: success >> >> I'm not seeing any network communications, physical hardware or >> hyper-v clusters. I suppose that the fact system VMs are being >> created on Hyper-V means that essentially CloudStack is communicating >> with hyper-V., and controlling it correctly ... The only question is >> why the System VM deployment fails.
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
Hi Traiano, Curious to know which network your using? Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.com. On Friday, 18 September 2015 2:56 PM, Traiano Welcome <trai...@gmail.com> wrote: On Fri, Sep 18, 2015 at 12:20 PM, Anshul Gangwar <anshul.gang...@citrix.com> wrote: > From where you are using Hyper-V Agent? > This is the Citrix CloudPlatform hyperv agent 4.5.0.0-1 > Regards, > Anshul > > On 18-Sep-2015, at 2:40 PM, Traiano Welcome > <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: > > Hi Anshul > > > On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar > <anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com>> wrote: > Attach some Hyper-V agent log. They will give better idea on what is > happening on host. > > > Thanks for the feedback. Attached is the paste of the windows event > manager error we're seeing that relates to the agent. > I'm not sure if this is the correct log but I can't find any other > logs written by the hyper-v agent (is there a way to set this up?) > > On the controller side, I see an agent directory, but it is empty ... > How to enable agent logging on the controller side? > > > Regards, > Anshul > > On 17-Sep-2015, at 3:27 PM, Traiano Welcome > <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: > > Hi All > > I am trying to configure a zone based on a hyper-v cluster (ms windows > 2012 hyper-v cluster of two nodes, using SMBv3). > > The zone configuration completes successfully, including adding > primary and secondary SMBv3 storage and adding hyper-v cluster node. > However, I see that CloudStack is continuously building the system > VMs, and destroying them in an endless loop, without ever running them > successfully > > What I'm seeing in the hyper-v management console: > > -The Cloudstack controller is successfully is launching and > destroying the system vms: > - System VMs are created and destroyed continuously (Console Proxy nd > Secondary Storage VM) (confirmed by the cloudstack manager logs) > - System VMs appear as stopped in the CloudStack web interface, and > periodically vanish. > > Looking through the logs the VM build process fails with this message: > > --- > . > . > > 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230) > (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to > com.cloud.agent.api.StartCommand fail on exceptionObject reference not > set to an instance of an object. > . > . > . > --- > > Details of my setup: > > 1. Hyper-V Version: Hypervisor version 6.2 > 2. CloudStack version/distribution: 4.5 > 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual > 10Gibps NIC (teamed) > 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 node > 5. Zone Network Details: Basic. > 6. The CloudStack controller is successfully managing VMs running on > XenServer 6.2. > > Attached is a log dump (addresses sanitized) of the whole vm build cycle. > > Has anyone seen this kind of issue before when integrating Hyper-V > with CloudStack ? > Is there a set of additional things I could try to troubleshoot this, > or configurations I could try? > > Here are the basic troubleshooting tests I'v etried: > > - Seeding the hyper-v image to secondary storage: success > - Mounting the secondary and primary storage from the Windows 2012 > storage node to the CloudStck controller VM: success > - Browsing to the agent port via https from the controller to the > Hyper-V nodes: success > - Creating VMs manually on the Hyper-V nodes: success > > I'm not seeing any network communications, physical hardware or > hyper-v clusters. I suppose that the fact system VMs are being > created on Hyper-V means that essentially CloudStack is communicating > with hyper-V., and controlling it correctly ... The only question is > why the System VM deployment fails. > > I suspect physical network addressing configuration, but I'm not sure > how to confirm this ... > > Many thanks in advance for any assistance! > Traiano > > > P.S: Some key debug messages in the logs (leading up to the system vms > being destroyed): > > > -- --- > > 2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] > (secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm > created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null, > private IP: null > . > . > 2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq > 83-346478:
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
Hi Anshul
On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar
<anshul.gang...@citrix.com> wrote:
> Attach some Hyper-V agent log. They will give better idea on what is
> happening on host.
>
Thanks for the feedback. Attached is the paste of the windows event
manager error we're seeing that relates to the agent.
I'm not sure if this is the correct log but I can't find any other
logs written by the hyper-v agent (is there a way to set this up?)
On the controller side, I see an agent directory, but it is empty ...
How to enable agent logging on the controller side?
> Regards,
> Anshul
>
>> On 17-Sep-2015, at 3:27 PM, Traiano Welcome <trai...@gmail.com> wrote:
>>
>> Hi All
>>
>> I am trying to configure a zone based on a hyper-v cluster (ms windows
>> 2012 hyper-v cluster of two nodes, using SMBv3).
>>
>> The zone configuration completes successfully, including adding
>> primary and secondary SMBv3 storage and adding hyper-v cluster node.
>> However, I see that CloudStack is continuously building the system
>> VMs, and destroying them in an endless loop, without ever running them
>> successfully
>>
>> What I'm seeing in the hyper-v management console:
>>
>> -The Cloudstack controller is successfully is launching and
>> destroying the system vms:
>> - System VMs are created and destroyed continuously (Console Proxy nd
>> Secondary Storage VM) (confirmed by the cloudstack manager logs)
>> - System VMs appear as stopped in the CloudStack web interface, and
>> periodically vanish.
>>
>> Looking through the logs the VM build process fails with this message:
>>
>> ---
>> .
>> .
>>
>> 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230)
>> (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to
>> com.cloud.agent.api.StartCommand fail on exceptionObject reference not
>> set to an instance of an object.
>> .
>> .
>> .
>> ---
>>
>> Details of my setup:
>>
>> 1. Hyper-V Version: Hypervisor version 6.2
>> 2. CloudStack version/distribution: 4.5
>> 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual
>> 10Gibps NIC (teamed)
>> 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012
>> node
>> 5. Zone Network Details: Basic.
>> 6. The CloudStack controller is successfully managing VMs running on
>> XenServer 6.2.
>>
>> Attached is a log dump (addresses sanitized) of the whole vm build cycle.
>>
>> Has anyone seen this kind of issue before when integrating Hyper-V
>> with CloudStack ?
>> Is there a set of additional things I could try to troubleshoot this,
>> or configurations I could try?
>>
>> Here are the basic troubleshooting tests I'v etried:
>>
>> - Seeding the hyper-v image to secondary storage: success
>> - Mounting the secondary and primary storage from the Windows 2012
>> storage node to the CloudStck controller VM: success
>> - Browsing to the agent port via https from the controller to the
>> Hyper-V nodes: success
>> - Creating VMs manually on the Hyper-V nodes: success
>>
>> I'm not seeing any network communications, physical hardware or
>> hyper-v clusters. I suppose that the fact system VMs are being
>> created on Hyper-V means that essentially CloudStack is communicating
>> with hyper-V., and controlling it correctly ... The only question is
>> why the System VM deployment fails.
>>
>> I suspect physical network addressing configuration, but I'm not sure
>> how to confirm this ...
>>
>> Many thanks in advance for any assistance!
>> Traiano
>>
>>
>> P.S: Some key debug messages in the logs (leading up to the system vms
>> being destroyed):
>>
>>
>> -- ---
>>
>> 2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
>> (secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm
>> created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null,
>> private IP: null
>> .
>> .
>> 2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq
>> 83-346478: { Ans: , MgmtId: 117586639591181, via: 83, Ver: v1,
>> Flags:100010,
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
>> }
>> .
>> .
>> 2015-09-17 04:45:07,692 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
>> (DirectAgent-293:ctx-de6632d6) (logid:138757fe) Sendin
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
>From where you are using Hyper-V Agent?
Regards,
Anshul
On 18-Sep-2015, at 2:40 PM, Traiano Welcome
<trai...@gmail.com<mailto:trai...@gmail.com>> wrote:
Hi Anshul
On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar
<anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com>> wrote:
Attach some Hyper-V agent log. They will give better idea on what is happening
on host.
Thanks for the feedback. Attached is the paste of the windows event
manager error we're seeing that relates to the agent.
I'm not sure if this is the correct log but I can't find any other
logs written by the hyper-v agent (is there a way to set this up?)
On the controller side, I see an agent directory, but it is empty ...
How to enable agent logging on the controller side?
Regards,
Anshul
On 17-Sep-2015, at 3:27 PM, Traiano Welcome
<trai...@gmail.com<mailto:trai...@gmail.com>> wrote:
Hi All
I am trying to configure a zone based on a hyper-v cluster (ms windows
2012 hyper-v cluster of two nodes, using SMBv3).
The zone configuration completes successfully, including adding
primary and secondary SMBv3 storage and adding hyper-v cluster node.
However, I see that CloudStack is continuously building the system
VMs, and destroying them in an endless loop, without ever running them
successfully
What I'm seeing in the hyper-v management console:
-The Cloudstack controller is successfully is launching and
destroying the system vms:
- System VMs are created and destroyed continuously (Console Proxy nd
Secondary Storage VM) (confirmed by the cloudstack manager logs)
- System VMs appear as stopped in the CloudStack web interface, and
periodically vanish.
Looking through the logs the VM build process fails with this message:
---
.
.
2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230)
(logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to
com.cloud.agent.api.StartCommand fail on exceptionObject reference not
set to an instance of an object.
.
.
.
---
Details of my setup:
1. Hyper-V Version: Hypervisor version 6.2
2. CloudStack version/distribution: 4.5
3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual
10Gibps NIC (teamed)
4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 node
5. Zone Network Details: Basic.
6. The CloudStack controller is successfully managing VMs running on
XenServer 6.2.
Attached is a log dump (addresses sanitized) of the whole vm build cycle.
Has anyone seen this kind of issue before when integrating Hyper-V
with CloudStack ?
Is there a set of additional things I could try to troubleshoot this,
or configurations I could try?
Here are the basic troubleshooting tests I'v etried:
- Seeding the hyper-v image to secondary storage: success
- Mounting the secondary and primary storage from the Windows 2012
storage node to the CloudStck controller VM: success
- Browsing to the agent port via https from the controller to the
Hyper-V nodes: success
- Creating VMs manually on the Hyper-V nodes: success
I'm not seeing any network communications, physical hardware or
hyper-v clusters. I suppose that the fact system VMs are being
created on Hyper-V means that essentially CloudStack is communicating
with hyper-V., and controlling it correctly ... The only question is
why the System VM deployment fails.
I suspect physical network addressing configuration, but I'm not sure
how to confirm this ...
Many thanks in advance for any assistance!
Traiano
P.S: Some key debug messages in the logs (leading up to the system vms
being destroyed):
-- ---
2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
(secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm
created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null,
private IP: null
.
.
2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq
83-346478: { Ans: , MgmtId: 117586639591181, via: 83, Ver: v1,
Flags:100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
}
.
.
2015-09-17 04:45:07,692 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-293:ctx-de6632d6) (logid:138757fe) Sending cmd to
https://xx.yy.zz.223:8250/api/HypervResource/com.cloud.agent.api.StartCommand
cmd
data:{"vm":{"id":2412,"name":"s-2412VM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
host=172.16.116.219 port=8250 name=s-2412-VM zone=9 pod=9
guid=s-2412","enableHA":false,"limitCpuUse":false,"enableDynamicallyScale
Vm":false
Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
Hi All
I am trying to configure a zone based on a hyper-v cluster (ms windows
2012 hyper-v cluster of two nodes, using SMBv3).
The zone configuration completes successfully, including adding
primary and secondary SMBv3 storage and adding hyper-v cluster node.
However, I see that CloudStack is continuously building the system
VMs, and destroying them in an endless loop, without ever running them
successfully
What I'm seeing in the hyper-v management console:
-The Cloudstack controller is successfully is launching and
destroying the system vms:
- System VMs are created and destroyed continuously (Console Proxy nd
Secondary Storage VM) (confirmed by the cloudstack manager logs)
- System VMs appear as stopped in the CloudStack web interface, and
periodically vanish.
Looking through the logs the VM build process fails with this message:
---
.
.
2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230)
(logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to
com.cloud.agent.api.StartCommand fail on exceptionObject reference not
set to an instance of an object.
.
.
.
---
Details of my setup:
1. Hyper-V Version: Hypervisor version 6.2
2. CloudStack version/distribution: 4.5
3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual
10Gibps NIC (teamed)
4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 node
5. Zone Network Details: Basic.
6. The CloudStack controller is successfully managing VMs running on
XenServer 6.2.
Attached is a log dump (addresses sanitized) of the whole vm build cycle.
Has anyone seen this kind of issue before when integrating Hyper-V
with CloudStack ?
Is there a set of additional things I could try to troubleshoot this,
or configurations I could try?
Here are the basic troubleshooting tests I'v etried:
- Seeding the hyper-v image to secondary storage: success
- Mounting the secondary and primary storage from the Windows 2012
storage node to the CloudStck controller VM: success
- Browsing to the agent port via https from the controller to the
Hyper-V nodes: success
- Creating VMs manually on the Hyper-V nodes: success
I'm not seeing any network communications, physical hardware or
hyper-v clusters. I suppose that the fact system VMs are being
created on Hyper-V means that essentially CloudStack is communicating
with hyper-V., and controlling it correctly ... The only question is
why the System VM deployment fails.
I suspect physical network addressing configuration, but I'm not sure
how to confirm this ...
Many thanks in advance for any assistance!
Traiano
P.S: Some key debug messages in the logs (leading up to the system vms
being destroyed):
-- ---
2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
(secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm
created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null,
private IP: null
.
.
2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq
83-346478: { Ans: , MgmtId: 117586639591181, via: 83, Ver: v1,
Flags:100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
}
.
.
2015-09-17 04:45:07,692 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
(DirectAgent-293:ctx-de6632d6) (logid:138757fe) Sending cmd to
https://xx.yy.zz.223:8250/api/HypervResource/com.cloud.agent.api.StartCommand
cmd
data:{"vm":{"id":2412,"name":"s-2412VM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
host=172.16.116.219 port=8250 name=s-2412-VM zone=9 pod=9
guid=s-2412","enableHA":false,"limitCpuUse":false,"enableDynamicallyScale
Vm":false,","params":{},"uuid":"cfe1f3c6-6b1d-4984-8752-67d33a9cbb26","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e069cb94-8f38-4cf6-b39b-459230c2e18a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4281fa3a-eb9f-3556-82a6-34fc2b8bbc8b","id":14,"poolType":"SMB","host":"xx.yy.zz.221","path":"/Primary","port":445,"url":"SMB://xx.yy.zz.221/Primary/?ROLE=Primary=4281fa3a-eb9f-3556-82a6-34fc2b8bbc8b"}},"name":"ROOT-2412","size":0,"path":"e069cb94-8f38-4cf6-b39b-459230c2e18a","volumeId":2246,"vmName":"s-2412-VM"
Re: Cloudstack and Hyper-v: "Unable to start VM on Host[-94-Routing] due to com.cloud.agent.api.StartCommand fail"
Attach some Hyper-V agent log. They will give better idea on what is happening
on host.
Regards,
Anshul
> On 17-Sep-2015, at 3:27 PM, Traiano Welcome <trai...@gmail.com> wrote:
>
> Hi All
>
> I am trying to configure a zone based on a hyper-v cluster (ms windows
> 2012 hyper-v cluster of two nodes, using SMBv3).
>
> The zone configuration completes successfully, including adding
> primary and secondary SMBv3 storage and adding hyper-v cluster node.
> However, I see that CloudStack is continuously building the system
> VMs, and destroying them in an endless loop, without ever running them
> successfully
>
> What I'm seeing in the hyper-v management console:
>
> -The Cloudstack controller is successfully is launching and
> destroying the system vms:
> - System VMs are created and destroyed continuously (Console Proxy nd
> Secondary Storage VM) (confirmed by the cloudstack manager logs)
> - System VMs appear as stopped in the CloudStack web interface, and
> periodically vanish.
>
> Looking through the logs the VM build process fails with this message:
>
> ---
> .
> .
>
> 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230)
> (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to
> com.cloud.agent.api.StartCommand fail on exceptionObject reference not
> set to an instance of an object.
> .
> .
> .
> ---
>
> Details of my setup:
>
> 1. Hyper-V Version: Hypervisor version 6.2
> 2. CloudStack version/distribution: 4.5
> 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual
> 10Gibps NIC (teamed)
> 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 node
> 5. Zone Network Details: Basic.
> 6. The CloudStack controller is successfully managing VMs running on
> XenServer 6.2.
>
> Attached is a log dump (addresses sanitized) of the whole vm build cycle.
>
> Has anyone seen this kind of issue before when integrating Hyper-V
> with CloudStack ?
> Is there a set of additional things I could try to troubleshoot this,
> or configurations I could try?
>
> Here are the basic troubleshooting tests I'v etried:
>
> - Seeding the hyper-v image to secondary storage: success
> - Mounting the secondary and primary storage from the Windows 2012
> storage node to the CloudStck controller VM: success
> - Browsing to the agent port via https from the controller to the
> Hyper-V nodes: success
> - Creating VMs manually on the Hyper-V nodes: success
>
> I'm not seeing any network communications, physical hardware or
> hyper-v clusters. I suppose that the fact system VMs are being
> created on Hyper-V means that essentially CloudStack is communicating
> with hyper-V., and controlling it correctly ... The only question is
> why the System VM deployment fails.
>
> I suspect physical network addressing configuration, but I'm not sure
> how to confirm this ...
>
> Many thanks in advance for any assistance!
> Traiano
>
>
> P.S: Some key debug messages in the logs (leading up to the system vms
> being destroyed):
>
>
> -- ---
>
> 2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
> (secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm
> created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null,
> private IP: null
> .
> .
> 2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq
> 83-346478: { Ans: , MgmtId: 117586639591181, via: 83, Ver: v1,
> Flags:100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
> }
> .
> .
> 2015-09-17 04:45:07,692 DEBUG [c.c.h.h.r.HypervDirectConnectResource]
> (DirectAgent-293:ctx-de6632d6) (logid:138757fe) Sending cmd to
> https://xx.yy.zz.223:8250/api/HypervResource/com.cloud.agent.api.StartCommand
> cmd
> data:{"vm":{"id":2412,"name":"s-2412VM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"Debian
> GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
> host=172.16.116.219 port=8250 name=s-2412-VM zone=9 pod=9
> guid=s-2412","enableHA":false,"limitCpuUse":false,"enableDynamicallyScale
> Vm":false,","params":{},"uuid":"cfe1f3c6-6b1d-4984-8752-67d33a9cbb26","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO&qu
RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
Thank you, Somesh
Will try to upgrade.
Best regards
Yuri
From: somesh.na...@citrix.com
To: users@cloudstack.apache.org
Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error
on XenServer
Date: Mon, 26 Jan 2015 18:49:54 +
Yes, XS will have the correct view since it is the one that's managing the
memory. The problem is the way CS calculates the memory overhead. I don't
think there is much we can do in terms of configuration.
This is how CS arrives at total ram:
ram = (long)((ram - dom0Ram - _xsMemoryUsed) * _xsVirtualizationFactor);
In 4.3.x we derive dom0Ram from dom0's memory-dynamic-max, [dom0Ram =
vm.getMemoryStaticMax(conn);].
The fix was to derive it from dom0's memory-static-max, [dom0Ram =
vm.getMemoryStaticMax(conn);].
I see the fix in 4.4.x. So you may want to upgrade to that version to fix
this issue.
Regards,
Somesh
-Original Message-
From: Yuri Kogun [mailto:yko...@outlook.com]
Sent: Monday, January 26, 2015 11:10 AM
To: users@cloudstack.apache.org
Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error
on XenServer
Hi Somesh,
Thank you for the reply, yes I found the issue in the Jira.
The affected host was running about 31 VM's and we don't use memory over
provisioning at all. Unfortunately cloudstack fails to restart he VM and it
is a big issue for us as the restart is a part of a nightly job and we pick
it up only in the morning. The only way for us address the issue is to remove
'active' tag from the host in question to stop it from being used for
starting new Vm's.
I also noticed that the available memory is reported correctly in the xen
server console, but not inside cloudstack in the management log.
Best regards
Yuri
From: somesh.na...@citrix.com
To: users@cloudstack.apache.org
Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error
on XenServer
Date: Mon, 26 Jan 2015 15:56:01 +
Yes, seen this quite a few times. I believe you already found
CLOUDSTACK-2344.
Basically, the issue happens when cloudstack's view of available memory is
out of sync (more) than that of XS. This could happen due to incorrect
calculation of memory overhead and Dom0 memory. It is also possible that
your memory overprovisioning value is set too high so please verify that.
You might also want to check how many VMs were running on that particular
host when this error is thrown. When a host is running too many VMs (excess
of 60) then there is a possibility for such issues.
Having said that, I believe there are subsequent attempts by cloudstack to
start the VM on other hosts and the VM eventually starts. If not then we may
be looking at a potential defect.
Regards,
Somesh
-Original Message-
From: Yuri Kogun [mailto:yko...@outlook.com]
Sent: Monday, January 26, 2015 10:26 AM
To: users@cloudstack.apache.org
Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on
XenServer
Hi,
I wonder if somebody experienced a similar issue. We have a very busy dev
cloudstack installation with 10 hosts, 3 clusters and more than 300 user
VM's running across the clusters. The cpu over-provisioning is set to 3.
From time to time we are getting the following an error when starting the
VM.
2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid:
ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jan 26 01:45:24 GMT 2015
finished: Mon Jan 26 01:45:24 GMT 2015
status: failure
residentOn: com.xensource.xenapi.Host@fcaebca8
progress: 1.0
type: none/
result:
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
I traced the job executor for the VM and it looks like the process have
reported that the host have enough RAM 2060627968 to start the VM, which
requestes 1572864000 but for some reason the command failed on the Xen
server.
2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373)
Deployment found - P0=VM[User|i-3-160816-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))]
:
Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT--Pool(30))]
2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM
state transitted from :Starting to Starting with event: OperationRetryvm's
original host id: null new host id: 35 host id before state transition: null
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847
RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
Hi Somesh,
Thank you for the reply, yes I found the issue in the Jira.
The affected host was running about 31 VM's and we don't use memory over
provisioning at all. Unfortunately cloudstack fails to restart he VM and it is
a big issue for us as the restart is a part of a nightly job and we pick it up
only in the morning. The only way for us address the issue is to remove
'active' tag from the host in question to stop it from being used for starting
new Vm's.
I also noticed that the available memory is reported correctly in the xen
server console, but not inside cloudstack in the management log.
Best regards
Yuri
From: somesh.na...@citrix.com
To: users@cloudstack.apache.org
Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error
on XenServer
Date: Mon, 26 Jan 2015 15:56:01 +
Yes, seen this quite a few times. I believe you already found CLOUDSTACK-2344.
Basically, the issue happens when cloudstack's view of available memory is
out of sync (more) than that of XS. This could happen due to incorrect
calculation of memory overhead and Dom0 memory. It is also possible that your
memory overprovisioning value is set too high so please verify that. You
might also want to check how many VMs were running on that particular host
when this error is thrown. When a host is running too many VMs (excess of 60)
then there is a possibility for such issues.
Having said that, I believe there are subsequent attempts by cloudstack to
start the VM on other hosts and the VM eventually starts. If not then we may
be looking at a potential defect.
Regards,
Somesh
-Original Message-
From: Yuri Kogun [mailto:yko...@outlook.com]
Sent: Monday, January 26, 2015 10:26 AM
To: users@cloudstack.apache.org
Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on
XenServer
Hi,
I wonder if somebody experienced a similar issue. We have a very busy dev
cloudstack installation with 10 hosts, 3 clusters and more than 300 user VM's
running across the clusters. The cpu over-provisioning is set to 3. From time
to time we are getting the following an error when starting the VM.
2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid:
ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jan 26 01:45:24 GMT 2015
finished: Mon Jan 26 01:45:24 GMT 2015
status: failure
residentOn: com.xensource.xenapi.Host@fcaebca8
progress: 1.0
type: none/
result:
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
I traced the job executor for the VM and it looks like the process have
reported that the host have enough RAM 2060627968 to start the VM, which
requestes 1572864000 but for some reason the command failed on the Xen server.
2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373)
Deployment found - P0=VM[User|i-3-160816-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))]
:
Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT--Pool(30))]
2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM
state transitted from :Starting to Starting with event: OperationRetryvm's
original host id: null new host id: 35 host id before state transition: null
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Hosts's
actual total CPU: 44688 and CPU after applying overprovisioning: 58094
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) We are
allocating VM, increasing the used capacity of this host:35
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current
Used CPU: 52500 , Free CPU:5594 ,Requested CPU: 1500
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current
Used RAM: 61220061184 , Free RAM:2060627968 ,Requested RAM: 1572864000
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) CPU
STATS after allocation: for host: 35, old used: 52500, old reserved: 0,
actual total: 44688, total with o
verprovisioning: 58094; new used:54000, reserved:0; requested
cpu:1500,alloc_from_last:false
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) RAM
STATS after
RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
Yes, seen this quite a few times. I believe you already found CLOUDSTACK-2344.
Basically, the issue happens when cloudstack's view of available memory is out
of sync (more) than that of XS. This could happen due to incorrect calculation
of memory overhead and Dom0 memory. It is also possible that your memory
overprovisioning value is set too high so please verify that. You might also
want to check how many VMs were running on that particular host when this error
is thrown. When a host is running too many VMs (excess of 60) then there is a
possibility for such issues.
Having said that, I believe there are subsequent attempts by cloudstack to
start the VM on other hosts and the VM eventually starts. If not then we may be
looking at a potential defect.
Regards,
Somesh
-Original Message-
From: Yuri Kogun [mailto:yko...@outlook.com]
Sent: Monday, January 26, 2015 10:26 AM
To: users@cloudstack.apache.org
Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on
XenServer
Hi,
I wonder if somebody experienced a similar issue. We have a very busy dev
cloudstack installation with 10 hosts, 3 clusters and more than 300 user VM's
running across the clusters. The cpu over-provisioning is set to 3. From time
to time we are getting the following an error when starting the VM.
2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid:
ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jan 26 01:45:24 GMT 2015
finished: Mon Jan 26 01:45:24 GMT 2015
status: failure
residentOn: com.xensource.xenapi.Host@fcaebca8
progress: 1.0
type: none/
result:
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
I traced the job executor for the VM and it looks like the process have
reported that the host have enough RAM 2060627968 to start the VM, which
requestes 1572864000 but for some reason the command failed on the Xen server.
2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373)
Deployment found - P0=VM[User|i-3-160816-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))]
:
Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT--Pool(30))]
2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM state
transitted from :Starting to Starting with event: OperationRetryvm's original
host id: null new host id: 35 host id before state transition: null
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Hosts's
actual total CPU: 44688 and CPU after applying overprovisioning: 58094
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) We are
allocating VM, increasing the used capacity of this host:35
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current
Used CPU: 52500 , Free CPU:5594 ,Requested CPU: 1500
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current
Used RAM: 61220061184 , Free RAM:2060627968 ,Requested RAM: 1572864000
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) CPU STATS
after allocation: for host: 35, old used: 52500, old reserved: 0, actual total:
44688, total with o
verprovisioning: 58094; new used:54000, reserved:0; requested
cpu:1500,alloc_from_last:false
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) RAM STATS
after allocation: for host: 35, old used: 61220061184, old reserved: 0, total:
63280689152; new use
d: 62792925184, reserved: 0; requested mem: 1572864000,alloc_from_last:false
Please let me know if somebody had a similar problem and managed to fix it. We
are running xenserver6.2 with cloudstack 4.3.0.1
Bast regards
Yuri
Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
Hi,
I wonder if somebody experienced a similar issue. We have a very busy dev
cloudstack installation with 10 hosts, 3 clusters and more than 300 user VM's
running across the clusters. The cpu over-provisioning is set to 3. From time
to time we are getting the following an error when starting the VM.
2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid:
ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jan 26 01:45:24 GMT 2015
finished: Mon Jan 26 01:45:24 GMT 2015
status: failure
residentOn: com.xensource.xenapi.Host@fcaebca8
progress: 1.0
type: none/
result:
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
I traced the job executor for the VM and it looks like the process have
reported that the host have enough RAM 2060627968 to start the VM, which
requestes 1572864000 but for some reason the command failed on the Xen server.
2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373)
Deployment found - P0=VM[User|i-3-160816-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))]
:
Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT--Pool(30))]
2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM state
transitted from :Starting to Starting with event: OperationRetryvm's original
host id: null new host id: 35 host id before state transition: null
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Hosts's
actual total CPU: 44688 and CPU after applying overprovisioning: 58094
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) We are
allocating VM, increasing the used capacity of this host:35
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current
Used CPU: 52500 , Free CPU:5594 ,Requested CPU: 1500
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Current
Used RAM: 61220061184 , Free RAM:2060627968 ,Requested RAM: 1572864000
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) CPU STATS
after allocation: for host: 35, old used: 52500, old reserved: 0, actual total:
44688, total with o
verprovisioning: 58094; new used:54000, reserved:0; requested
cpu:1500,alloc_from_last:false
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) RAM STATS
after allocation: for host: 35, old used: 61220061184, old reserved: 0, total:
63280689152; new use
d: 62792925184, reserved: 0; requested mem: 1572864000,alloc_from_last:false
Please let me know if somebody had a similar problem and managed to fix it. We
are running xenserver6.2 with cloudstack 4.3.0.1
Bast regards
Yuri
RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on XenServer
Yes, XS will have the correct view since it is the one that's managing the
memory. The problem is the way CS calculates the memory overhead. I don't think
there is much we can do in terms of configuration.
This is how CS arrives at total ram:
ram = (long)((ram - dom0Ram - _xsMemoryUsed) * _xsVirtualizationFactor);
In 4.3.x we derive dom0Ram from dom0's memory-dynamic-max, [dom0Ram =
vm.getMemoryStaticMax(conn);].
The fix was to derive it from dom0's memory-static-max, [dom0Ram =
vm.getMemoryStaticMax(conn);].
I see the fix in 4.4.x. So you may want to upgrade to that version to fix this
issue.
Regards,
Somesh
-Original Message-
From: Yuri Kogun [mailto:yko...@outlook.com]
Sent: Monday, January 26, 2015 11:10 AM
To: users@cloudstack.apache.org
Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on
XenServer
Hi Somesh,
Thank you for the reply, yes I found the issue in the Jira.
The affected host was running about 31 VM's and we don't use memory over
provisioning at all. Unfortunately cloudstack fails to restart he VM and it is
a big issue for us as the restart is a part of a nightly job and we pick it up
only in the morning. The only way for us address the issue is to remove
'active' tag from the host in question to stop it from being used for starting
new Vm's.
I also noticed that the available memory is reported correctly in the xen
server console, but not inside cloudstack in the management log.
Best regards
Yuri
From: somesh.na...@citrix.com
To: users@cloudstack.apache.org
Subject: RE: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error
on XenServer
Date: Mon, 26 Jan 2015 15:56:01 +
Yes, seen this quite a few times. I believe you already found CLOUDSTACK-2344.
Basically, the issue happens when cloudstack's view of available memory is
out of sync (more) than that of XS. This could happen due to incorrect
calculation of memory overhead and Dom0 memory. It is also possible that your
memory overprovisioning value is set too high so please verify that. You
might also want to check how many VMs were running on that particular host
when this error is thrown. When a host is running too many VMs (excess of 60)
then there is a possibility for such issues.
Having said that, I believe there are subsequent attempts by cloudstack to
start the VM on other hosts and the VM eventually starts. If not then we may
be looking at a potential defect.
Regards,
Somesh
-Original Message-
From: Yuri Kogun [mailto:yko...@outlook.com]
Sent: Monday, January 26, 2015 10:26 AM
To: users@cloudstack.apache.org
Subject: Unable to start VM due to the HOST_NOT_ENOUGH_FREE_MEMORY error on
XenServer
Hi,
I wonder if somebody experienced a similar issue. We have a very busy dev
cloudstack installation with 10 hosts, 3 clusters and more than 300 user VM's
running across the clusters. The cpu over-provisioning is set to 3. From time
to time we are getting the following an error when starting the VM.
2015-01-26 01:45:25,547 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-578a4e5d) Task failed! Task record: uuid:
ff81a41e-3340-e7d5-6f8c-c99d4a910bb0
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jan 26 01:45:24 GMT 2015
finished: Mon Jan 26 01:45:24 GMT 2015
status: failure
residentOn: com.xensource.xenapi.Host@fcaebca8
progress: 1.0
type: none/
result:
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 1587544064, 1446559744]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
I traced the job executor for the VM and it looks like the process have
reported that the host have enough RAM 2060627968 to start the VM, which
requestes 1572864000 but for some reason the command failed on the Xen server.
2015-01-26 01:45:20,005 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373)
Deployment found - P0=VM[User|i-3-160816-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type--Pool(Id))]
:
Dest[Zone(1)-Pod(1)-Cluster(5)-Host(35)-Storage(Volume(158708|ROOT--Pool(30))]
2015-01-26 01:45:20,124 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) VM
state transitted from :Starting to Starting with event: OperationRetryvm's
original host id: null new host id: 35 host id before state transition: null
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) Hosts's
actual total CPU: 44688 and CPU after applying overprovisioning: 58094
2015-01-26 01:45:20,138 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-8:ctx-c47b7a2f job-205846/job-205847 ctx-ea6dc373) We are
allocating VM, increasing the used capacity of this host:35
2015-01-26 01:45:20,138 DEBUG
Re: Unable to start VM due to concurrent operation
Hi,
please once check the secondary storage path for download vm is right or
nor?
Thanks,Prapul Sriram
On Thursday, 1 January 2015 2:03 PM, Mohamed Infaz
infaz...@cse.mrt.ac.lk wrote:
Hi Somesh,
Yes i was successful in stopping both the system vm's and starting them
back i also could access them through the console that cloudstack provides.
May be the reason for that code snippet is that i didn't register the
template but i ran this script to install the system vm's.
1.
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
\
2. -m /srv/images/secondary \
3. -u
http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2
\
4. -h kvm \
5. -F
this was on the docs. How about upgrading my cloudstack version to 4.4.
This whole thing happened in 4.3.
Thank you.
On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote:
You mean the system VMs for Zone 1 are running? Are they showing as
connected in the DB?
If they are, can you stop these VMs and see if they start okay.
From: Mohamed Infaz [infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 21:26
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Unable to start VM due to concurrent operation
systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i
used.
On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk
wrote:
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it
is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com
wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
-
That's pointing to missing system VM template. Please verify that you
have correctly provisioned the secondary storage with the correct
system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image.
When i
want to create an instance i get the following error message saying
Unable
to start VM due to concurrent operation i did some searching on the
topic
and it said version 4.3 system template solved the issue. These are some
of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: {
Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: {
Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl]
(ExternalNetworkMonitor-1:ctx-998a86df) External devices stats collector
is
running...
2014-12-31 05:25:15,362 DEBUG
[c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterMonitor-1:ctx-dfdc73ce) Found 0 running routers.
2014-12-31 05:25:15,366 DEBUG
[c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 routers to update status.
2014-12-31 05:25:15,367 DEBUG
[c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0
Re: Unable to start VM due to concurrent operation
hi, And the status of the template i am trying to create the vm from is fully downloaded. It's status is ready. Thank you. On 2 January 2015 at 21:04, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi, This is the screen shot. It stops around 96-98% On 2 January 2015 at 20:58, Somesh Naidu somesh.na...@citrix.com wrote: What's the status of template are you trying to create a VM from? Please share either DB output or UI screenshot. -Original Message- From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk] Sent: Friday, January 02, 2015 7:03 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation Hi all, I did redeploy cloudstack and i used the 1. /usr/share/cloudstack-common/ scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F The following script to deploy system vm's. After some time they started working! Now when i try to deploy and vm i am getting the following error message. com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Template 3 has not been completely downloaded to zone 1 I did reregister the latest system vm template and i was able to successfully download it. But there are two inbuilt templates where it says in the status that the connection was time out. And one of these in built template is an in built old system vm template. How can i solve this issue now there is a system vm running properly in the environment and i have the latest template registered but it seems so cloudstack searches for those two templates which it couldn't download. Because of this i still can't create a virtual machine. Thank you. On 2 January 2015 at 08:08, Abhinandan Prateek abhinandan.prat...@shapeblue.com wrote: The sec store has some issues, you can login into KVM host and check if the sec storage mount point is visible and the said folder is there. Check if ssvm is running fine. It is also possible that the sec store is fine but the template that is being used to create VM is in inconsistent state, then you need to remove and re-register the template. -abhi On 01-Jan-2015, at 8:35 pm, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi, I updated my cloudstack deployment to 4.4.2 to 4.3.1. Now when we try to deploy a VM i am getting following error message. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[30|vm=58|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 10.8.100.201:/mnt/secondary/template/tmpl/2/205 /mnt/8c6fa424-cbc7-3421-8f05-d69a43ffa15c) unexpected exit status 32: mount.nfs: mounting 10.8.100.201:/mnt/secondary/template/tmpl/2/205 failed, reason given by server: No such file or directory Thank you. On 1 January 2015 at 14:03, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Yes i was successful in stopping both the system vm's and starting them back i also could access them through the console that cloudstack provides. May be the reason for that code snippet is that i didn't register the template but i ran this script to install the system vm's. 1. /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F this was on the docs. How about upgrading my cloudstack version to 4.4. This whole thing happened in 4.3. Thank you. On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote: You mean the system VMs for Zone 1 are running? Are they showing as connected in the DB? If they are, can you stop these VMs and see if they start okay. From: Mohamed Infaz [infaz...@cse.mrt.ac.lk] Sent: Tuesday, December 30, 2014 21:26 To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i used. On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Thank you for the reply. Well both of my system vm's are running and it is downloading the iso image. And i was able to ping them. The system vm template i am using is for the version 4.3 cloudstack. Thank you. On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com wrote: Mohamed, The log snippet
Re: Unable to start VM due to concurrent operation
Hi, This is the screen shot. It stops around 96-98% On 2 January 2015 at 20:58, Somesh Naidu somesh.na...@citrix.com wrote: What's the status of template are you trying to create a VM from? Please share either DB output or UI screenshot. -Original Message- From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk] Sent: Friday, January 02, 2015 7:03 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation Hi all, I did redeploy cloudstack and i used the 1. /usr/share/cloudstack-common/ scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F The following script to deploy system vm's. After some time they started working! Now when i try to deploy and vm i am getting the following error message. com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Template 3 has not been completely downloaded to zone 1 I did reregister the latest system vm template and i was able to successfully download it. But there are two inbuilt templates where it says in the status that the connection was time out. And one of these in built template is an in built old system vm template. How can i solve this issue now there is a system vm running properly in the environment and i have the latest template registered but it seems so cloudstack searches for those two templates which it couldn't download. Because of this i still can't create a virtual machine. Thank you. On 2 January 2015 at 08:08, Abhinandan Prateek abhinandan.prat...@shapeblue.com wrote: The sec store has some issues, you can login into KVM host and check if the sec storage mount point is visible and the said folder is there. Check if ssvm is running fine. It is also possible that the sec store is fine but the template that is being used to create VM is in inconsistent state, then you need to remove and re-register the template. -abhi On 01-Jan-2015, at 8:35 pm, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi, I updated my cloudstack deployment to 4.4.2 to 4.3.1. Now when we try to deploy a VM i am getting following error message. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[30|vm=58|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 10.8.100.201:/mnt/secondary/template/tmpl/2/205 /mnt/8c6fa424-cbc7-3421-8f05-d69a43ffa15c) unexpected exit status 32: mount.nfs: mounting 10.8.100.201:/mnt/secondary/template/tmpl/2/205 failed, reason given by server: No such file or directory Thank you. On 1 January 2015 at 14:03, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Yes i was successful in stopping both the system vm's and starting them back i also could access them through the console that cloudstack provides. May be the reason for that code snippet is that i didn't register the template but i ran this script to install the system vm's. 1. /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F this was on the docs. How about upgrading my cloudstack version to 4.4. This whole thing happened in 4.3. Thank you. On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote: You mean the system VMs for Zone 1 are running? Are they showing as connected in the DB? If they are, can you stop these VMs and see if they start okay. From: Mohamed Infaz [infaz...@cse.mrt.ac.lk] Sent: Tuesday, December 30, 2014 21:26 To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i used. On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Thank you for the reply. Well both of my system vm's are running and it is downloading the iso image. And i was able to ping them. The system vm template i am using is for the version 4.3 cloudstack. Thank you. On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com wrote: Mohamed, The log snippet you have shared doesn't seem to be relevant to the concurrent operation exception (can't see that exception stack). The message that did catch my attention though is, - 2014-12-31 05:25:07,972
RE: Unable to start VM due to concurrent operation
What's the status of template are you trying to create a VM from? Please share either DB output or UI screenshot. -Original Message- From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk] Sent: Friday, January 02, 2015 7:03 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation Hi all, I did redeploy cloudstack and i used the 1. /usr/share/cloudstack-common/ scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F The following script to deploy system vm's. After some time they started working! Now when i try to deploy and vm i am getting the following error message. com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Template 3 has not been completely downloaded to zone 1 I did reregister the latest system vm template and i was able to successfully download it. But there are two inbuilt templates where it says in the status that the connection was time out. And one of these in built template is an in built old system vm template. How can i solve this issue now there is a system vm running properly in the environment and i have the latest template registered but it seems so cloudstack searches for those two templates which it couldn't download. Because of this i still can't create a virtual machine. Thank you. On 2 January 2015 at 08:08, Abhinandan Prateek abhinandan.prat...@shapeblue.com wrote: The sec store has some issues, you can login into KVM host and check if the sec storage mount point is visible and the said folder is there. Check if ssvm is running fine. It is also possible that the sec store is fine but the template that is being used to create VM is in inconsistent state, then you need to remove and re-register the template. -abhi On 01-Jan-2015, at 8:35 pm, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi, I updated my cloudstack deployment to 4.4.2 to 4.3.1. Now when we try to deploy a VM i am getting following error message. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[30|vm=58|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 10.8.100.201:/mnt/secondary/template/tmpl/2/205 /mnt/8c6fa424-cbc7-3421-8f05-d69a43ffa15c) unexpected exit status 32: mount.nfs: mounting 10.8.100.201:/mnt/secondary/template/tmpl/2/205 failed, reason given by server: No such file or directory Thank you. On 1 January 2015 at 14:03, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Yes i was successful in stopping both the system vm's and starting them back i also could access them through the console that cloudstack provides. May be the reason for that code snippet is that i didn't register the template but i ran this script to install the system vm's. 1. /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F this was on the docs. How about upgrading my cloudstack version to 4.4. This whole thing happened in 4.3. Thank you. On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote: You mean the system VMs for Zone 1 are running? Are they showing as connected in the DB? If they are, can you stop these VMs and see if they start okay. From: Mohamed Infaz [infaz...@cse.mrt.ac.lk] Sent: Tuesday, December 30, 2014 21:26 To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i used. On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Thank you for the reply. Well both of my system vm's are running and it is downloading the iso image. And i was able to ping them. The system vm template i am using is for the version 4.3 cloudstack. Thank you. On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com wrote: Mohamed, The log snippet you have shared doesn't seem to be relevant to the concurrent operation exception (can't see that exception stack). The message that did catch my attention though is, - 2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-463d2f57) System vm template is not ready at data center 1, wait until it is ready to launch secondary storage vm 2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx
Re: Unable to start VM due to concurrent operation
Hi all, I did redeploy cloudstack and i used the 1. /usr/share/cloudstack-common/ scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F The following script to deploy system vm's. After some time they started working! Now when i try to deploy and vm i am getting the following error message. com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Template 3 has not been completely downloaded to zone 1 I did reregister the latest system vm template and i was able to successfully download it. But there are two inbuilt templates where it says in the status that the connection was time out. And one of these in built template is an in built old system vm template. How can i solve this issue now there is a system vm running properly in the environment and i have the latest template registered but it seems so cloudstack searches for those two templates which it couldn't download. Because of this i still can't create a virtual machine. Thank you. On 2 January 2015 at 08:08, Abhinandan Prateek abhinandan.prat...@shapeblue.com wrote: The sec store has some issues, you can login into KVM host and check if the sec storage mount point is visible and the said folder is there. Check if ssvm is running fine. It is also possible that the sec store is fine but the template that is being used to create VM is in inconsistent state, then you need to remove and re-register the template. -abhi On 01-Jan-2015, at 8:35 pm, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi, I updated my cloudstack deployment to 4.4.2 to 4.3.1. Now when we try to deploy a VM i am getting following error message. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[30|vm=58|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 10.8.100.201:/mnt/secondary/template/tmpl/2/205 /mnt/8c6fa424-cbc7-3421-8f05-d69a43ffa15c) unexpected exit status 32: mount.nfs: mounting 10.8.100.201:/mnt/secondary/template/tmpl/2/205 failed, reason given by server: No such file or directory Thank you. On 1 January 2015 at 14:03, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Yes i was successful in stopping both the system vm's and starting them back i also could access them through the console that cloudstack provides. May be the reason for that code snippet is that i didn't register the template but i ran this script to install the system vm's. 1. /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \ 2. -m /srv/images/secondary \ 3. -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \ 4. -h kvm \ 5. -F this was on the docs. How about upgrading my cloudstack version to 4.4. This whole thing happened in 4.3. Thank you. On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote: You mean the system VMs for Zone 1 are running? Are they showing as connected in the DB? If they are, can you stop these VMs and see if they start okay. From: Mohamed Infaz [infaz...@cse.mrt.ac.lk] Sent: Tuesday, December 30, 2014 21:26 To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: Unable to start VM due to concurrent operation systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i used. On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote: Hi Somesh, Thank you for the reply. Well both of my system vm's are running and it is downloading the iso image. And i was able to ping them. The system vm template i am using is for the version 4.3 cloudstack. Thank you. On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com wrote: Mohamed, The log snippet you have shared doesn't seem to be relevant to the concurrent operation exception (can't see that exception stack). The message that did catch my attention though is, - 2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-463d2f57) System vm template is not ready at data center 1, wait until it is ready to launch secondary storage vm 2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage VM yet - That's pointing to missing system VM template. Please verify that you have correctly provisioned the secondary storage with the correct system VM template (as per the docs). -Somesh -Original Message- From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk
Re: Unable to start VM due to concurrent operation
The sec store has some issues, you can login into KVM host and check if the sec
storage mount point is visible and the said folder is there. Check if ssvm is
running fine.
It is also possible that the sec store is fine but the template that is being
used to create VM is in inconsistent state, then you need to remove and
re-register the template.
-abhi
On 01-Jan-2015, at 8:35 pm, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote:
Hi,
I updated my cloudstack deployment to 4.4.2 to 4.3.1. Now when we try to
deploy a VM i am getting following error
message. com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:2] is unreachable: Unable to create
Vol[30|vm=58|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error: Child process (/bin/mount
10.8.100.201:/mnt/secondary/template/tmpl/2/205
/mnt/8c6fa424-cbc7-3421-8f05-d69a43ffa15c) unexpected exit status 32:
mount.nfs: mounting 10.8.100.201:/mnt/secondary/template/tmpl/2/205 failed,
reason given by server: No such file or directory
Thank you.
On 1 January 2015 at 14:03, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote:
Hi Somesh,
Yes i was successful in stopping both the system vm's and starting them
back i also could access them through the console that cloudstack provides.
May be the reason for that code snippet is that i didn't register the
template but i ran this script to install the system vm's.
1.
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
\
2. -m /srv/images/secondary \
3. -u
http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2
\
4. -h kvm \
5. -F
this was on the docs. How about upgrading my cloudstack version to 4.4.
This whole thing happened in 4.3.
Thank you.
On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote:
You mean the system VMs for Zone 1 are running? Are they showing as
connected in the DB?
If they are, can you stop these VMs and see if they start okay.
From: Mohamed Infaz [infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 21:26
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Unable to start VM due to concurrent operation
systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i
used.
On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk
wrote:
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it
is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com
wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
-
That's pointing to missing system VM template. Please verify that you
have correctly provisioned the secondary storage with the correct
system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the
setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image.
When i
want to create an instance i get the following error message saying
Unable
to start VM due to concurrent operation i did some searching on the
topic
and it said version 4.3 system template solved the issue. These are
some
of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: {
Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand
Re: Unable to start VM due to concurrent operation
Hi,
I updated my cloudstack deployment to 4.4.2 to 4.3.1. Now when we try to
deploy a VM i am getting following error
message. com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:2] is unreachable: Unable to create
Vol[30|vm=58|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error: Child process (/bin/mount
10.8.100.201:/mnt/secondary/template/tmpl/2/205
/mnt/8c6fa424-cbc7-3421-8f05-d69a43ffa15c) unexpected exit status 32:
mount.nfs: mounting 10.8.100.201:/mnt/secondary/template/tmpl/2/205 failed,
reason given by server: No such file or directory
Thank you.
On 1 January 2015 at 14:03, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote:
Hi Somesh,
Yes i was successful in stopping both the system vm's and starting them
back i also could access them through the console that cloudstack provides.
May be the reason for that code snippet is that i didn't register the
template but i ran this script to install the system vm's.
1.
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
\
2. -m /srv/images/secondary \
3. -u
http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2
\
4. -h kvm \
5. -F
this was on the docs. How about upgrading my cloudstack version to 4.4.
This whole thing happened in 4.3.
Thank you.
On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote:
You mean the system VMs for Zone 1 are running? Are they showing as
connected in the DB?
If they are, can you stop these VMs and see if they start okay.
From: Mohamed Infaz [infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 21:26
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Unable to start VM due to concurrent operation
systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i
used.
On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk
wrote:
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it
is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com
wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
-
That's pointing to missing system VM template. Please verify that you
have correctly provisioned the secondary storage with the correct
system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the
setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image.
When i
want to create an instance i get the following error message saying
Unable
to start VM due to concurrent operation i did some searching on the
topic
and it said version 4.3 system template solved the issue. These are
some
of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: {
Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: {
Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG
Re: Unable to start VM due to concurrent operation
Hi Somesh,
Yes i was successful in stopping both the system vm's and starting them
back i also could access them through the console that cloudstack provides.
May be the reason for that code snippet is that i didn't register the
template but i ran this script to install the system vm's.
1.
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
\
2. -m /srv/images/secondary \
3. -u
http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2
\
4. -h kvm \
5. -F
this was on the docs. How about upgrading my cloudstack version to 4.4.
This whole thing happened in 4.3.
Thank you.
On 1 January 2015 at 01:39, Somesh Naidu somesh.na...@citrix.com wrote:
You mean the system VMs for Zone 1 are running? Are they showing as
connected in the DB?
If they are, can you stop these VMs and see if they start okay.
From: Mohamed Infaz [infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 21:26
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Unable to start VM due to concurrent operation
systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i
used.
On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk
wrote:
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it
is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com
wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
-
That's pointing to missing system VM template. Please verify that you
have correctly provisioned the secondary storage with the correct
system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image.
When i
want to create an instance i get the following error message saying
Unable
to start VM due to concurrent operation i did some searching on the
topic
and it said version 4.3 system template solved the issue. These are some
of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data
center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: {
Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: {
Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl]
(ExternalNetworkMonitor-1:ctx-998a86df) External devices stats collector
is
running...
2014-12-31 05:25:15,362 DEBUG
[c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterMonitor-1:ctx-dfdc73ce) Found 0 running routers.
2014-12-31 05:25:15,366 DEBUG
[c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 routers to update status.
2014-12-31 05:25:15,367 DEBUG
[c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 networks to update RvR
status.
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Snapshot scheduler.poll is being called
at
2014-12-30 23:55:15 GMT
RE: Unable to start VM due to concurrent operation
You mean the system VMs for Zone 1 are running? Are they showing as connected
in the DB?
If they are, can you stop these VMs and see if they start okay.
From: Mohamed Infaz [infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 21:26
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Unable to start VM due to concurrent operation
systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i used.
On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote:
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com
wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage
VM yet
-
That's pointing to missing system VM template. Please verify that you
have correctly provisioned the secondary storage with the correct system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image. When i
want to create an instance i get the following error message saying
Unable
to start VM due to concurrent operation i did some searching on the topic
and it said version 4.3 system template solved the issue. These are some
of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: { Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: { Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl]
(ExternalNetworkMonitor-1:ctx-998a86df) External devices stats collector
is
running...
2014-12-31 05:25:15,362 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterMonitor-1:ctx-dfdc73ce) Found 0 running routers.
2014-12-31 05:25:15,366 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 routers to update status.
2014-12-31 05:25:15,367 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 networks to update RvR
status.
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Snapshot scheduler.poll is being called at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Got 0 snapshots to be executed at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:18,535 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 3
2014-12-31 05:25:18,698 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-10:null) Ping from 2
2014-12-31 05:25:19,834 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-f10d32f0) Found the following agents behind on ping:
[1]
2014-12-31 05:25:19,835 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-f10d32f0)
Ping timeout for host 1, do invstigation
2014-12-31 05:25:19,837 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) Investigating why host 1 has disconnected
with event PingTimeout
2014-12-31 05:25:19,837 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool
RE: Unable to start VM due to concurrent operation
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the concurrent
operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center 1,
wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage VM
yet
-
That's pointing to missing system VM template. Please verify that you have
correctly provisioned the secondary storage with the correct system VM template
(as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image. When i
want to create an instance i get the following error message saying Unable
to start VM due to concurrent operation i did some searching on the topic
and it said version 4.3 system template solved the issue. These are some of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: { Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: { Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl]
(ExternalNetworkMonitor-1:ctx-998a86df) External devices stats collector is
running...
2014-12-31 05:25:15,362 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterMonitor-1:ctx-dfdc73ce) Found 0 running routers.
2014-12-31 05:25:15,366 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 routers to update status.
2014-12-31 05:25:15,367 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 networks to update RvR status.
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Snapshot scheduler.poll is being called at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Got 0 snapshots to be executed at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:18,535 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 3
2014-12-31 05:25:18,698 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-10:null) Ping from 2
2014-12-31 05:25:19,834 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-f10d32f0) Found the following agents behind on ping: [1]
2014-12-31 05:25:19,835 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-f10d32f0)
Ping timeout for host 1, do invstigation
2014-12-31 05:25:19,837 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) Investigating why host 1 has disconnected
with event PingTimeout
2014-12-31 05:25:19,837 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) checking if agent (1) is alive
2014-12-31 05:25:19,839 DEBUG [c.c.a.t.Request]
(AgentTaskPool-14:ctx-43a85937) Seq 1-449186863: Sending { Cmd , MgmtId:
248795600505608, via: 1(virtualops-h4), Ver: v1, Flags: 100011,
[{com.cloud.agent.api.CheckHealthCommand:{wait:50}}] }
2014-12-31 05:25:19,844 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-7:null) Seq 1-449186863: Processing: { Ans: ,
MgmtId: 248795600505608, via: 1, Ver: v1, Flags: 10,
[{com.cloud.agent.api.CheckHealthAnswer:{result:true,details:resource
is alive,wait:0}}] }
2014-12-31 05:25:19,844 DEBUG [c.c.a.t.Request]
(AgentTaskPool-14:ctx-43a85937) Seq 1-449186863: Received: { Ans: ,
MgmtId: 248795600505608, via: 1, Ver: v1, Flags: 10, { CheckHealthAnswer } }
2014-12-31 05:25:19,844 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) Details from executing class
com.cloud.agent.api.CheckHealthCommand: resource is alive
2014-12-31 05:25
Re: Unable to start VM due to concurrent operation
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage
VM yet
-
That's pointing to missing system VM template. Please verify that you have
correctly provisioned the secondary storage with the correct system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image. When i
want to create an instance i get the following error message saying Unable
to start VM due to concurrent operation i did some searching on the topic
and it said version 4.3 system template solved the issue. These are some of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: { Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: { Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl]
(ExternalNetworkMonitor-1:ctx-998a86df) External devices stats collector is
running...
2014-12-31 05:25:15,362 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterMonitor-1:ctx-dfdc73ce) Found 0 running routers.
2014-12-31 05:25:15,366 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 routers to update status.
2014-12-31 05:25:15,367 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 networks to update RvR status.
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Snapshot scheduler.poll is being called at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Got 0 snapshots to be executed at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:18,535 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 3
2014-12-31 05:25:18,698 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-10:null) Ping from 2
2014-12-31 05:25:19,834 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-f10d32f0) Found the following agents behind on ping:
[1]
2014-12-31 05:25:19,835 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-f10d32f0)
Ping timeout for host 1, do invstigation
2014-12-31 05:25:19,837 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) Investigating why host 1 has disconnected
with event PingTimeout
2014-12-31 05:25:19,837 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) checking if agent (1) is alive
2014-12-31 05:25:19,839 DEBUG [c.c.a.t.Request]
(AgentTaskPool-14:ctx-43a85937) Seq 1-449186863: Sending { Cmd , MgmtId:
248795600505608, via: 1(virtualops-h4), Ver: v1, Flags: 100011,
[{com.cloud.agent.api.CheckHealthCommand:{wait:50}}] }
2014-12-31 05:25:19,844 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-7:null) Seq 1-449186863: Processing: { Ans: ,
MgmtId: 248795600505608, via: 1, Ver: v1, Flags: 10,
[{com.cloud.agent.api.CheckHealthAnswer:{result:true,details:resource
is alive,wait:0}}] }
2014-12-31
Re: Unable to start VM due to concurrent operation
systemvm64template-2014-01-14-master-kvm.qcow2.bz2 This is the template i used.
On 31 December 2014 at 07:49, Mohamed Infaz infaz...@cse.mrt.ac.lk wrote:
Hi Somesh,
Thank you for the reply. Well both of my system vm's are running and it is
downloading the iso image. And i was able to ping them. The system vm
template i am using is for the version 4.3 cloudstack.
Thank you.
On 30 December 2014 at 23:59, Somesh Naidu somesh.na...@citrix.com
wrote:
Mohamed,
The log snippet you have shared doesn't seem to be relevant to the
concurrent operation exception (can't see that exception stack).
The message that did catch my attention though is,
-
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary storage
VM yet
-
That's pointing to missing system VM template. Please verify that you
have correctly provisioned the secondary storage with the correct system VM
template (as per the docs).
-Somesh
-Original Message-
From: Mohamed Infaz [mailto:infaz...@cse.mrt.ac.lk]
Sent: Tuesday, December 30, 2014 11:03 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Unable to start VM due to concurrent operation
Hi All,
I have successfully deployed cloudstack 4.3 with two hosts and the setup
runs the management server on another physical host. I had issues with
downloading an ISO but finally i was able to download an ISO image. When i
want to create an instance i get the following error message saying
Unable
to start VM due to concurrent operation i did some searching on the topic
and it said version 4.3 system template solved the issue. These are some
of
the MS logs that i collected what could be the issue?
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) System vm template is not ready at data center
1, wait until it is ready to launch secondary storage vm
2014-12-31 05:25:07,972 DEBUG [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-463d2f57) Zone 1 is not ready to launch secondary
storage
VM yet
2014-12-31 05:25:12,425 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) Ping from 4
2014-12-31 05:25:12,834 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Processing Seq 3-91695: { Cmd
, MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{com.cloud.agent.api.ConsoleProxyLoadReportCommand:{_proxyVmId:1,_loadInfo:{\n
\connections\: []\n},wait:0}}] }
2014-12-31 05:25:12,920 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 3-91695: Sending Seq 3-91695: { Ans: ,
MgmtId: 248795600505608, via: 3, Ver: v1, Flags: 100010,
[{com.cloud.agent.api.AgentControlAnswer:{result:true,wait:0}}] }
2014-12-31 05:25:15,239 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl]
(ExternalNetworkMonitor-1:ctx-998a86df) External devices stats collector
is
running...
2014-12-31 05:25:15,362 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterMonitor-1:ctx-dfdc73ce) Found 0 running routers.
2014-12-31 05:25:15,366 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 routers to update status.
2014-12-31 05:25:15,367 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-79d58896) Found 0 networks to update RvR
status.
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Snapshot scheduler.poll is being called at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:15,393 DEBUG [c.c.s.s.SnapshotSchedulerImpl]
(SnapshotPollTask:ctx-ae761f6b) Got 0 snapshots to be executed at
2014-12-30 23:55:15 GMT
2014-12-31 05:25:18,535 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 3
2014-12-31 05:25:18,698 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-10:null) Ping from 2
2014-12-31 05:25:19,834 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-f10d32f0) Found the following agents behind on ping:
[1]
2014-12-31 05:25:19,835 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-f10d32f0)
Ping timeout for host 1, do invstigation
2014-12-31 05:25:19,837 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) Investigating why host 1 has disconnected
with event PingTimeout
2014-12-31 05:25:19,837 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-43a85937) checking if agent (1) is alive
2014-12-31 05:25:19,839 DEBUG [c.c.a.t.Request]
(AgentTaskPool-14:ctx-43a85937) Seq 1-449186863: Sending { Cmd , MgmtId:
248795600505608, via: 1(virtualops-h4), Ver: v1, Flags: 100011,
[{com.cloud.agent.api.CheckHealthCommand:{wait:50}}] }
2014-12-31 05:25:19,844 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-7:null) Seq 1-449186863: Processing
Re: Unable to start VM
Hi, I have same error after computing node restaring ... Did you find solution? fix ? On Sat, Nov 16, 2013 at 3:17 PM, m2m isb isb...@gmail.com wrote: Hello All We have a problem about Starting VMs. We have a ClousdStack 4.1.1 running with KVM, Unable to start VMs with Following error. CloudStack management Server Log. /var/log/cloudstack/management/management-server.log 2013-11-16 14:30:56,859 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-14:job-926) DataCenter id = '2' provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning. 2013-11-16 14:30:56,874 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 28 2013-11-16 14:30:56,879 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) Hosts's actual total CPU: 59832 and CPU after applying overprovisioning: 119664 2013-11-16 14:30:56,879 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) release cpu from host: 28, old used: 21480,reserved: 4096, actual total: 59832, total with overprovisioning: 119664; new used: 17384,reserved:4096; movedfromreserved: false,moveToReserveredfalse 2013-11-16 14:30:56,879 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) release mem from host: 28, old used: 22682796032,reserved: 4294967296, total: 67519684608; new used: 18387828736,reserved:4294967296; movedfromreserved: false,moveToReserveredfalse 2013-11-16 14:30:56,895 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-14:job-926) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.StartVMCmd com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|m2m-vdgw02]Scope=interface com.cloud.dc.DataCenter; id=2 at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:728) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-11-16 14:30:56,895 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-14:job-926) Complete async job-926, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Unable to create a deployment for VM[User|m2m-vdgw02] CloudStack Computing Node 013-11-16 15:36:57,148 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) Execution is successful. 2013-11-16 15:36:57,149 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) Try to stop the vm at first 2013-11-16 15:36:57,152 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) Failed to stop VM :i-3-110-VM : org.libvirt.LibvirtException: ドメインは見つかりませんでした: UUID '40a685f6-405a-3fef-8591-b73f00a4cdd3' に一致するドメインがありません at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source) at org.libvirt.Connect.domainLookupByUUID(Unknown Source) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032) at com.cloud.agent.Agent.processRequest(Agent.java:525) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) at com.cloud.utils.nio.Task.run(Task.java:83) at
Unable to start VM
Hello All We have a problem about Starting VMs. We have a ClousdStack 4.1.1 running with KVM, Unable to start VMs with Following error. CloudStack management Server Log. /var/log/cloudstack/management/management-server.log 2013-11-16 14:30:56,859 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-14:job-926) DataCenter id = '2' provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning. 2013-11-16 14:30:56,874 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 28 2013-11-16 14:30:56,879 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) Hosts's actual total CPU: 59832 and CPU after applying overprovisioning: 119664 2013-11-16 14:30:56,879 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) release cpu from host: 28, old used: 21480,reserved: 4096, actual total: 59832, total with overprovisioning: 119664; new used: 17384,reserved:4096; movedfromreserved: false,moveToReserveredfalse 2013-11-16 14:30:56,879 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-14:job-926) release mem from host: 28, old used: 22682796032,reserved: 4294967296, total: 67519684608; new used: 18387828736,reserved:4294967296; movedfromreserved: false,moveToReserveredfalse 2013-11-16 14:30:56,895 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-14:job-926) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.StartVMCmd com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|m2m-vdgw02]Scope=interface com.cloud.dc.DataCenter; id=2 at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:728) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-11-16 14:30:56,895 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-14:job-926) Complete async job-926, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Unable to create a deployment for VM[User|m2m-vdgw02] CloudStack Computing Node 013-11-16 15:36:57,148 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) Execution is successful. 2013-11-16 15:36:57,149 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) Try to stop the vm at first 2013-11-16 15:36:57,152 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) Failed to stop VM :i-3-110-VM : org.libvirt.LibvirtException: ドメインは見つかりませんでした: UUID '40a685f6-405a-3fef-8591-b73f00a4cdd3' に一致するドメインがありません at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source) at org.libvirt.Connect.domainLookupByUUID(Unknown Source) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032) at com.cloud.agent.Agent.processRequest(Agent.java:525) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) at com.cloud.utils.nio.Task.run(Task.java:83) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-11-16 15:36:57,158 DEBUG
