Hi David I suspect that your problem is that the documentation hasn't been updated fully for 4.2.
You're probably using system VM template 'systemvmtemplate-4.2-vh7.ova' which is misnamed as it's for 4.1 The correct download path for the 4.2 template for ESXi is: http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh8.ova It can be a bit of a pain to kill off the old/wrong template. So to re-seed sec storage... Remember to use -F to force an overwrite when re-running the cloud-install-sys-tmplt script. Also disable the zone and destroy any system VMs. If you want to go belt and braces; find the old ESXi system VM templates in secondary storage and delete '...vh7' and the template.properties file too before re-running cloud-install-sys-tmplt. Regards, Paul Angus S: +44 20 3603 0540 | M: +447711418784 paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> From: Vosbury, David [mailto:david.vosb...@systemsmadesimple.com] Sent: 05 December 2013 14:34 To: users@cloudstack.apache.org Subject: System VMs not starting From: Vosbury, David Sent: Thursday, December 05, 2013 12:33 AM To: 'd...@cloudstack.apache.org'; 'users@cloudstack.apache.org' Subject: System VMs not starting Hello, I've been trying to get an new install of CloudStack 4.2 working with a Vsphere cluster. I've went through creating my zone with advanced networking. I can see the system VMs but they are not started. The secondary storage vm seems to keep appearing and disappearing however. When I look in vcenter I see that vms are being created one after another named root-2 and then root-192, root-193 etc. They keep creating one after another until I shutdown the management service. The vm's never start-up. Here is part of the management-server.log Has anyone seen this before? I've seen some references to bugs that sound like this problem (cloudstack-1228)that were supposedly fixed back in August. I don' t know if I have an earlier build of 4.2, but I downloaded and compiled it just last week and I'm having this issue. I've tried some of the workarounds listed in this bug report and still can't get the system vm's to start and stop respawing on my vmware hypervisor hosts. Any help would be greatly appreciated. I'm getting to the point where I'm going to look for another solution as I can't seem to get Cloudstack working with Vsphere. 2013-12-05 00:07:28,642 WARN [utils.crypt.EncryptionSecretKeyChecker] (Timer-2:null) Encryption already enabled, is check() called twice? 2013-12-05 00:07:29,394 DEBUG [cloud.server.ConfigurationServerImpl] (Timer-2:null) mount: warning: /systemvm_mnt seems to be mounted read-only. 2013-12-05 00:07:30,316 INFO [xen.resource.XenServerConnectionPool] (Timer-2:null) XenServer Connection Pool Configs: manage.xenserver.pool.master=true; sleep.interval.on.error=10000 2013-12-05 00:07:30,997 WARN [utils.crypt.EncryptionSecretKeyChecker] (Timer-2:null) Encryption already enabled, is check() called twice? 2013-12-05 00:07:34,050 WARN [apache.cloudstack.alerts] (Cluster-Notification-1:null) alertType:: 14 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: Management server node 10.2.150.100 is up 2013-12-05 00:07:34,071 WARN [cloud.cluster.ClusterManagerImpl] (Cluster-Notification-1:null) Notifying management server join event took 39 ms 2013-12-05 00:08:33,140 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-2:esxi2.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:08:33,145 DEBUG [agent.transport.Request] (DirectAgent-2:null) Seq 1-1861287941: Processing: { Ans: , MgmtId: 146454222325, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: The name 'ROOT-2' already exists.","wait":0}}] } 2013-12-05 00:08:33,163 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (consoleproxy-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4838d3f7<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4838d3f7>), no need to delete from object in store ref table 2013-12-05 00:08:33,164 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:08:33,169 INFO [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:08:56,958 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-4:esxi1.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-203/ROOT-203.vmdk was not found 2013-12-05 00:08:56,960 DEBUG [agent.transport.Request] (DirectAgent-4:null) Seq 2-2058223621: Processing: { Ans: , MgmtId: 146454222325, via: 2, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-203/ROOT-203.vmdk was not found","wait":0}}] } 2013-12-05 00:08:56,978 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (secstorage-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@6e0e449e<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@6e0e449e>), no need to delete from object in store ref table 2013-12-05 00:08:56,979 DEBUG [cloud.storage.VolumeManagerImpl] (secstorage-1:null) Unable to create Vol[203|vm=203|ROOT]:java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-203/ROOT-203.vmdk was not found 2013-12-05 00:08:56,980 INFO [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[203|vm=203|ROOT]:java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-203/ROOT-203.vmdk was not found 2013-12-05 00:08:57,231 DEBUG [cloud.capacity.CapacityManagerImpl] (consoleproxy-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:08:57,272 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Exception while trying to start console proxy com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:08:57,333 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:08:57,370 WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-203-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:08:57,376 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 203, will recycle it and start a new one 2013-12-05 00:09:02,891 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-15:esxi1.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: The name 'ROOT-203' already exists. 2013-12-05 00:09:02,896 DEBUG [agent.transport.Request] (DirectAgent-15:null) Seq 2-2058223626: Processing: { Ans: , MgmtId: 146454222325, via: 2, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: The name 'ROOT-203' already exists.","wait":0}}] } 2013-12-05 00:09:02,911 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (secstorage-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@555bb5a1<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@555bb5a1>), no need to delete from object in store ref table 2013-12-05 00:09:02,912 DEBUG [cloud.storage.VolumeManagerImpl] (secstorage-1:null) Unable to create Vol[203|vm=203|ROOT]:java.lang.RuntimeException: The name 'ROOT-203' already exists. 2013-12-05 00:09:02,912 INFO [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[203|vm=203|ROOT]:java.lang.RuntimeException: The name 'ROOT-203' already exists. 2013-12-05 00:09:03,011 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-1:esxi2.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:09:03,015 DEBUG [agent.transport.Request] (DirectAgent-1:null) Seq 1-1861287943: Processing: { Ans: , MgmtId: 146454222325, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: The name 'ROOT-2' already exists.","wait":0}}] } 2013-12-05 00:09:03,031 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (consoleproxy-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@10aba5d3<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@10aba5d3>), no need to delete from object in store ref table 2013-12-05 00:09:03,032 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:09:03,032 INFO [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:09:03,171 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:09:03,205 WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-203-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:09:03,209 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 203, will recycle it and start a new one 2013-12-05 00:09:03,312 DEBUG [cloud.capacity.CapacityManagerImpl] (consoleproxy-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:09:03,346 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Exception while trying to start console proxy com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:09:32,800 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-16:esxi2.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:09:32,804 DEBUG [agent.transport.Request] (DirectAgent-16:null) Seq 1-1861287944: Processing: { Ans: , MgmtId: 146454222325, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: The name 'ROOT-2' already exists.","wait":0}}] } 2013-12-05 00:09:32,816 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (consoleproxy-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@761b309<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@761b309>), no need to delete from object in store ref table 2013-12-05 00:09:32,817 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:09:32,817 INFO [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:09:55,743 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-6:esxi1.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-204/ROOT-204.vmdk was not found 2013-12-05 00:09:55,747 DEBUG [agent.transport.Request] (DirectAgent-6:null) Seq 2-2058223629: Processing: { Ans: , MgmtId: 146454222325, via: 2, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-204/ROOT-204.vmdk was not found","wait":0}}] } 2013-12-05 00:09:55,762 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (secstorage-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5022c22d<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5022c22d>), no need to delete from object in store ref table 2013-12-05 00:09:55,763 DEBUG [cloud.storage.VolumeManagerImpl] (secstorage-1:null) Unable to create Vol[204|vm=204|ROOT]:java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-204/ROOT-204.vmdk was not found 2013-12-05 00:09:55,763 INFO [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[204|vm=204|ROOT]:java.lang.RuntimeException: File [iSCSI-Datastore-1] ROOT-204/ROOT-204.vmdk was not found 2013-12-05 00:09:55,979 DEBUG [cloud.capacity.CapacityManagerImpl] (consoleproxy-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:09:56,007 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Exception while trying to start console proxy com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:09:56,047 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:09:56,074 WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-204-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:09:56,078 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 204, will recycle it and start a new one 2013-12-05 00:10:02,419 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-7:esxi2.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: The name 'ROOT-204' already exists. 2013-12-05 00:10:02,420 DEBUG [agent.transport.Request] (DirectAgent-7:null) Seq 1-1861287947: Processing: { Ans: , MgmtId: 146454222325, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: The name 'ROOT-204' already exists.","wait":0}}] } 2013-12-05 00:10:02,435 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (secstorage-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@1d6a89e1<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@1d6a89e1>), no need to delete from object in store ref table 2013-12-05 00:10:02,436 DEBUG [cloud.storage.VolumeManagerImpl] (secstorage-1:null) Unable to create Vol[204|vm=204|ROOT]:java.lang.RuntimeException: The name 'ROOT-204' already exists. 2013-12-05 00:10:02,436 INFO [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[204|vm=204|ROOT]:java.lang.RuntimeException: The name 'ROOT-204' already exists. 2013-12-05 00:10:02,704 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2 2013-12-05 00:10:02,731 WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-204-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-05 00:10:02,735 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 204, will recycle it and start a new one 2013-12-05 00:10:03,762 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-23:esxi2.ammscloud.com) clone volume from base image failed due to Exception: java.lang.RuntimeException java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:10:03,766 DEBUG [agent.transport.Request] (DirectAgent-23:null) Seq 1-1861287948: Processing: { Ans: , MgmtId: 146454222325, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException: The name 'ROOT-2' already exists.","wait":0}}] } 2013-12-05 00:10:03,780 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (consoleproxy-1:null) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@53e3d60f<mailto:org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@53e3d60f>), no need to delete from object in store ref table 2013-12-05 00:10:03,781 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:10:03,781 INFO [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create Vol[2|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists. 2013-12-05 00:10:04,006 DEBUG [cloud.capacity.CapacityManagerImpl] (consoleproxy-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 1 2013-12-05 00:10:04,020 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Exception while trying to start console proxy com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1 David Vosbury, Applications Systems Analyst Phone: (800) 893-9883 x7052 Mobile: (315) 679-7641 david.vosb...@systemsmadesimple.com<mailto:david.vosb...@systemsmadesimple.com> [cid:image001.jpg@01CCCA0D.B5979EA0] One of the Inc. 500 Fastest Growing Privately Held Companies in the U.S. This is a PRIVATE message. If you are not the intended recipient, please delete without copying and kindly advise us by telephone or e-mail of the mistake in delivery. NOTE: Regardless of content, this e-mail shall not operate to bind Systems Made Simple (SMS) to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
