[
https://issues.apache.org/jira/browse/CLOUDSTACK-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudha Ponnaganti updated CLOUDSTACK-4090:
-----------------------------------------
Summary: [Upgrade] System VM's are failed to start on a new zone(VMWARE
Nexus Zone) which is created after upgrade (was: System VM's are failed to
start on a new zone(VMWARE Nexus Zone) which is created after upgrade )
> [Upgrade] System VM's are failed to start on a new zone(VMWARE Nexus Zone)
> which is created after upgrade
> ----------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4090
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4090
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: VMware
> Affects Versions: 4.2.0
> Reporter: Sailaja Mada
> Priority: Blocker
> Fix For: 4.2.0
>
> Attachments: apilog.log, Bforeupgraedecloud-backup.dmp,
> postupgradecloud-backup.dmp
>
>
> Steps:
> 1. Upgraded from 305 Patch A to 4.2 with Xenserver Zone
> 2. Enabled Nexus global config value
> 3. Configure one more Adv zone with VMWARE cluster with Nexus 1000v Switch
> enabled
> 4. Seeded secondary storage with new 4.2 vMWARE template (vh7)
> Physical network 1 :
> Mgmt Traffic label name - vSwitch0,,vmwaresvs
> Physical network 1 :
> Public & Guest Traffic label name - nexuspp91,,nexusdvs
> Observation:System VM's are failed to start on a new zone(VMWARE Nexus Zone)
> which is created after upgrade
> After upgrade I have enabled Nexus vSwtich and configured zone with Nexus
> cluster. Now system VM’s are coming up now : I I am using vh7 template) -
> 2013-08-05 21:01:52,084 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-08-05 21:01:52,091 DEBUG [storage.motion.AncientDataMotionStrategy]
> (consoleproxy-1:null) copyAsync inspecting src type TEMPLATE copyAsync
> inspecting dest type TEMPLATE
> 2013-08-05 21:01:52,174 DEBUG [agent.transport.Request] (consoleproxy-1:null)
> Seq 8-268894239: Sending { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
> Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
> Template
> (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
> Template
> (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
> }
> 2013-08-05 21:01:52,175 DEBUG [agent.transport.Request] (consoleproxy-1:null)
> Seq 8-268894239: Executing: { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
> Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
> Template
> (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
> Template
> (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
> }
> 2013-08-05 21:01:52,186 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-204:null) Seq 8-268894239: Executing request
> 2013-08-05 21:01:52,318 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 5 routers to update status.
> 2013-08-05 21:01:52,323 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-08-05 21:01:52,330 INFO [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Template routing-8 is not setup yet, setup
> template from secondary storage with uuid name:
> 431f50e387b738feabae43f819e69bbf
> 2013-08-05 21:01:52,345 INFO [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Executing copyTemplateFromSecondaryToPrimary.
> secondaryStorage: nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/,
> templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
> 2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.5e4f2a84
> 2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.d778a15
> 2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.6b16fc2f
> 2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.8558063
> 2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.5b202dbe
> 2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.20c1ddf
> 2013-08-05 21:01:52,382 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.3a381db8
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.5223dd26
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.2c371333
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount:
> /var/lib/cloud/mnt/VM/7674049379768.5ad80648
> 2013-08-05 21:01:52,383 WARN [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create a mount point
> 2013-08-05 21:01:52,384 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-204:10.102.192.20) Unable to create mount point for
> nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/
> 2013-08-05 21:01:52,384 INFO [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Secondary storage mount point: /mnt/sec
> 2013-08-05 21:01:52,388 INFO [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Executing command: tar --no-same-owner -xf
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:52,388 DEBUG [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Executing: tar --no-same-owner -xf
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:53,873 WARN [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Exception: tar --no-same-owner -xf
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> java.io.IOException: Cannot run program "tar" (in directory
> "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or
> directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
> at com.cloud.utils.script.Script.execute(Script.java:183)
> at com.cloud.utils.script.Script.execute(Script.java:161)
> at
> com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:145)
> at
> com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or
> directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
> at java.lang.ProcessImpl.start(ProcessImpl.java:81)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
> ... 16 more
> 2013-08-05 21:01:53,878 ERROR [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Unable to unpack snapshot OVA file at:
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:53,879 ERROR [storage.resource.VmwareStorageProcessor]
> (DirectAgent-204:10.102.192.20) Unable to execute
> PrimaryStorageDownloadCommand due to exception
> java.lang.Exception: Unable to unpack snapshot OVA file at:
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> at
> com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:149)
> at
> com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-05 21:01:53,880 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-204:null) Seq 8-268894239: Response Received:
> 2013-08-05 21:01:53,880 DEBUG [agent.transport.Request]
> (DirectAgent-204:null) Seq 8-268894239: Processing: { Ans: , MgmtId:
> 7674049379768, via: 8, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unable
> to execute PrimaryStorageDownloadCommand due to exception","wait":0}}] }
> 2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (consoleproxy-1:null)
> Seq 8-268894239: Received: { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1,
> Flags: 10, { CopyCmdAnswer } }
> 2013-08-05 21:01:53,906 INFO [storage.volume.VolumeServiceImpl]
> (consoleproxy-1:null) releasing lock for VMTemplateStoragePool 11
> 2013-08-05 21:01:53,906 WARN [utils.db.Merovingian2] (consoleproxy-1:null)
> Was unable to find lock for the key template_spool_ref11 and thread id
> 144163998
> 2013-08-05 21:01:53,907 DEBUG [cloud.storage.VolumeManagerImpl]
> (consoleproxy-1:null) Unable to create Vol[37|vm=21|ROOT]:Unable to execute
> PrimaryStorageDownloadCommand due to exception
> 2013-08-05 21:01:53,922 INFO [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:202]
> is unreachable: Unable to create Vol[37|vm=21|ROOT]:Unable to execute
> PrimaryStorageDownloadCommand due to exception
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
> at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:617)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1665)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-05 21:01:53,953 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Cleaning up resources for the vm
> VM[ConsoleProxy|v-21-VM] in Starting state
> 2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null)
> Seq 8-268894240: Sending { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
> Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}]
> }
> 2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null)
> Seq 8-268894240: Executing: { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
> Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}]
> }
> 2013-08-05 21:01:53,990 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-163:null) Seq 8-268894240: Executing request
> 2013-08-05 21:01:53,991 INFO [vmware.resource.VmwareResource]
> (DirectAgent-163:10.102.192.20) Executing resource StopCommand:
> {"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}
> mysql> select * from physical_network_traffic_types;
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> | id | uuid | physical_network_id |
> traffic_type | xen_network_label | kvm_network_label | vmware_network_label |
> simulator_network_label | ovm_network_label | vlan | lxc_network_label |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> | 1 | bb74693a-b32a-499d-94b6-397e871f96c7 | 200 |
> Management | NULL | NULL | NULL |
> NULL | NULL | NULL | cloudbr0 |
> | 2 | 58a43a6a-f4f1-4936-b140-a3429d7b95ee | 200 | Public
> | NULL | NULL | NULL | NULL
> | NULL | NULL | cloudbr0 |
> | 3 | 22e56281-dfaa-4bb0-aedc-d4866d4c118a | 200 | Guest
> | NULL | NULL | NULL | NULL
> | NULL | NULL | cloudbr0 |
> | 4 | dbe80dd9-c70c-4b38-bfb7-4e8bfeca0768 | 200 | Storage
> | NULL | NULL | NULL | NULL
> | NULL | NULL | cloudbr0 |
> | 5 | 63c8710b-b644-4a6d-bb58-5886176c0688 | 202 | Public
> | NULL | NULL | nexuspp91,,nexusdvs | NULL
> | NULL | NULL | cloudbr0 |
> | 6 | 6ba2196e-fc1a-4822-ba4d-ae2a8af2df53 | 202 | Guest
> | NULL | NULL | nexuspp91,,nexusdvs | NULL
> | NULL | NULL | cloudbr0 |
> | 7 | 4397e50a-4d30-4f19-b75c-ce4f3d070cb8 | 201 |
> Management | NULL | NULL | vSwitch0,,vmwaresvs |
> NULL | NULL | NULL | cloudbr0 |
> | 8 | 261b29ef-521b-4a6a-8b91-2328431f6bb5 | 201 | Storage
> | NULL | NULL | vSwitch0,,vmwaresvs | NULL
> | NULL | NULL | cloudbr0 |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> 8 rows in set (0.00 sec)
> mysql>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
