Sailaja Mada created CLOUDSTACK-4090:
----------------------------------------
Summary: System VM's are failed to start after upgrade with VMWARE
Nexus Zone
Key: CLOUDSTACK-4090
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4090
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: VMware
Affects Versions: 4.2.0
Reporter: Sailaja Mada
Priority: Blocker
Steps:
1. Upgraded from 305 Patch A to 4.2 with Xenserver Zone
2. Enabled Nexus global config value
3. Configure one more Adv zone with VMWARE cluster with Nexus 1000v Switch
enabled
Physical network 1 :
Mgmt Traffic label name - vSwitch0,,vmwaresvs
Physical network 1 :
Public & Guest Traffic label name - nexuspp91,,nexusdvs
Observation:
System VM's are failed to start after upgrade with VMWARE Nexus Zone
After upgrade I have enabled Nexus vSwtich and configured zone with Nexus
cluster. Now system VM’s are coming up now : I I am using vh7 template) -
2013-08-05 21:01:52,084 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-08-05 21:01:52,091 DEBUG [storage.motion.AncientDataMotionStrategy]
(consoleproxy-1:null) copyAsync inspecting src type TEMPLATE copyAsync
inspecting dest type TEMPLATE
2013-08-05 21:01:52,174 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 8-268894239: Sending { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template
(vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template
(vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
}
2013-08-05 21:01:52,175 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 8-268894239: Executing: { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template
(vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template
(vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
}
2013-08-05 21:01:52,186 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-204:null) Seq 8-268894239: Executing request
2013-08-05 21:01:52,318 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 5 routers to update status.
2013-08-05 21:01:52,323 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-08-05 21:01:52,330 INFO [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Template routing-8 is not setup yet, setup
template from secondary storage with uuid name: 431f50e387b738feabae43f819e69bbf
2013-08-05 21:01:52,345 INFO [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Executing copyTemplateFromSecondaryToPrimary.
secondaryStorage: nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/,
templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.5e4f2a84
2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.d778a15
2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.6b16fc2f
2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.8558063
2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.5b202dbe
2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.20c1ddf
2013-08-05 21:01:52,382 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.3a381db8
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.5223dd26
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.2c371333
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount:
/var/lib/cloud/mnt/VM/7674049379768.5ad80648
2013-08-05 21:01:52,383 WARN [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create a mount point
2013-08-05 21:01:52,384 ERROR [vmware.manager.VmwareManagerImpl]
(DirectAgent-204:10.102.192.20) Unable to create mount point for
nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/
2013-08-05 21:01:52,384 INFO [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Secondary storage mount point: /mnt/sec
2013-08-05 21:01:52,388 INFO [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Executing command: tar --no-same-owner -xf
/mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:52,388 DEBUG [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Executing: tar --no-same-owner -xf
/mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:53,873 WARN [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Exception: tar --no-same-owner -xf
/mnt/sec/template/tmpl/1/8/routing-8.ova
java.io.IOException: Cannot run program "tar" (in directory
"/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or
directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at com.cloud.utils.script.Script.execute(Script.java:183)
at com.cloud.utils.script.Script.execute(Script.java:161)
at
com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:145)
at
com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or
directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
... 16 more
2013-08-05 21:01:53,878 ERROR [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Unable to unpack snapshot OVA file at:
/mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:53,879 ERROR [storage.resource.VmwareStorageProcessor]
(DirectAgent-204:10.102.192.20) Unable to execute PrimaryStorageDownloadCommand
due to exception
java.lang.Exception: Unable to unpack snapshot OVA file at:
/mnt/sec/template/tmpl/1/8/routing-8.ova
at
com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:149)
at
com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-05 21:01:53,880 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-204:null) Seq 8-268894239: Response Received:
2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (DirectAgent-204:null)
Seq 8-268894239: Processing: { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unable
to execute PrimaryStorageDownloadCommand due to exception","wait":0}}] }
2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 8-268894239: Received: { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1,
Flags: 10, { CopyCmdAnswer } }
2013-08-05 21:01:53,906 INFO [storage.volume.VolumeServiceImpl]
(consoleproxy-1:null) releasing lock for VMTemplateStoragePool 11
2013-08-05 21:01:53,906 WARN [utils.db.Merovingian2] (consoleproxy-1:null) Was
unable to find lock for the key template_spool_ref11 and thread id 144163998
2013-08-05 21:01:53,907 DEBUG [cloud.storage.VolumeManagerImpl]
(consoleproxy-1:null) Unable to create Vol[37|vm=21|ROOT]:Unable to execute
PrimaryStorageDownloadCommand due to exception
2013-08-05 21:01:53,922 INFO [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:202] is
unreachable: Unable to create Vol[37|vm=21|ROOT]:Unable to execute
PrimaryStorageDownloadCommand due to exception
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:617)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1665)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-05 21:01:53,953 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleaning up resources for the vm VM[ConsoleProxy|v-21-VM]
in Starting state
2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 8-268894240: Sending { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}]
}
2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 8-268894240: Executing: { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}]
}
2013-08-05 21:01:53,990 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-163:null) Seq 8-268894240: Executing request
2013-08-05 21:01:53,991 INFO [vmware.resource.VmwareResource]
(DirectAgent-163:10.102.192.20) Executing resource StopCommand:
{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}
mysql> select * from physical_network_traffic_types;
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
| id | uuid | physical_network_id |
traffic_type | xen_network_label | kvm_network_label | vmware_network_label |
simulator_network_label | ovm_network_label | vlan | lxc_network_label |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
| 1 | bb74693a-b32a-499d-94b6-397e871f96c7 | 200 | Management
| NULL | NULL | NULL | NULL
| NULL | NULL | cloudbr0 |
| 2 | 58a43a6a-f4f1-4936-b140-a3429d7b95ee | 200 | Public
| NULL | NULL | NULL | NULL
| NULL | NULL | cloudbr0 |
| 3 | 22e56281-dfaa-4bb0-aedc-d4866d4c118a | 200 | Guest
| NULL | NULL | NULL | NULL
| NULL | NULL | cloudbr0 |
| 4 | dbe80dd9-c70c-4b38-bfb7-4e8bfeca0768 | 200 | Storage
| NULL | NULL | NULL | NULL
| NULL | NULL | cloudbr0 |
| 5 | 63c8710b-b644-4a6d-bb58-5886176c0688 | 202 | Public
| NULL | NULL | nexuspp91,,nexusdvs | NULL
| NULL | NULL | cloudbr0 |
| 6 | 6ba2196e-fc1a-4822-ba4d-ae2a8af2df53 | 202 | Guest
| NULL | NULL | nexuspp91,,nexusdvs | NULL
| NULL | NULL | cloudbr0 |
| 7 | 4397e50a-4d30-4f19-b75c-ce4f3d070cb8 | 201 | Management
| NULL | NULL | vSwitch0,,vmwaresvs | NULL
| NULL | NULL | cloudbr0 |
| 8 | 261b29ef-521b-4a6a-8b91-2328431f6bb5 | 201 | Storage
| NULL | NULL | vSwitch0,,vmwaresvs | NULL
| NULL | NULL | cloudbr0 |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
8 rows in set (0.00 sec)
mysql>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
