Sailaja Mada created CLOUDSTACK-5027:
----------------------------------------
Summary: [VMWARE] Failed to Deploy instance when primary storage
of a different zone are in maintenance mode
Key: CLOUDSTACK-5027
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5027
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, VMware
Affects Versions: 4.2.1
Reporter: Sailaja Mada
Priority: Critical
Attachments: failureloigs.rar
Steps:
1. Configure 2 vMWARE zones each with Zone wide primary storages
2. Moved Zone 1 primary storages into maintenance mode
3. Tried to deploy VM on second VMWARE zone which has Primary storage up state.
Observation:
[VMWARE] Failed to Deploy instance when primary storage of a different zone are
in maintenance mode
2013-11-03 10:21:07,088 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Checking
if we need to prepare 1 volumes for VM[DomainRouter|r-30-VM]
2013-11-03 10:21:07,099 DEBUG [storage.image.TemplateDataFactoryImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) template 8
is already in store:1, type:Image
2013-11-03 10:21:07,104 DEBUG [storage.datastore.PrimaryDataStoreImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Not found
(templateId:8poolId:6) in template_spool_ref, persisting it
2013-11-03 10:21:07,121 DEBUG [storage.image.TemplateDataFactoryImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) template 8
is already in store:6, type:Primary
2013-11-03 10:21:07,123 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Found
template routing-8 in storage pool 6 with VMTemplateStoragePool id: 28
2013-11-03 10:21:07,187 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-3:null) SeqA 10-30340: Processing Seq 10-30340: { Cmd ,
MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":17,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-11-03 10:21:07,331 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Acquire
lock on VMTemplateStoragePool 28 with timeout 3600 seconds
2013-11-03 10:21:07,333 INFO [storage.volume.VolumeServiceImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) lock is
acquired for VMTemplateStoragePool 28
2013-11-03 10:21:07,341 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-3:null) SeqA 10-30340: Sending Seq 10-30340: { Ans: ,
MgmtId: 94838926819810, via: 10, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-11-03 10:21:07,373 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) copyAsync
inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2013-11-03 10:21:07,387 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) copy
object failed:
java.lang.NullPointerException
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:411)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:58)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:446)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2567)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2631)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2764)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1896)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1996)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1974)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:189)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2043)
at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1948)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2129)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3425)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2985)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2971)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-11-03 10:21:07,479 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) copy failed
com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:231)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:411)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:58)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:446)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2567)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2631)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2764)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1896)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1996)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:189)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2043)
at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1948)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2129)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3425)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2985)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2971)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-11-03 10:21:07,615 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Cleaning
up resources for the vm VM[DomainRouter|r-30-VM] in Starting state
2013-11-03 10:21:07,616 DEBUG [agent.transport.Request]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Seq
9-894185288: Sending { Cmd , MgmtId: 94838926819810, via: 9, Ver: v1, Flags:
100111,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"r-30-VM","wait":0}}]
}
2013-11-03 10:21:07,617 DEBUG [agent.transport.Request]
(Job-Executor-17:job-155 = [ d4cadcb1-7e06-4ee2-9d08-0585964fc934 ]) Seq
9-894185288: Executing: { Cmd , MgmtId: 94838926819810, via: 9, Ver: v1,
Flags: 100111,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"r-30-VM","wait":0}}]
}
2013-11-03 10:21:07,646 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-148:null) Seq 9-894185288: Executing request
2013-11-03 10:21:07,647 INFO [vmware.resource.VmwareResource]
(DirectAgent-148:10.102.192.18) Executing resource StopCommand:
{"isProxy":false,"executeInSequence":true,"vmName":"r-30-VM","wait":0}
2013-11-03 10:21:07,676 DEBUG [vmware.mo.HostMO]
(DirectAgent-148:10.102.192.18) find VM r-30-VM on host
--
This message was sent by Atlassian JIRA
(v6.1#6144)
