[
https://issues.apache.org/jira/browse/CLOUDSTACK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103408#comment-14103408
]
ASF subversion and git services commented on CLOUDSTACK-4587:
-------------------------------------------------------------
Commit 8bb4022f3729154f85036c4a7f55e2de783e4909 in cloudstack's branch
refs/heads/master from [~likithas]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=8bb4022 ]
CLOUDSTACK-4587. System VMs fail to start when the primary storage that has the
System VMs is put into maintenance.
During VM start while configuring its disk devices, obtain the matching disk
for a volume in storage
using both the volume's path and volume's datastore information.
> VM is failing to deploy on a Legacy zone after adding zone wide primary
> storage and moving cluster wide primary storage to maintenance mode
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4587
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4587
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Storage Controller
> Affects Versions: 4.2.1
> Reporter: Sailaja Mada
> Assignee: Likitha Shetty
> Fix For: 4.4.0
>
> Attachments: failureloigs.rar, logs.rar, vmdeploylogs.rar
>
>
> Steps:
> 1. Upgraded from 307 to 4.2 using VMWARE Cluster with Cluster level primary
> storage
> 2. Add Zone wide primary storage after upgrade
> 3. Move the cluster level scope storage to Maintenance mode
> 4. Try to deploy new VM.
> Observation:
> VM is failing to deploy on a Legacy zone after adding zone wide primary
> storage and moving cluster wide primary storage to maintenance mode
> 2013-09-01 14:21:10,323 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Network
> id=207 is already implemented
> 2013-09-01 14:21:10,354 DEBUG [network.guru.PodBasedNetworkGuru]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> Allocated a nic
> NicProfile[119-35-4a8f547b-7bff-4b3f-ba10-b0146af4d1bb-10.102.195.111-null
> for VM[DomainRouter|r-35-VM]
> 2013-09-01 14:21:10,390 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> Checking if we need to prepare 1 volumes for VM[DomainRouter|r-35-VM]
> 2013-09-01 14:21:10,400 DEBUG [storage.image.TemplateDataFactoryImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> template 203 is already in store:2, type:Image
> 2013-09-01 14:21:10,409 DEBUG [storage.image.TemplateDataFactoryImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> template 203 is already in store:203, type:Primary
> 2013-09-01 14:21:10,410 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Found
> template 203-2-3cfe22ca-3a33-39a0-bf5e-0dab154869fd in storage pool 203 with
> VMTemplateStoragePool id: 11
> 2013-09-01 14:21:10,421 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Acquire
> lock on VMTemplateStoragePool 11 with timeout 3600 seconds
> 2013-09-01 14:21:10,423 INFO [storage.volume.VolumeServiceImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) lock is
> acquired for VMTemplateStoragePool 11
> 2013-09-01 14:21:10,431 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
> 2013-09-01 14:21:10,519 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) copy
> object failed:
> java.lang.NullPointerException
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:411)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:440)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:569)
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2536)
> at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1972)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1950)
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188)
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2034)
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1939)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2120)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-09-01 14:21:10,520 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) copy
> failed
> com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:231)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:411)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:440)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:569)
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2536)
> at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1972)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1950)
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188)
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2034)
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1939)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2120)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-09-01 14:21:10,525 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-152:null) Seq 5-1348865999: Response Received:
> 2013-09-01 14:21:10,525 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 5-1348865999: Received: { Ans: , MgmtId:
> 227594284004867, via: 5, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2013-09-01 14:21:10,530 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-172:null) Seq 6-1919947690: Executing request
> 2013-09-01 14:21:10,558 INFO [storage.volume.VolumeServiceImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> releasing lock for VMTemplateStoragePool 11
> 2013-09-01 14:21:10,558 WARN [utils.db.Merovingian2]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Was
> unable to find lock for the key template_spool_ref11 and thread id 595423976
> 2013-09-01 14:21:10,558 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable
> to create Vol[98|vm=35|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> 2013-09-01 14:21:10,558 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable
> to contact resource.
> 2013-09-01 14:21:10,558 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable
> to create Vol[98|vm=35|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> 2013-09-01 14:21:10,558 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Unable
> to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:203]
> is unreachable: Unable to create
> Vol[98|vm=35|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2544)
> at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2592)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1972)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1950)
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188)
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2034)
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1939)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2120)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-09-01 14:21:10,592 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-172:10.102.192.20) Datastore summary info, storageId:
> 55e6d095-7e22-3f4a-b533-874a9fede8df, localPath: /cpg_vol/sailaja/vmwareps1,
> poolType: NetworkFilesystem, capacity: 879609303040, free: 376848912384,
> used: 502760390656
> 2013-09-01 14:21:10,679 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-172:null) Seq 6-1919947690: Response Received:
> 2013-09-01 14:21:10,679 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 6-1919947690: Received: { Ans: , MgmtId:
> 227594284004867, via: 6, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2013-09-01 14:21:10,685 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-271:null) Seq 6-1919947691: Executing request
> 2013-09-01 14:21:10,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ])
> Cleaning up resources for the vm VM[DomainRouter|r-35-VM] in Starting state
> 2013-09-01 14:21:10,696 DEBUG [agent.transport.Request]
> (Job-Executor-119:job-199 = [ 4837d9ec-d365-4653-960c-83fa1dc4fa87 ]) Seq
> 6-1919947692: Sending { Cmd , MgmtId: 227594284004867, via: 6, Ver: v1,
> Flags: 100111,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"r-35-VM","wait":0}}]
> }
--
This message was sent by Atlassian JIRA
(v6.2#6252)
