Sailaja Mada created CLOUDSTACK-4477:
----------------------------------------
Summary: NPE & Failed to migrate DATA volume from zone wide
primary storage 1 to Zone wide primary storage2 on a mixed zone (Xen & VMWARE)
after adding Zone
Key: CLOUDSTACK-4477
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4477
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller
Reporter: Sailaja Mada
Priority: Critical
Steps:
1. Configure Adv zone with VMWARE and Xen Clusters in 307 ( Cluster level
Primary storage)
2. Upgrade to 4.2
3. Add 2nd VMWARE cluster with Cluster level primary storages
4. Add 2 zone wide primary storage's
5. Put cluster1 Primary storage into maintenance
6. Add new DATA volume ( which got deployed on Zone wide primary storage1)
7. Attach this volume to an instance(VMWARE)
8. Tried to Migrate this volume to Zone wide primary storage2
Observation:
NPE while migrating DATA volume from zone wide primary storage 1 to Zone wide
primary storage2 on a mixed zone (Xen & VMWARE) after adding Zone
2013-08-23 20:32:18,063 DEBUG [agent.transport.Request]
(StorageManager-Scavenger-1:null) Seq 7-1010569007: Executing: { Cmd , MgmtId:
187767034175903, via: 7, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":18,"mountPoint":"/cpg_vol/sailaja/legacyzwps1","path":"357c7fd7962231e99d6482134a65d435","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"c214298f-00cd-32c7-95a9-dd81d8b3dbd1","deviceId":0},"wait":0}}]
}
2013-08-23 20:32:18,063 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-376:null) Seq 7-1010569007: Executing request
2013-08-23 20:32:18,070 WARN [agent.manager.DirectAgentAttache]
(DirectAgent-376:null) Resource returned null answer!
2013-08-23 20:32:18,070 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-376:null) Seq 7-1010569007: Response Received:
2013-08-23 20:32:18,071 DEBUG [agent.transport.Request] (DirectAgent-376:null)
Seq 7-1010569007: Processing: { Ans: , MgmtId: 187767034175903, via: 7, Ver:
v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"Resource returned
null answer","wait":0}}] }
2013-08-23 20:32:18,071 DEBUG [agent.manager.AgentAttache]
(DirectAgent-376:null) Seq 7-1010569007: No more commands found
2013-08-23 20:32:18,071 DEBUG [agent.transport.Request]
(StorageManager-Scavenger-1:null) Seq 7-1010569007: Received: { Ans: , MgmtId:
187767034175903, via: 7, Ver: v1, Flags: 110, { Answer } }
2013-08-23 20:32:18,072 INFO [cloud.template.TemplateManagerImpl]
(StorageManager-Scavenger-1:null) Will retry evicte template: CentOS
5.3(64-bit) no GUI (vSphere) from storage pool: null
2013-08-23 20:32:18,098 DEBUG [cloud.storage.StorageManagerImpl]
(StorageManager-Scavenger-1:null) Secondary storage garbage collector found 0
volumes to cleanup on volume_store_ref for store:
263abbf4-2997-4dd7-b1ab-9b2abc3a3f2b
2013-08-23 20:32:18,116 DEBUG [cache.allocator.StorageCacheRandomAllocator]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Can't find
staging storage in zone: 2
2013-08-23 20:32:18,546 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) copy
object failed:
java.lang.NullPointerException
at
com.cloud.hypervisor.guru.VMwareGuru.resolveNameInGuid(VMwareGuru.java:403)
at
com.cloud.hypervisor.guru.VMwareGuru.getCommandHostDelegation(VMwareGuru.java:357)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.hypervisor.HypervisorGuruManagerImpl.getGuruProcessedCommandTargetHost(HypervisorGuruManagerImpl.java:64)
at
org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:95)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:317)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:406)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:791)
at
com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2239)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-23 20:32:18,567 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) copy failed
com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:231)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:317)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:406)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:791)
at
com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2239)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-23 20:32:18,576 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ])
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@3ad84031), no need
to delete from object in store ref table
2013-08-23 20:32:18,589 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ])
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@647e4eaf), no need
to delete from object in store ref table
2013-08-23 20:32:18,653 DEBUG [agent.transport.Request]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq
13-1361381061: Sending { Cmd , MgmtId: 187767034175903, via: 13, Ver: v1,
Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7262b35e-920e-495e-8a85-7a5c31e5903d","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f528f6ca-31e0-333a-88a1-93acce50e381","id":208,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/legacyzwps2","port":2049}},"name":"sailajadata1","size":5368709120,"path":"bfcdce36b632440d9012c93f16bcea24","volumeId":109,"vmName":"i-4-7-VM","accountId":4,"id":109,"hypervisorType":"VMware"}},"wait":0}}]
}
2013-08-23 20:32:18,654 DEBUG [agent.transport.Request]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq
13-1361381061: Executing: { Cmd , MgmtId: 187767034175903, via: 13, Ver: v1,
Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7262b35e-920e-495e-8a85-7a5c31e5903d","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f528f6ca-31e0-333a-88a1-93acce50e381","id":208,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/legacyzwps2","port":2049}},"name":"sailajadata1","size":5368709120,"path":"bfcdce36b632440d9012c93f16bcea24","volumeId":109,"vmName":"i-4-7-VM","accountId":4,"id":109,"hypervisorType":"VMware"}},"wait":0}}]
}
2013-08-23 20:32:18,654 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-462:null) Seq 13-1361381061: Executing request
2013-08-23 20:32:18,803 DEBUG [xen.resource.XenServerStorageProcessor]
(DirectAgent-462:null) Failed to delete volume
The uuid you supplied was invalid.
at com.xensource.xenapi.Types.checkResponse(Types.java:927)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VDI.getByUuid(VDI.java:326)
at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.deleteVolume(XenServerStorageProcessor.java:452)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:114)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:53)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:617)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-23 20:32:18,839 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-462:null) Seq 13-1361381061: Response Received:
2013-08-23 20:32:18,840 DEBUG [agent.transport.Request] (DirectAgent-462:null)
Seq 13-1361381061: Processing: { Ans: , MgmtId: 187767034175903, via: 13, Ver:
v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"The
uuid you supplied was invalid.","wait":0}}] }
2013-08-23 20:32:18,840 DEBUG [agent.transport.Request]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq
13-1361381061: Received: { Ans: , MgmtId: 187767034175903, via: 13, Ver: v1,
Flags: 10, { Answer } }
2013-08-23 20:32:18,850 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ])
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@fd77e71), no need
to delete from object in store ref table
2013-08-23 20:32:18,850 ERROR [cloud.storage.VolumeManagerImpl]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) migrate
volume failed:com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException
2013-08-23 20:32:18,851 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Failed to
migrate volume:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:208] is
unreachable: migrate volume failed:
com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
at
com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2244)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-23 20:32:18,854 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Complete
async job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to migrate volume
2013-08-23 20:32:19,792 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2013-08-23 20:32:19,815 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Zone 2 is ready to launch secondary storage VM
2013-08-23 20:32:20,100 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is ready to launch console proxy
2013-08-23 20:32:20,110 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 2 is ready to launch console proxy
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
