[
https://issues.apache.org/jira/browse/CLOUDSTACK-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
edison su reassigned CLOUDSTACK-4477:
-------------------------------------
Assignee: edison su
> NPE & Failed to migrate DATA volume from zone wide primary storage 1 to Zone
> wide primary storage2 on a mixed zone (Xen & VMWARE) after adding Zone
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4477
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4477
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Storage Controller
> Reporter: Sailaja Mada
> Assignee: edison su
> Priority: Critical
> Fix For: 4.2.1
>
> Attachments: migratelogs.rar
>
>
> Steps:
> 1. Configure Adv zone with VMWARE and Xen Clusters in 307 ( Cluster level
> Primary storage)
> 2. Upgrade to 4.2
> 3. Add 2nd VMWARE cluster with Cluster level primary storages
> 4. Add 2 zone wide primary storage's
> 5. Put cluster1 Primary storage into maintenance
> 6. Add new DATA volume ( which got deployed on Zone wide primary storage1)
> 7. Attach this volume to an instance(VMWARE)
> 8. Tried to Migrate this volume to Zone wide primary storage2
> Observation:
> NPE while migrating DATA volume from zone wide primary storage 1 to Zone wide
> primary storage2 on a mixed zone (Xen & VMWARE) after adding Zone
> 2013-08-23 20:32:18,063 DEBUG [agent.transport.Request]
> (StorageManager-Scavenger-1:null) Seq 7-1010569007: Executing: { Cmd ,
> MgmtId: 187767034175903, via: 7, Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":18,"mountPoint":"/cpg_vol/sailaja/legacyzwps1","path":"357c7fd7962231e99d6482134a65d435","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"c214298f-00cd-32c7-95a9-dd81d8b3dbd1","deviceId":0},"wait":0}}]
> }
> 2013-08-23 20:32:18,063 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-376:null) Seq 7-1010569007: Executing request
> 2013-08-23 20:32:18,070 WARN [agent.manager.DirectAgentAttache]
> (DirectAgent-376:null) Resource returned null answer!
> 2013-08-23 20:32:18,070 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-376:null) Seq 7-1010569007: Response Received:
> 2013-08-23 20:32:18,071 DEBUG [agent.transport.Request]
> (DirectAgent-376:null) Seq 7-1010569007: Processing: { Ans: , MgmtId:
> 187767034175903, via: 7, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"Resource returned
> null answer","wait":0}}] }
> 2013-08-23 20:32:18,071 DEBUG [agent.manager.AgentAttache]
> (DirectAgent-376:null) Seq 7-1010569007: No more commands found
> 2013-08-23 20:32:18,071 DEBUG [agent.transport.Request]
> (StorageManager-Scavenger-1:null) Seq 7-1010569007: Received: { Ans: ,
> MgmtId: 187767034175903, via: 7, Ver: v1, Flags: 110, { Answer } }
> 2013-08-23 20:32:18,072 INFO [cloud.template.TemplateManagerImpl]
> (StorageManager-Scavenger-1:null) Will retry evicte template: CentOS
> 5.3(64-bit) no GUI (vSphere) from storage pool: null
> 2013-08-23 20:32:18,098 DEBUG [cloud.storage.StorageManagerImpl]
> (StorageManager-Scavenger-1:null) Secondary storage garbage collector found 0
> volumes to cleanup on volume_store_ref for store:
> 263abbf4-2997-4dd7-b1ab-9b2abc3a3f2b
> 2013-08-23 20:32:18,116 DEBUG [cache.allocator.StorageCacheRandomAllocator]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Can't
> find staging storage in zone: 2
> 2013-08-23 20:32:18,546 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) copy
> object failed:
> java.lang.NullPointerException
> at
> com.cloud.hypervisor.guru.VMwareGuru.resolveNameInGuid(VMwareGuru.java:403)
> at
> com.cloud.hypervisor.guru.VMwareGuru.getCommandHostDelegation(VMwareGuru.java:357)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.hypervisor.HypervisorGuruManagerImpl.getGuruProcessedCommandTargetHost(HypervisorGuruManagerImpl.java:64)
> at
> org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:95)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:317)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:406)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:791)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2239)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,567 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) copy
> failed
> com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:231)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:317)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:406)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:791)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2239)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,576 WARN
> [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-31:job-137 = [
> 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Unsupported data object (VOLUME,
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@3ad84031), no
> need to delete from object in store ref table
> 2013-08-23 20:32:18,589 WARN
> [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-31:job-137 = [
> 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Unsupported data object (VOLUME,
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@647e4eaf), no
> need to delete from object in store ref table
> 2013-08-23 20:32:18,653 DEBUG [agent.transport.Request]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq
> 13-1361381061: Sending { Cmd , MgmtId: 187767034175903, via: 13, Ver: v1,
> Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7262b35e-920e-495e-8a85-7a5c31e5903d","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f528f6ca-31e0-333a-88a1-93acce50e381","id":208,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/legacyzwps2","port":2049}},"name":"sailajadata1","size":5368709120,"path":"bfcdce36b632440d9012c93f16bcea24","volumeId":109,"vmName":"i-4-7-VM","accountId":4,"id":109,"hypervisorType":"VMware"}},"wait":0}}]
> }
> 2013-08-23 20:32:18,654 DEBUG [agent.transport.Request]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq
> 13-1361381061: Executing: { Cmd , MgmtId: 187767034175903, via: 13, Ver: v1,
> Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7262b35e-920e-495e-8a85-7a5c31e5903d","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f528f6ca-31e0-333a-88a1-93acce50e381","id":208,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/legacyzwps2","port":2049}},"name":"sailajadata1","size":5368709120,"path":"bfcdce36b632440d9012c93f16bcea24","volumeId":109,"vmName":"i-4-7-VM","accountId":4,"id":109,"hypervisorType":"VMware"}},"wait":0}}]
> }
> 2013-08-23 20:32:18,654 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-462:null) Seq 13-1361381061: Executing request
> 2013-08-23 20:32:18,803 DEBUG [xen.resource.XenServerStorageProcessor]
> (DirectAgent-462:null) Failed to delete volume
> The uuid you supplied was invalid.
> at com.xensource.xenapi.Types.checkResponse(Types.java:927)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VDI.getByUuid(VDI.java:326)
> at
> com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.deleteVolume(XenServerStorageProcessor.java:452)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:114)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:53)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:617)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,839 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-462:null) Seq 13-1361381061: Response Received:
> 2013-08-23 20:32:18,840 DEBUG [agent.transport.Request]
> (DirectAgent-462:null) Seq 13-1361381061: Processing: { Ans: , MgmtId:
> 187767034175903, via: 13, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"The uuid you
> supplied was invalid.","wait":0}}] }
> 2013-08-23 20:32:18,840 DEBUG [agent.transport.Request]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq
> 13-1361381061: Received: { Ans: , MgmtId: 187767034175903, via: 13, Ver: v1,
> Flags: 10, { Answer } }
> 2013-08-23 20:32:18,850 WARN
> [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-31:job-137 = [
> 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Unsupported data object (VOLUME,
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@fd77e71), no
> need to delete from object in store ref table
> 2013-08-23 20:32:18,850 ERROR [cloud.storage.VolumeManagerImpl]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) migrate
> volume failed:com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> 2013-08-23 20:32:18,851 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Failed
> to migrate volume:
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:208]
> is unreachable: migrate volume failed:
> com.cloud.utils.exception.CloudRuntimeException:
> java.lang.NullPointerException
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2244)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,854 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-31:job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Complete
> async job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ], jobStatus: 2,
> resultCode: 530, result: Error Code: 530 Error text: Failed to migrate volume
> 2013-08-23 20:32:19,792 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
> 2013-08-23 20:32:19,815 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Zone 2 is ready to launch secondary storage VM
> 2013-08-23 20:32:20,100 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> 2013-08-23 20:32:20,110 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Zone 2 is ready to launch console proxy
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
