sadhu suresh created CLOUDSTACK-4423:
----------------------------------------
Summary: ceph:migrate instance form one primary to another primary
failing
Key: CLOUDSTACK-4423
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4423
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Reporter: sadhu suresh
Priority: Critical
1.deploy advanced zone with kvm cluster(single cluster with ubuntu host+nfs as
primary storage)
2.add RBD based primary storage
3.create compute offering and create a instance using this offering
4.stop the running VM which was created in step 3
5.perform migrate instance from one primary to another primary
Actual result:
Migrate instance is failing with below error
"failed to migration: com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:2] is unreachable: migrate volume failed:
com.cloud.utils.exception.CloudRuntimeException: Failed to copy
cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
content of management log:
2013-08-21 04:25:07,080 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-25:null) submit async job-21 = [
92188256-af48-4878-8f5d-8f1d1a23ab65 ], details: AsyncJobVO {id:21, userId: 2,
accountId: 2, sessionKey: null, instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdOriginator: null,
cmdInfo:
{"response":"json","sessionkey":"Dle2oW4Rjxm7blpoMLnWiHE/ZVM\u003d","virtualmachineid":"694963ca-4ed6-4815-af80-5a77135b97c9","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","httpmethod":"GET","_":"1377019844223","ctxAccountId":"2","ctxStartEventId":"86"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 7175246184473,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-08-21 04:25:07,084 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-21 = [
92188256-af48-4878-8f5d-8f1d1a23ab65 ]
2013-08-21 04:25:07,110 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state
transitted from :Stopped to Migrating with event: StorageMigrationRequestedvm's
original host id: 1 new host id: null host id before state transition: null
2013-08-21 04:25:07,193 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copyAsync
inspecting src type VOLUME copyAsync inspecting dest type VOLUME
2013-08-21 04:25:07,197 DEBUG [cache.allocator.StorageCacheRandomAllocator]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Can't find
staging storage in zone: 1
2013-08-21 04:25:07,233 DEBUG [agent.transport.Request] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424131: Sending { Cmd ,
MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"08d53ff3-8884-30a4-a1d1-ac621abcc688","id":3,"poolType":"RBD","host":"10.147.41.3","path":"cloudkvm","port":6789}},"name":"ROOT-4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"path":"volumes/2/4","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"executeInSequence":false,"wait":10800}}]
}
2013-08-21 04:25:07,571 DEBUG [agent.transport.Request] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424131: Received: { Ans: ,
MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-08-21 04:25:07,571 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copy to
image store failed: com.cloud.utils.exception.CloudRuntimeException: Failed to
copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
2013-08-21 04:25:07,590 DEBUG [agent.transport.Request] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 3-891420680: Sending { Cmd ,
MgmtId: 7175246184473, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"wait":0}}]
}
2013-08-21 04:25:07,652 DEBUG [agent.transport.Request] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 3-891420680: Received: { Ans: ,
MgmtId: 7175246184473, via: 3, Ver: v1, Flags: 10, { Answer } }
2013-08-21 04:25:07,655 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 4 is
not found on image store 1, so no need to delete
2013-08-21 04:25:07,668 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unsupported
data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@581a95c4), no need
to delete from object in store ref table
2013-08-21 04:25:07,684 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unsupported
data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@53310048), no need
to delete from object in store ref table
2013-08-21 04:25:07,747 DEBUG [agent.transport.Request] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424132: Sending { Cmd ,
MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7aca5603-0b2c-4de6-8dee-45f192bf113f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","id":2,"poolType":"RBD","host":"10.147.41.3","path":"data","port":6789}},"name":"ROOT-4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId":9,"vmName":"i-2-4-VM","accountId":2,"id":9,"hypervisorType":"KVM"}},"wait":0}}]
}
2013-08-21 04:25:07,927 DEBUG [agent.transport.Request] (Job-Executor-9:job-21
= [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq 1-462424132: Received: { Ans: ,
MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10, { Answer } }
2013-08-21 04:25:07,936 INFO [storage.volume.VolumeServiceImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 9 is
not referred anywhere, remove it from volumes table
2013-08-21 04:25:07,947 ERROR [cloud.storage.VolumeManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) migrate
volume failed:com.cloud.utils.exception.CloudRuntimeException: Failed to copy
cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
2013-08-21 04:25:07,948 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Failed to
migration: com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:2] is unreachable: migrate volume failed:
com.cloud.utils.exception.CloudRuntimeException: Failed to copy
cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
2013-08-21 04:25:07,953 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state
transitted from :Migrating to Stopped with event: AgentReportStoppedvm's
original host id: 1 new host id: null host id before state transition: null
2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's
actual total CPU: 9572 and CPU after applying overprovisioning: 9572
2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's
actual total RAM: 16816889856 and RAM after applying overprovisioning:
16816889856
2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release cpu
from host: 1, old used: 2000,reserved: 100, actual total: 9572, total with
overprovisioning: 9572; new used: 1900,reserved:100; movedfromreserved:
false,moveToReserveredfalse
2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release mem
from host: 1, old used: 2013265920,reserved: 134217728, total: 16816889856; new
used: 1879048192,reserved:134217728; movedfromreserved:
false,moveToReserveredfalse
2013-08-21 04:25:07,974 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
2013-08-21 04:25:07,984 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Complete
async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to migration:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is
unreachable: migrate volume failed:
com.cloud.utils.exception.CloudRuntimeException: Failed to copy
cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
2013-08-21 04:25:10,287 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-24:null) Async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]
completed
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
