youngsikyun created CLOUDSTACK-10063: ----------------------------------------
Summary: Cloudstack proceeded after volume migration failed and failed, but failed Key: CLOUDSTACK-10063 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10063 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: API, cloudstack-agent, Secondary Storage, XenServer Affects Versions: 4.5.1 Environment: XenServer 6.2.0 Reporter: youngsikyun Fix For: 4.5.1 The volume migration from Cloudstack has been unsuccessful. When I try again, the following log fails and fails. I do not know how to solve it. 2017-09-04 09:33:09,744 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-29:ctx-22ba811c job-4558169) (logid:0f25981e) Add job-4558169 into job monitoring 2017-09-04 09:33:09,748 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-15:ctx-6ed21df6 ctx-50260cc7) (logid:4d42b821) submit async job-4558169, details: AsyncJobVO {id:4558169, userId: 6227, accountId: 6188, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.volume.MigrateVolumeCmdByAdmin, cmdInfo: {"response":"json","sessionkey":"wf6FeFTvYvChL8kARL54Na/LdrI\u003d","ctxDetails":"{\"com.cloud.storage.Volume\":\"a2d110f9-6033-4160-b958-93d132355a7d\",\"com.cloud.storage.StoragePool\":\"93784b55-d8e4-3f37-8ef7-ba7357f0b8da\"}","cmdEventType":"VOLUME.MIGRATE","ctxUserId":"6227","storageid":"93784b55-d8e4-3f37-8ef7-ba7357f0b8da","httpmethod":"GET","volumeid":"a2d110f9-6033-4160-b958-93d132355a7d","_":"1504485025736","ctxAccountId":"6188","ctxStartEventId":"15164623"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 147945607443714, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2017-09-04 09:33:09,749 DEBUG [c.c.a.ApiServlet] (catalina-exec-15:ctx-6ed21df6 ctx-50260cc7) (logid:4d42b821) ===END=== 10.222.111.13 -- GET command=migrateVolume&storageid=93784b55-d8e4-3f37-8ef7-ba7357f0b8da&volumeid=a2d110f9-6033-4160-b958-93d132355a7d&response=json&sessionkey=wf6FeFTvYvChL8kARL54Na%2FLdrI%3D&_=1504485025736 2017-09-04 09:33:09,753 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-0f70cf12) (logid:cabbd82c) No need to calibrate cpu capacity, host:2555 usedCpu: 11750 reservedCpu: 0 2017-09-04 09:33:09,753 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-0f70cf12) (logid:cabbd82c) No need to calibrate memory capacity, host:2555 usedMem: 23085449216 reservedMem: 0 2017-09-04 09:33:09,753 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:ctx-22ba811c job-4558169) (logid:e0d7183d) Executing AsyncJobVO {id:4558169, userId: 6227, accountId: 6188, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.volume.MigrateVolumeCmdByAdmin, cmdInfo: {"response":"json","sessionkey":"wf6FeFTvYvChL8kARL54Na/LdrI\u003d","ctxDetails":"{\"com.cloud.storage.Volume\":\"a2d110f9-6033-4160-b958-93d132355a7d\",\"com.cloud.storage.StoragePool\":\"93784b55-d8e4-3f37-8ef7-ba7357f0b8da\"}","cmdEventType":"VOLUME.MIGRATE","ctxUserId":"6227","storageid":"93784b55-d8e4-3f37-8ef7-ba7357f0b8da","httpmethod":"GET","volumeid":"a2d110f9-6033-4160-b958-93d132355a7d","_":"1504485025736","ctxAccountId":"6188","ctxStartEventId":"15164623"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 147945607443714, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2017-09-04 09:33:09,806 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) copyAsync inspecting src type VOLUME copyAsync inspecting dest type VOLUME 2017-09-04 09:33:09,809 DEBUG [o.a.c.s.c.a.StorageCacheRandomAllocator] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) Can't find staging storage in zone: 2 2017-09-04 09:33:09,820 DEBUG [o.a.c.s.v.VolumeObject] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) Failed to update state com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Creating via CreateOnlyRequested at com.cloud.utils.fsm.StateMachine2.getTransition(StateMachine2.java:98) at com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:84) at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:114) at org.apache.cloudstack.storage.datastore.ObjectInDataStoreManagerImpl.update(ObjectInDataStoreManagerImpl.java:314) at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:296) at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:327) at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:450) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:68) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:73) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:996) at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:923) at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:2026) at com.cloud.storage.VolumeApiServiceImpl.migrateVolume(VolumeApiServiceImpl.java:2012) at sun.reflect.GeneratedMethodAccessor953.invoke(Unknown Source) -- 2017-09-04 09:33:09,820 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) copy failed com.cloud.utils.exception.CloudRuntimeException: Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Creating via CreateOnlyRequested at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:336) at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:327) at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:450) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:68) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:73) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:996) at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:923) at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:2026) at com.cloud.storage.VolumeApiServiceImpl.migrateVolume(VolumeApiServiceImpl.java:2012) at sun.reflect.GeneratedMethodAccessor953.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) -- 2017-09-04 09:33:09,837 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@d302de4), no need to delete from object in store ref table 2017-09-04 09:33:09,874 DEBUG [o.a.c.s.v.VolumeServiceImpl] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) Marking volume that was never created as destroyed: Vol[541782|vm=null|DATADISK] 2017-09-04 09:33:09,878 ERROR [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) Migrate volume failed:com.cloud.utils.exception.CloudRuntimeException: Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Creating via CreateOnlyRequested 2017-09-04 09:33:09,878 DEBUG [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-29:ctx-22ba811c job-4558169 ctx-e6c2227b) (logid:e0d7183d) Failed to migrate volume com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2688] is unreachable: Migrate volume failed: com.cloud.utils.exception.CloudRuntimeException: Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Creating via CreateOnlyRequested at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:928) at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:2026) at com.cloud.storage.VolumeApiServiceImpl.migrateVolume(VolumeApiServiceImpl.java:2012) at sun.reflect.GeneratedMethodAccessor953.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) -- 2017-09-04 09:33:09,886 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-29:ctx-22ba811c job-4558169) (logid:e0d7183d) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.MigrateVolumeCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:2688] is unreachable: Migrate volume failed: com.cloud.utils.exception.CloudRuntimeException: Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Creating via CreateOnlyRequested at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:2030) at com.cloud.storage.VolumeApiServiceImpl.migrateVolume(VolumeApiServiceImpl.java:2012) at sun.reflect.GeneratedMethodAccessor953.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) -- This message was sent by Atlassian JIRA (v6.4.14#64029)