Abhinav Roy created CLOUDSTACK-5678:
---------------------------------------
Summary: Cold Storage migration is failing
Key: CLOUDSTACK-5678
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5678
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: hyperv , 4.3
Reporter: Abhinav Roy
Priority: Blocker
Fix For: 4.3.0
Steps to Reproduce:
================
1.Bring up CS in advanced zone with 2 hosts in a hyper-v cluster using CIFS for
both primary and secondary storage
2.Deploy one or two guest vms using default cent os template.
3.Add one more primary storage (say primary2) to the cluster
4.Stop one of the vms deployed at step2
5.Migrate the storage from source primary to new primary storage primary2
Expected Result:
=============
Storage migration should succeed when the vm is in stopped state
Actual Result:
===========
013-12-30 17:21:03,163 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-9672ce4b)
===START=== 10.144.7.20 -- GET
command=migrateVirtualMachine&storageid=d65ffa8c-fdd7-360a-b729-c0d9ba5df050&virtualmachineid=9226ccd6-b409-4452-821f-5b33a24137a5&response=json&sessionkey=lIed4xLaMsDL4yCyAJYyUK0pAJk%3D&_=1388403965445
2013-12-30 17:21:03,194 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-6:ctx-9672ce4b ctx-5d353029) submit async job-129, details:
AsyncJobVO {id:129, userId: 2, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
{"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"9226ccd6-b409-4452-821f-5b33a24137a5","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"d65ffa8c-fdd7-360a-b729-c0d9ba5df050","httpmethod":"GET","_":"1388403965445","ctxAccountId":"2","ctxStartEventId":"274"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2013-12-30 17:21:03,194 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-9672ce4b
ctx-5d353029) ===END=== 10.144.7.20 -- GET
command=migrateVirtualMachine&storageid=d65ffa8c-fdd7-360a-b729-c0d9ba5df050&virtualmachineid=9226ccd6-b409-4452-821f-5b33a24137a5&response=json&sessionkey=lIed4xLaMsDL4yCyAJYyUK0pAJk%3D&_=1388403965445
2013-12-30 17:21:03,196 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-48:ctx-46f6bd2b) Add job-129 into job monitoring
2013-12-30 17:21:03,196 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-48:ctx-46f6bd2b) Executing AsyncJobVO {id:129, userId: 2,
accountId: 2, instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
{"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"9226ccd6-b409-4452-821f-5b33a24137a5","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"d65ffa8c-fdd7-360a-b729-c0d9ba5df050","httpmethod":"GET","_":"1388403965445","ctxAccountId":"2","ctxStartEventId":"274"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2013-12-30 17:21:03,212 DEBUG [c.c.u.d.T.Transaction]
(Job-Executor-48:ctx-46f6bd2b ctx-5d353029) Rolling back the transaction: Time
= 4 Name = Job-Executor-48; called by
-TransactionLegacy.rollback:896-TransactionLegacy.removeUpTo:839-TransactionLegacy.close:663-Transaction.execute:41-Transaction.execute:46-VirtualMachineManagerImpl.migrateVmStorageThroughJobQueue:4475-VirtualMachineManagerImpl.storageMigration:1558-UserVmManagerImpl.vmStorageMigration:4069-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:616
2013-12-30 17:21:03,213 ERROR [c.c.a.ApiAsyncJobDispatcher]
(Job-Executor-48:ctx-46f6bd2b) Unexpected exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
com.cloud.utils.exception.CloudRuntimeException: Unable to serialize:
com.cloud.vm.VmWorkStorageMigration@40df9e44
at
org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.toObjectSerializedString(JobSerializerHelper.java:118)
at com.cloud.vm.VmWorkSerializer.serialize(VmWorkSerializer.java:63)
at
com.cloud.vm.VirtualMachineManagerImpl$9.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:4504)
at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
at
com.cloud.vm.VirtualMachineManagerImpl.migrateVmStorageThroughJobQueue(VirtualMachineManagerImpl.java:4475)
at
com.cloud.vm.VirtualMachineManagerImpl.storageMigration(VirtualMachineManagerImpl.java:1558)
at
com.cloud.vm.UserVmManagerImpl.vmStorageMigration(UserVmManagerImpl.java:4069)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy169.vmStorageMigration(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd.execute(MigrateVMCmd.java:150)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.NotSerializableException:
org.apache.cloudstack.storage.datastore.db.PrimaryDataStoreDaoImpl
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1180)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at
org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.toObjectSerializedString(JobSerializerHelper.java:112)
... 40 more
2013-12-30 17:21:03,222 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-48:ctx-46f6bd2b) Complete async job-129, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to serialize: com.cloud.vm.VmWorkStorageMigration@40df9e44"}
2013-12-30 17:21:03,226 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-48:ctx-46f6bd2b) Done executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-129
Workaround :
=========================
Add a new entry in cloud.configurations table with
vm.job.enabled set to "false"
restart management server. After this Live VM migration works fine.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
