[
https://issues.apache.org/jira/browse/CLOUDSTACK-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710331#comment-14710331
]
Adrian Sender commented on CLOUDSTACK-8718:
-------------------------------------------
If we let the copy finish, sometimes we see the following error when starting
the instance, sometimes it starts the copy process to primary storage all over
again.
When trying to start the instance:
2015-08-25 10:22:34,867 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-189:ctx-c5390b6c) (logid:5837a364) Host 172.26.7.22
OpaqueRef:920068bb-2da6-b174-2a68-3f3045895328: Catch XenAPIException: The SR
operation cannot be performed because a device underlying the SR is in use by
the host.
And sometimes this error:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is
unreachable: Unable to create Vol[278|vm=268|ROOT]:Can't find template VDI
under:
172.26.7.7:/mnt/SAS-EMC-JBOB-Enclosure-0-0-9/cloudstack/secondary/template/tmpl/13/259
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1264)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1322)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1005)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4726)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:549)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
> Rethrow exception com.cloud.exception.ConcurrentOperationException: There are
> concurrent operations on VM
> ---------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-8718
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8718
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation, Management Server
> Affects Versions: 4.5.1
> Environment: Cloudplatform 4.5
> Xenserver 6.5
> EMC Fiber Channel Primary Storage
> Freenas NFS Secondary Storage
> Citrix Case Detail:
> Case Information:
> Case Number 70353012 Status Active - Technical Support
> Severity 2
> Subject Rethrow exception
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM
> Reporter: Adrian Sender
> Priority: Blocker
> Attachments: apilog.log, catalina.zip, clouddb-10.08.15.sql,
> management-server.log.2015-08-04.gz, management-server.log.2015-08-05.gz,
> management-server.zip, management-server.zip
>
>
> Cannot deploy instances from large size root disk templates. Small root disk
> works without any problem.
> We can deploy large templates in 4.3 without any issues.
> Strange behavior, the template might take 7 hours to copy depending on the
> size of the root disk.
> Once xenserver completes the task and we try and start the instance in CCP
> the copy process starts all over again.
> [root@xenserver-01 ~]# xe task-list|less
> uuid ( RO) : 539cd570-2348-e06a-0585-a6ca0198660a
> name-label ( RO): Async.VDI.copy
> name-description ( RO):
> status ( RO): pending
> progress ( RO): 0.580
> 2015-07-20 14:36:50,429 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136 ctx-30d5ecdd)
> (logid:5808e018) The task item for vm VM[User|i-8-189-VM] has been inactive
> for 3602
> 2015-07-20 14:36:50,429 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136 ctx-30d5ecdd)
> (logid:5808e018) Invocation exception, caused by:
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM[User|i-8-189-VM]
> 2015-07-20 14:36:50,430 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136 ctx-30d5ecdd)
> (logid:5808e018) Rethrow exception
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM[User|i-8-189-VM]
> 2015-07-20 14:36:50,430 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136) (logid:5808e018) Done
> with run of VM work job: com.cloud.vm.VmWorkStart for VM 189, job origin: 2133
> 2015-07-20 14:36:50,430 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136) (logid:5808e018) Unable
> to complete AsyncJobVO {id:2136, userId: 8, accountId: 8, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAAvXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gzQmhjM04zYjNKa3hw,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 192022089101382, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Mon Jul 20 14:36:49 EST 2015}, job origin:2133
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM[User|i-8-189-VM]
> at
> com.cloud.vm.VirtualMachineManagerImpl.changeToStartState(VirtualMachineManagerImpl.java:738)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:841)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4726)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:549)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to
> concurrent operation
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:628)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:242)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3933)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3522)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3510)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy197.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:453)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:549)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.cloud.exception.ConcurrentOperationException: There are
> concurrent operations on VM[User|i-8-189-VM]
> ... 35 more
> 2015-07-20 14:36:50,623 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-12:ctx-8d825486 job-2133) (logid:5808e018) Complete async
> job-2133, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
> to start a VM due to concurrent operation"}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
