[
https://issues.apache.org/jira/browse/CLOUDSTACK-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725522#comment-14725522
]
Adrian Sender commented on CLOUDSTACK-8718:
-------------------------------------------
Altering the job.cancel.threshold.minutes has resolved deploying large imported
templates and ConcurrentOperationException caused by the long copy time from
secondary to primary storage.
Should this be closed off or are there any follow up items?
> Rethrow exception com.cloud.exception.ConcurrentOperationException: There are
> concurrent operations on VM
> ---------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-8718
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8718
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation, Management Server
> Affects Versions: 4.5.1
> Environment: Cloudplatform 4.5
> Xenserver 6.5
> EMC Fiber Channel Primary Storage
> Freenas NFS Secondary Storage
> Citrix Case Detail:
> Case Information:
> Case Number 70353012 Status Active - Technical Support
> Severity 2
> Subject Rethrow exception
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM
> Reporter: Adrian Sender
> Priority: Blocker
> Attachments: apilog.log, catalina.zip, clouddb-10.08.15.sql,
> management-server.log.2015-08-04.gz, management-server.log.2015-08-05.gz,
> management-server.zip, management-server.zip
>
>
> Cannot deploy instances from large size root disk templates. Small root disk
> works without any problem.
> We can deploy large templates in 4.3 without any issues.
> Strange behavior, the template might take 7 hours to copy depending on the
> size of the root disk.
> Once xenserver completes the task and we try and start the instance in CCP
> the copy process starts all over again.
> [root@xenserver-01 ~]# xe task-list|less
> uuid ( RO) : 539cd570-2348-e06a-0585-a6ca0198660a
> name-label ( RO): Async.VDI.copy
> name-description ( RO):
> status ( RO): pending
> progress ( RO): 0.580
> 2015-07-20 14:36:50,429 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136 ctx-30d5ecdd)
> (logid:5808e018) The task item for vm VM[User|i-8-189-VM] has been inactive
> for 3602
> 2015-07-20 14:36:50,429 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136 ctx-30d5ecdd)
> (logid:5808e018) Invocation exception, caused by:
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM[User|i-8-189-VM]
> 2015-07-20 14:36:50,430 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136 ctx-30d5ecdd)
> (logid:5808e018) Rethrow exception
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM[User|i-8-189-VM]
> 2015-07-20 14:36:50,430 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136) (logid:5808e018) Done
> with run of VM work job: com.cloud.vm.VmWorkStart for VM 189, job origin: 2133
> 2015-07-20 14:36:50,430 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-9:ctx-116ace03 job-2133/job-2136) (logid:5808e018) Unable
> to complete AsyncJobVO {id:2136, userId: 8, accountId: 8, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAAvXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gzQmhjM04zYjNKa3hw,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 192022089101382, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Mon Jul 20 14:36:49 EST 2015}, job origin:2133
> com.cloud.exception.ConcurrentOperationException: There are concurrent
> operations on VM[User|i-8-189-VM]
> at
> com.cloud.vm.VirtualMachineManagerImpl.changeToStartState(VirtualMachineManagerImpl.java:738)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:841)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4726)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:549)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to
> concurrent operation
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:628)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:242)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3933)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3522)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3510)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy197.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:453)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:549)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.cloud.exception.ConcurrentOperationException: There are
> concurrent operations on VM[User|i-8-189-VM]
> ... 35 more
> 2015-07-20 14:36:50,623 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-12:ctx-8d825486 job-2133) (logid:5808e018) Complete async
> job-2133, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
> to start a VM due to concurrent operation"}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
