[
https://issues.apache.org/jira/browse/CLOUDSTACK-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
edison su closed CLOUDSTACK-5737.
---------------------------------
need to turn off concurrent commands send to hypervisor
> KVM - Recurring Snapshots - Few snapshots fail to get created becasue of
> "org.libvirt.LibvirtException: Storage pool not found: no storage pool with
> matching uuid"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5737
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5737
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: Build from 4.3
> Reporter: Sangeetha Hariharan
> Assignee: edison su
> Priority: Critical
> Fix For: 4.3.0
>
> Attachments: kvm.rar
>
>
> Set up:
> Advanced zone set up with 2 KVM (rhel 63) hosts.
> Steps to reproduce the problem:
> Deploy about 17 Vms in both hosts.
> Start "Hourly" snapshot for all the ROOT volumes of these Vms.
> Few snapshots fail to get created becasue of "org.libvirt.LibvirtException:
> Storage pool not found: no storage pool with matching uuid"
> Following exception seen in management server logs:
> 2014-01-02 19:43:01,296 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-165:ctx-9fc82249 ctx-387acac1) Complete async job-238,
> jobStatus: SUCCEEDED, resultCode: 0, result:
> rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAEA
> 2014-01-02 19:43:01,299 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null)
> Seq 1-1038163041: Processing: { Ans: , MgmtId: 82324189320212, via: 1, Ver:
> v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage pool not found: no storage pool with
> matching uuid\n\tat
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:620)\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:1214)\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.delete(LibvirtStoragePool.java:256)\n\tat
>
> com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMStorageProcessor.java:807)\n\tat
>
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)\n\tat
>
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)\n\tat
>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)\n\tat
> com.cloud.agent.Agent.processRequest(Agent.java:498)\n\tat
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)\n\tat
> com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
> java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
> 2014-01-02 19:43:01,299 DEBUG [c.c.a.t.Request]
> (Job-Executor-164:ctx-af4c7558 ctx-7d147b95) Seq 1-1038163041: Received: {
> Ans: , MgmtId: 82324189320212, via: 1, Ver: v1, Flags: 10, { Answer } }
> 2014-01-02 19:43:01,300 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null)
> Seq 1-1038163040: Processing: { Ans: , MgmtId: 82324189320212, via: 1, Ver:
> v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage pool not found: no storage pool with
> matching uuid\n\tat
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:620)\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:1214)\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.delete(LibvirtStoragePool.java:256)\n\tat
>
> com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMStorageProcessor.java:807)\n\tat
>
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)\n\tat
>
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)\n\tat
>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)\n\tat
> com.cloud.agent.Agent.processRequest(Agent.java:498)\n\tat
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)\n\tat
> com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
> java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
> 2014-01-02 19:43:01,300 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null)
> Seq 1-1038163042: Processing: { Ans: , MgmtId: 82324189320212, via: 1, Ver:
> v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage pool not found: no storage pool with
> matching uuid\n\tat
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:620)\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:1214)\n\tat
>
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.delete(LibvirtStoragePool.java:256)\n\tat
>
> com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMStorageProcessor.java:807)\n\tat
>
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)\n\tat
>
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)\n\tat
>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)\n\tat
> com.cloud.agent.Agent.processRequest(Agent.java:498)\n\tat
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)\n\tat
> com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
> java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
> 2014-01-02 19:43:01,300 DEBUG [c.c.a.t.Request]
> (Job-Executor-162:ctx-6eadd346 ctx-bfe82c20) Seq 1-1038163040: Received: {
> Ans: , MgmtId: 82324189320212, via: 1, Ver: v1, Flags: 10, { Answer } }
> 2014-01-02 19:43:01,301 DEBUG [c.c.a.t.Request]
> (Job-Executor-163:ctx-b302ed15 ctx-d6e2ae7f) Seq 1-1038163042: Received: {
> Ans: , MgmtId: 82324189320212, via: 1, Ver: v1, Flags: 10, { Answer } }
> 2014-01-02 19:43:01,330 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-6:ctx-8795a455) Add job-236 into job monitoring
> 2014-01-02 19:43:01,330 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-6:ctx-8795a455) Executing AsyncJobVO {id:236, userId: 1,
> accountId: 4, instanceType: Snapshot, instanceId: 16, cmd:
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo:
> {"id":"16","ctxUserId":"1","volumeid":"27","ctxAccountId":"4","ctxStartEventId":"1","policyid":"13"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 82324189320212, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Thu Jan 02 18:44:19 EST 2014}
> 2014-01-02 19:43:01,343 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-165:ctx-9fc82249) Done executing
> com.cloud.storage.VmWorkTakeVolumeSnapshot for job-238
> 2014-01-02 19:43:01,326 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (Job-Executor-164:ctx-af4c7558 ctx-7d147b95) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage pool not found: no storage pool with
> matching uuid
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:620)
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:1214)
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.delete(LibvirtStoragePool.java:256)
> at
> com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMStorageProcessor.java:807)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)
> at com.cloud.agent.Agent.processRequest(Agent.java:498)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:142)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:301)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy160.takeSnapshot(Unknown Source)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1371)
> at
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:1723)
> at
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2478)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy195.handleVmWorkJob(Unknown Source)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-01-02 19:43:01,346 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (Job-Executor-162:ctx-6eadd346 ctx-bfe82c20) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Storage pool not found: no storage pool with
> matching uuid
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:620)
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deleteStoragePool(LibvirtStorageAdaptor.java:1214)
> at
> com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.delete(LibvirtStoragePool.java:256)
> at
> com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMStorageProcessor.java:807)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)
> at com.cloud.agent.Agent.processRequest(Agent.java:498)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)
> mysql> select id,volume_id,name,status from snapshots where status="Error";
> +----+-----------+-------------------------------------------------+--------+
> | id | volume_id | name | status |
> +----+-----------+-------------------------------------------------+--------+
> | 13 | 30 | TestVM-tiny-host-1ps-0-4_ROOT-30_20140102234419 | Error |
> | 14 | 29 | TestVM-tiny-host-1ps-0-3_ROOT-29_20140102234419 | Error |
> | 15 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140102234419 | Error |
> | 29 | 28 | TestVM-tiny-host-1ps-0-2_ROOT-28_20140103014419 | Error |
> +----+-----------+-------------------------------------------------+--------+
> 4 rows in set (0.00 sec)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
