Hi Martin, This happens when multiple threads tries to access the same table at same time. Its a race case situation.
Regards Vihar K On Fri, Aug 29, 2014 at 5:08 AM, Kirk Kosinski <[email protected]> wrote: > Hi, Mārtiņš. Unfortunately CloudStack doesn't support this yet. > > Best regards, > Kirk > > On 08/26/2014 03:17 AM, Mārtiņš Jakubovičs wrote: > > Hello, > > > > I test schaduled volume snapshots and when I create in same time rules > > for few machines (in my case I got 5 VM's) they all fails except one. > > > > http://pastebin.com/E60emjuJ > > > > How I can solve this? > > > > INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-125a21bc) checking > > health of usage server > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45 > > job-9783) Add job-9783 into job monitoring > > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > > (API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) VOLSS: > > createSnapshotCmd starts:1409047550501 > > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-49:ctx-1bf25e45 job-9783 > > ctx-9d7e935b) Was unable to find lock for the key vm_instance1173 and > > thread id 158740505 > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db > > job-9785) Add job-9785 into job monitoring > > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > > (API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) VOLSS: > > createSnapshotCmd starts:1409047550587 > > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-50:ctx-bb4641db job-9785 > > ctx-26da1a0f) Was unable to find lock for the key vm_instance1172 and > > thread id 736777923 > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859 > > job-9787) Add job-9787 into job monitoring > > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > > (API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) VOLSS: > > createSnapshotCmd starts:1409047550660 > > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-51:ctx-1f378859 job-9787 > > ctx-ae2be6e3) Was unable to find lock for the key vm_instance1171 and > > thread id 142175984 > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de > > job-9788) Add job-9788 into job monitoring > > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > > (API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) VOLSS: > > createSnapshotCmd starts:1409047550750 > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15 > > job-9790) Add job-9790 into job monitoring > > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-52:ctx-197114de job-9788 > > ctx-f29734ce) Was unable to find lock for the key vm_instance1170 and > > thread id 1643232589 > > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > > (API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) VOLSS: > > createSnapshotCmd starts:1409047550795 > > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-53:ctx-1e698b15 job-9790 > > ctx-97cf6519) Was unable to find lock for the key vm_instance1168 and > > thread id 1037433024 > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025 > > job-9783/job-9784) Add job-9784 into job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2 > > job-9785/job-9786) Add job-9786 into job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70 > > job-9787/job-9789) Add job-9789 into job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2 > > job-9788/job-9791) Add job-9791 into job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6 > > job-9790/job-9792) Add job-9792 into job monitoring > > WARN [o.a.c.h.x.XenServerResourceNewBase] > > (DirectAgent-303:ctx-c4ae6319) No event for task > > OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff > > WARN [o.a.c.h.x.XenServerResourceNewBase] > > (DirectAgent-119:ctx-cac68540) No event for task > > OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a > > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-119:ctx-cac68540) Host > > 10.10.10.66 OpaqueRef:18b050f6-a1db-1ba5-f290-bcd58811caad: Unable to > > remove SR > > WARN [c.c.h.x.r.XenServerStorageProcessor] > > (DirectAgent-119:ctx-cac68540) BackupSnapshot Failed due to No event for > > task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a > > java.util.concurrent.TimeoutException: No event for task > > OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a > > at > > > org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124) > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > > at > > > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > > at > > > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > > at > > > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > > at > > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70 > > job-9787/job-9789) Remove job-9789 from job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859 > > job-9787) Remove job-9787 from job monitoring > > WARN [c.c.h.x.r.XenServerStorageProcessor] > > (DirectAgent-303:ctx-c4ae6319) BackupSnapshot Failed due to No event for > > task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff > > java.util.concurrent.TimeoutException: No event for task > > OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff > > at > > > org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124) > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > > at > > > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > > at > > > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > > at > > > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > > at > > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025 > > job-9783/job-9784) Remove job-9784 from job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45 > > job-9783) Remove job-9783 from job monitoring > > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-176:ctx-830c05b9) > > callHostPlugin failed for cmd: mountNfsSecondaryStorage with args > > remoteDir: 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817, localDir: > > /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3, due to You gave > > an invalid session reference. It may have been invalidated by a server > > restart, or timed out. You should get a new session handle, using one > > of the session.login_ calls. This error does not invalidate the current > > connection. The handle parameter echoes the bad value given. > > You gave an invalid session reference. It may have been invalidated by > > a server restart, or timed out. You should get a new session handle, > > using one of the session.login_ calls. This error does not invalidate > > the current connection. The handle parameter echoes the bad value given. > > at com.xensource.xenapi.Types.checkResponse(Types.java:218) > > at com.xensource.xenapi.Connection.dispatch(Connection.java:350) > > at > > > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) > > at com.xensource.xenapi.Event.properFrom(Event.java:310) > > at > > > org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:116) > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3506) > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:68) > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > > at > > > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > > at > > > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > > at > > > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > > at > > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > WARN [c.c.h.x.r.XenServerStorageProcessor] > > (DirectAgent-176:ctx-830c05b9) Could not mount secondary storage > > 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host > > /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 > > WARN [c.c.h.x.r.XenServerStorageProcessor] > > (DirectAgent-176:ctx-830c05b9) BackupSnapshot Failed due to Could not > > mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 > > on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 > > com.cloud.utils.exception.CloudRuntimeException: Could not mount > > secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on > > host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:73) > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > > at > > > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > > at > > > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > > at > > > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > > at > > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Task > > failed! Task record: uuid: > > cd20da12-7cbc-18cf-9629-c74dd1e3fa53 > > nameLabel: Async.VDI.copy > > nameDescription: > > allowedOperations: [cancel] > > currentOperations: {} > > created: Tue Aug 26 13:07:15 EEST 2014 > > finished: Thu Jan 01 03:00:00 EET 1970 > > status: pending > > residentOn: com.xensource.xenapi.Host@40960732 > > progress: 0.0 > > type: <none/> > > result: > > errorInfo: [] > > otherConfig: {} > > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > > subtasks: [com.xensource.xenapi.Task@1e6e981e] > > > > WARN [c.c.h.x.r.XenServerStorageProcessor] > > (DirectAgent-288:ctx-0efa2c48) unable to destroy > > task(OpaqueRef:be099f81-ab1b-0808-6652-af015577a575) due to You gave an > > invalid object reference. The object may have recently been deleted. > > The class parameter gives the type of reference given, and the handle > > parameter echoes the bad value given. > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6 > > job-9790/job-9792) Remove job-9792 from job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15 > > job-9790) Remove job-9790 from job monitoring > > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Host > > 10.10.10.66 OpaqueRef:bf007bb1-b83c-9e14-75f7-6790b52de61b: Unable to > > remove SR > > WARN [c.c.h.x.r.XenServerStorageProcessor] > > (DirectAgent-288:ctx-0efa2c48) BackupSnapshot Failed due to Task failed! > > Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53 > > nameLabel: Async.VDI.copy > > nameDescription: > > allowedOperations: [cancel] > > currentOperations: {} > > created: Tue Aug 26 13:07:15 EEST 2014 > > finished: Thu Jan 01 03:00:00 EET 1970 > > status: pending > > residentOn: com.xensource.xenapi.Host@40960732 > > progress: 0.0 > > type: <none/> > > result: > > errorInfo: [] > > otherConfig: {} > > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > > subtasks: [com.xensource.xenapi.Task@1e6e981e] > > > > Task failed! Task record: uuid: > > cd20da12-7cbc-18cf-9629-c74dd1e3fa53 > > nameLabel: Async.VDI.copy > > nameDescription: > > allowedOperations: [cancel] > > currentOperations: {} > > created: Tue Aug 26 13:07:15 EEST 2014 > > finished: Thu Jan 01 03:00:00 EET 1970 > > status: pending > > residentOn: com.xensource.xenapi.Host@40960732 > > progress: 0.0 > > type: <none/> > > result: > > errorInfo: [] > > otherConfig: {} > > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > > subtasks: [com.xensource.xenapi.Task@1e6e981e] > > > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3293) > > at > > > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:431) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > > at > > > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > > at > > > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > > at > > > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > > at > > > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > > at > > > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > > at > > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2 > > job-9788/job-9791) Remove job-9791 from job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de > > job-9788) Remove job-9788 from job monitoring > > WARN [o.a.c.alerts] (CapacityChecker:ctx-3116e57f) alertType:: 3 // > > dataCenterId:: 5 // podId:: 5 // clusterId:: null // message:: System > > Alert: Remaining unallocated Storage is low in cluster DEMO Cluster-1 > > pod Demo-Rack-1 of availability zone Demo > > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task > > (job-9785) has been pending for 168 seconds > > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task > > (job-9786) has been pending for 166 seconds > > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task > > (job-9785) has been pending for 168 seconds > > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task > > (job-9786) has been pending for 166 seconds > > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task > > (job-9785) has been pending for 222 seconds > > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task > > (job-9786) has been pending for 221 seconds > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2 > > job-9785/job-9786) Remove job-9786 from job monitoring > > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db > > job-9785) Remove job-9785 from job monitoring > > > >
