I have upgraded to 4.3 and am see this error in the logs on a very regular basis.
appears to be: https://issues.apache.org/jira/browse/CLOUDSTACK-5834 Any ideas on how to determine the VBD it is referring to? I am wondering if I can mark it as deleted/inactive in the db. 2014-07-02 22:44:32,434 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-15:null) SeqA 8-3981: Sending Seq 8-3981: { Ans: , MgmtId : 233845174730255, via: 8, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2014-07-02 22:44:34,428 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-e7021ad2) VmStatsCollector is running... 2014-07-02 22:44:34,464 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-215:ctx-09244596) Seq 1-1007289330: Executing request 2014-07-02 22:44:35,267 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.87875 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.4 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.11499999999999999 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.145 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.21 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.13 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.8 2014-07-02 22:44:35,268 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Vm cpu utilization 0.25 2014-07-02 22:44:35,309 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-215:ctx-09244596) Error while collecting disk stats from : You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:209) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VBDMetrics.getIoReadKbs(VBDMetrics.java:210) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVmStats(CitrixResourceBase.java:2863) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2763) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:493) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-07-02 22:44:35,310 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-215:ctx-09244596) Seq 1-1007289330: Response Received: 2014-07-02 22:44:35,311 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e7021ad2) Seq 1-1007289330: Received: { Ans: , MgmtId: 23384517473 0255, via: 1, Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2014-07-02 22:44:35,358 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-364:ctx-77c17ef2) Seq 2-1359610876: Executing request 2014-07-02 22:44:35,959 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-b89b8193) Zone 1 is ready to launch console proxy 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.35000000000000003 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.26 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.245 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.74 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.17500000000000002 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.26 2014-07-02 22:44:36,002 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.26 2014-07-02 22:44:36,003 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.185 2014-07-02 22:44:36,003 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.24000000000000002 2014-07-02 22:44:36,003 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.19499999999999998 2014-07-02 22:44:36,003 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.15500000000000003 2014-07-02 22:44:36,003 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.12000000000000001 2014-07-02 22:44:36,003 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Vm cpu utilization 0.20499999999999996 2014-07-02 22:44:36,044 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-77c17ef2) Error while collecting disk stats from : You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:209) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VBDMetrics.getIoReadKbs(VBDMetrics.java:210) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVmStats(CitrixResourceBase.java:2863) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2763) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:493) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-07-02 22:44:36,045 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-364:ctx-77c17ef2) Seq 2-1359610876: Response Received: 2014-07-02 22:44:36,045 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e7021ad2) Seq 2-1359610876: Received: { Ans: , MgmtId: 233845174730255, via: 2, Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2014-07-02 22:44:36,150 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-2e22de6c) Zone 1 is ready to launch secondary storage VM 2014-07-02 22:44:37,430 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 8-3982: Processing Seq 8-3982: { Cmd , MgmtId: -1, via: 8, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":60,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2014-07-02 22:44:37,434 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 8-3982: Sending Seq 8-3982: { Ans: , MgmtId: 233845174730255, via: 8, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2014-07-02 22:44:39,016 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-220:ctx-3d57a24d) Seq 1-1007288322: Executing request 2014-07-02 22:44:39,306 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-220:ctx-3d57a24d) Seq 1-1007288322: Response Received: 2014-07-02 22:44:39,306 DEBUG [c.c.a.t.Request] (DirectAgent-220:ctx-3d57a24d) Seq 1-1007288322: Processing: { Ans: , MgmtId: 233845174730255, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } 2014-07-02 22:44:42,430 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 8-3983: Processing Seq 8-3983: { Cmd , MgmtId: -1, via: 8, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":60,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2014-07-02 22:44:42,434 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 8-3983: Sending Seq 8-3983: { Ans: , MgmtId: 233845174730255, via: 8, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2014-07-02 22:44:43,300 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-9ee27fbb) HostStatsCollector is running... 2014-07-02 22:44:43,310 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-77:ctx-c8dcabb6) Seq 1-1007289331: Executing request 2014-07-02 22:44:43,932 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-77:ctx-c8dcabb6) Seq 1-1007289331: Response Received: 2014-07-02 22:44:43,932 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-9ee27fbb) Seq 1-1007289331: Received: { Ans: , MgmtId: 233845174730255, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2014-07-02 22:44:43,940 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-226:ctx-cc6ee5cb) Seq 2-1359610877: Executing request 2014-07-02 22:44:44,151 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a9bca887) Found 0 routers to update status. 2014-07-02 22:44:44,153 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a9bca887) Found 0 networks to update RvR status. 2014-07-02 22:44:44,316 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-226:ctx-cc6ee5cb) Seq 2-1359610877: Response Received: 2014-07-02 22:44:44,316 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-9ee27fbb) Seq 2-1359610877: Received: { Ans: , MgmtId: 233845174730255, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
