Have you added os/hypervisor mappings inside the DB? I vaguely remember 7.5
not having needed mapping and was considered to be 6.5, thus a manual fix
was needed.
Perhaps Rohit can sched some light?
Anyways, a full log would be great (pastebin or other online service
please).
Regards
On Tue, 16 Jul 2019, 00:59 Nick Thompson, <nick.thomp...@neos.co.nz> wrote:
> Hey,
>
> Since we upgraded to the 4.11 branch (currently 4.11.3) and virtual
> routers have become HVM on XenServer/XCP-ng we have had problems with the
> virtual routers randomly rebooting themselves. We still have some running
> in the older paravirtualized mode and they seem to be fine (it may be that
> the Management server can't communicate to these virtual routers since they
> are an older template?). Other running Windows/Linux VMs are fine.
>
> CloudStack Cluster: XCP-ng 7.5 (previously XenServer 6.5, same issue was
> happening)
> CloudStack: 4.11.3 (same issue in 4.11.2, was working fine in V4.9.3
>
> When digging through the management-server.log, I have found the following;
>
> >grep -n "Error while collecting network stats from router"
> /var/log/cloudstack/management/management-server.log
> 919060:2019-07-16 10:28:43,756 WARN
> [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-1931d792)
> (logid:6e236728) Error while collecting network stats from router:
> r-2280-VM from host: 71; details: Exception: java.lang.Exception
> 919328:2019-07-16 10:28:50,940 WARN
> [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-1931d792)
> (logid:6e236728) Error while collecting network stats from router:
> r-2280-VM from host: 71; details: Exception: java.lang.Exception
> 920533:2019-07-16 10:29:54,768 WARN
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Error while collecting network stats from router:
> r-2280-VM from host: 71; details: Exception: java.lang.Exception
> 920621:2019-07-16 10:30:01,952 WARN
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Error while collecting network stats from router:
> r-2280-VM from host: 71; details: Exception: java.lang.Exception
>
> >less +920621 /var/log/cloudstack/management/management-server.log
> 2019-07-16 10:30:01,952 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Seq 71-7505811728966836555: Received: { Ans: , MgmtId:
> 226842157555374, via: 71(hostname), Ver: v1, Flags: 10, {
> NetworkUsageAnswer } }
> 2019-07-16 10:30:01,952 DEBUG [c.c.a.m.AgentManagerImpl]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Details from executing class
> com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.Exception
> Message: vpc network usage plugin call failed
> Stack: java.lang.Exception: vpc network usage plugin call failed
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.executeNetworkUsage(XenServer56NetworkUsageCommandWrapper.java:84)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:41)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:33)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1737)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2019-07-16 10:30:01,952 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Error while collecting network stats from router:
> r-2280-VM from host: 71; details: Exception: java.lang.Exception
> Message: vpc network usage plugin call failed
> Stack: java.lang.Exception: vpc network usage plugin call failed
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.executeNetworkUsage(XenServer56NetworkUsageCommandWrapper.java:84)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:41)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:33)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1737)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2019-07-16 10:30:01,973 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Seq 71-7505811728966836557: Sending { Cmd , MgmtId:
> 226842157555374, via: 71(hostname), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.200","forceStop":true,"volumesToDisconnect":[],"vmName":"r-2280-VM","executeInSequence":false,"wait":0}}]
> }
> 2019-07-16 10:30:01,973 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
> (logid:2b94fd5d) Seq 71-7505811728966836557: Executing: { Cmd , MgmtId:
> 226842157555374, via: 71(hostname), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.200","forceStop":true,"volumesToDisconnect":[],"vmName":"r-2280-VM","executeInSequence":false,"wait":0}}]
> }
> 2019-07-16 10:30:01,973 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-430:ctx-77c57645) (logid:8733e444) Seq 71-7505811728966836557:
> Executing request
> 2019-07-16 10:30:01,995 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> (DirectAgent-430:ctx-77c57645) (logid:2b94fd5d) 9. The VM r-2280-VM is in
> Stopping state
> 2019-07-16 10:30:02,303 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-390:ctx-3b9a78cd) (logid:27f6ec94) Seq 67-8459448950062117779:
> Response Received
>
>
> Any thoughts would be greatly appreciated.
>
> Cheers,
> Nick.
>
