Hi Sonali, Can you please check the MS logs for NetworkUsageCommand and answer for the errors on VR start after upgrade.
Thanks, Jayapal On 06-Jul-2015, at 1:44 PM, Sonali Jadhav <[email protected]> wrote: > Hi, > > I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully > executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > > sysvm.log 2>&1 &" to update VRs. > > I also cross checked from ACS, version of each VR. It was 4.5. > > But I was seeing these errors in management logs from few VRs. so like you > said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted > them and then I was able to see that chain in iptables. So I rebooted all > those VRs with error logs. And now it seems to be ok. > > /Sonali > > -----Original Message----- > From: Jayapal Reddy Uradi [mailto:[email protected]] > Sent: Monday, July 6, 2015 12:45 PM > To: <[email protected]> > Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1 > > Hi Sonali, > > Can you give me some context on how this issue is seen. I want to understand > when these chains got removed in VR. > > Thanks, > Jayapal > > On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <[email protected]> wrote: > >> Sorry, error log was not for all VRs but some. >> >> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then >> there it was. I am suspecting VR was not updated ? weird but I was able to >> see version 4.5 of all VRs in ACS. >> I'll keep watch if error comes again. >> >> Thanks for help. >> >> /Sonali >> >> -----Original Message----- >> From: Jayapal Reddy Uradi [mailto:[email protected]] >> Sent: Monday, July 6, 2015 11:00 AM >> To: <[email protected]> >> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1 >> >> >> It seems VR does not have the usage related iptables chains. >> Please check the 'iptables -L -nv' for NETWORK_STATS chains. >> These rules supposed to be configured on VR start. >> >> Thanks, >> Jayapal >> >> >> >> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav >> <[email protected]<mailto:[email protected]>> >> wrote: >> >> Hi, >> >> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in >> management server logs, for each and every VR, >> >> >> 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] >> (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs >> 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] >> (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs >> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is >> running... >> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers. >> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status. >> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status. >> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] >> (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for >> external networking, so skipping usage check. >> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request >> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-284:ctx-0e4c6018) Executing command in VR: >> /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g >> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] >> (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at >> 2015-07-06 05:08:21 GMT >> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] >> (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 >> 05:08:21 GMT >> 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] >> (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to >> java.lang.NumberFormatException: For input string: "iptables" >> at >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> at java.lang.Long.parseLong(Long.java:441) >> at java.lang.Long.<init>(Long.java:702) >> at >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) >> at >> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received: >> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] >> (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { >> Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: >> java.lang.NumberFormatException\nMessage: For input string: >> \"iptables\"\nStack: java.lang.NumberFormatException: For input string: >> \"iptables\"\n\tat >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat >> java.lang.Long.parseLong(Long.java:441)\n\tat >> java.lang.Long.<init>(Long.java:702)\n\tat >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat >> >> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat >> >> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat >> >> com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat >> >> com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat >> >> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat >> >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat >> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat >> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat >> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat >> >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat >> java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat >> java.lang.Thread.run(Thread.java:744)\n","wait":0}}] } >> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] >> (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: >> , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } >> } >> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] >> (RouterMonitor-1:ctx-91d5dc5d) Details from executing class >> com.cloud.agent.api.NetworkUsageCommand: Exception: >> java.lang.NumberFormatException >> Message: For input string: "iptables" >> Stack: java.lang.NumberFormatException: For input string: "iptables" >> at >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> at java.lang.Long.parseLong(Long.java:441) >> at java.lang.Long.<init>(Long.java:702) >> at >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) >> at >> com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) >> at >> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> >> what exactly is this error? >> >> /sonali >> >
