Hi, I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd " nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &" to update VRs.
I also cross checked from ACS, version of each VR. It was 4.5. But I was seeing these errors in management logs from few VRs. so like you said I observed in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems to be ok. /Sonali -----Original Message----- From: Jayapal Reddy Uradi [mailto:[email protected]] Sent: Monday, July 6, 2015 12:45 PM To: <[email protected]> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1 Hi Sonali, Can you give me some context on how this issue is seen. I want to understand when these chains got removed in VR. Thanks, Jayapal On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <[email protected]> wrote: > Sorry, error log was not for all VRs but some. > > I don't see NETWORK_STATS chains for those. So, I just rebooted, and then > there it was. I am suspecting VR was not updated ? weird but I was able to > see version 4.5 of all VRs in ACS. > I'll keep watch if error comes again. > > Thanks for help. > > /Sonali > > -----Original Message----- > From: Jayapal Reddy Uradi [mailto:[email protected]] > Sent: Monday, July 6, 2015 11:00 AM > To: <[email protected]> > Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1 > > > It seems VR does not have the usage related iptables chains. > Please check the 'iptables -L -nv' for NETWORK_STATS chains. > These rules supposed to be configured on VR start. > > Thanks, > Jayapal > > > > On 06-Jul-2015, at 10:48 AM, Sonali Jadhav > <[email protected]<mailto:[email protected]>> > wrote: > > Hi, > > After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in > management server logs, for each and every VR, > > > 2015-07-06 07:08:20,721 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) Begin cleanup expired async-jobs > 2015-07-06 07:08:20,733 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-415ba06d) End cleanup expired async-jobs > 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) External devices stats collector is > running... > 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 204 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 219 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 220 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 211 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 212 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 213 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 222 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 223 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 215 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (RouterMonitor-1:ctx-91d5dc5d) Found 10 running routers. > 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 216 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (RouterStatusMonitor-1:ctx-50b5b42f) Found 11 routers to update status. > 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (RouterStatusMonitor-1:ctx-50b5b42f) Found 0 networks to update RvR status. > 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] > (ExternalNetworkMonitor-1:ctx-8a22f14d) Network 217 is not configured for > external networking, so skipping usage check. > 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Executing request > 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-284:ctx-0e4c6018) Executing command in VR: > /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g > 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] > (SnapshotPollTask:ctx-60440aa6) Snapshot scheduler.poll is being called at > 2015-07-06 05:08:21 GMT > 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] > (SnapshotPollTask:ctx-60440aa6) Got 0 snapshots to be executed at 2015-07-06 > 05:08:21 GMT > 2015-07-06 07:08:21,337 WARN [c.c.h.x.r.XenServer56Resource] > (DirectAgent-284:ctx-0e4c6018) Failed to get network usage stats due to > java.lang.NumberFormatException: For input string: "iptables" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:441) > at java.lang.Long.<init>(Long.java:702) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Response Received: > 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] > (DirectAgent-284:ctx-0e4c6018) Seq 5-6790865288121300428: Processing: { Ans: > , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: > java.lang.NumberFormatException\nMessage: For input string: > \"iptables\"\nStack: java.lang.NumberFormatException: For input string: > \"iptables\"\n\tat > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat > java.lang.Long.parseLong(Long.java:441)\n\tat > java.lang.Long.<init>(Long.java:702)\n\tat > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat > > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat > > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat > > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat > > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat > java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat > java.lang.Thread.run(Thread.java:744)\n","wait":0}}] } > 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] > (RouterMonitor-1:ctx-91d5dc5d) Seq 5-6790865288121300428: Received: { Ans: , > MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { NetworkUsageAnswer } } > 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] > (RouterMonitor-1:ctx-91d5dc5d) Details from executing class > com.cloud.agent.api.NetworkUsageCommand: Exception: > java.lang.NumberFormatException > Message: For input string: "iptables" > Stack: java.lang.NumberFormatException: For input string: "iptables" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Long.parseLong(Long.java:441) > at java.lang.Long.<init>(Long.java:702) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > what exactly is this error? > > /sonali >
