Silly question but Is your xenserver configured bridge mode related settings correctly ?
#xe-switch-network-backend bridge #echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables #echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables Thanks, Jayapal On 27-Mar-2015, at 1:50 PM, cs user <acldstk...@gmail.com<mailto:acldstk...@gmail.com>> wrote: Hi Somesh, arping looks good, the correct mac address is displayed and we get a unicast reply from the ip address. Erik, tried restarting dnsmasq, all looks fine. VR is able to perform outgoing dns requests. There is nothing in syslog/dnsmasq logs that I can see. No egress rules are in place. The system vm's are able to perform dig's against google's dns, but not the virtual router. It seems it is being blocked at the xen level. We're seeing the below in the logs when restarting a network (either ticking clear config or not). This appears to be similar to : https://issues.apache.org/jira/browse/CLOUDSTACK-7605 We are using basic zones, some have multiple pods, others don't. We see the same error in both. The routers come up though and go green, and dnsmasq is populated with the relevant info. DNS lookups work locally on the router, just not remotely. DHCP is working for new machines which get spun up. Is there a way to debug this? I've checked the logs on the router (cloud.log) can't see any errors in there. 2015-03-27 08:12:45,081 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Implementing the network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] elements and resources as a part of network restart 2015-03-27 08:12:45,096 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking SecurityGroupProvider to implemenet Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] 2015-03-27 08:12:45,103 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Asking VirtualRouter to implemenet Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] 2015-03-27 08:12:45,112 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is acquired for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()] 2015-03-27 08:12:45,119 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Skipping VR deployment: Found a running or starting VR in Pod null id=8 2015-03-27 08:12:45,120 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Lock is released for network id 204 as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(8)-Pod(null)-Cluster(null)-Host(null)-Storage()] 2015-03-27 08:12:45,123 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Failed to implement network Ntwk[9f5655bf-3101-45d9-83eb-d9061eadc2bb|Guest|47] elements and resources as a part of network restart due to java.lang.NullPointerException at com.cloud.network.element.VirtualRouterElement.getRouters(VirtualRouterElement.java:952) at com.cloud.network.element.VirtualRouterElement.prepareAggregatedExecution(VirtualRouterElement.java:1099) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1090) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430) at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy156.restartNetwork(Unknown Source) at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2015-03-27 08:12:45,125 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-16:ctx-0b0aa78a job-189235 ctx-77114e2e) Network id=204 failed to restart. 2015-03-27 08:12:45,140 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-16:ctx-0b0aa78a job-189235) Complete async job-189235, jobStatus: FAILED, resultCode: 530, result:org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to restart network"} 2015-03-27 08:12:45,152 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-16:ctx-0b0aa78a job-189235) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-189235 2015-03-27 08:12:45,158 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-16:ctx-0b0aa78a job-189235) Remove job-189235 from job monitoring On Thu, Mar 26, 2015 at 7:38 PM, Somesh Naidu <somesh.na...@citrix.com> wrote: You might want to do a arpping on the router's IP from one of the guests and see how many records are returned. Somesh CloudPlatform Escalations Citrix Systems, Inc. -----Original Message----- From: Erik Weber [mailto:terbol...@gmail.com] Sent: Thursday, March 26, 2015 12:54 PM To: users@cloudstack.apache.org Subject: Re: Virtual Routers not responding to dns requests I briefly remember having similar problems at some point, but do not recall details as version nor the solution. 1) Does it work if you restart dnsmasq on the VR? 2) is the VR able to do outgoing dns requests? 3) anything in syslog/dnsmasq logs? 4) any egress rules in place? Erik Den torsdag 26. mars 2015 skrev cs user <acldstk...@gmail.com> følgende: Hi All, We have upgraded from 4.3 to 4.4.2. After some issues with starting the systemvm's, the virtual routers no longer responding to dns requests from the vm's which we start (or existing ones). We have disabled the firewall on the virtual routers and ran tcpdump on them but we can't see any inbound traffic on port 53 (udp or tcp). If we log onto the virtual routers and dig locally against eth0 and the alias on eth0 both of these return fine with the correct IP. This is using Xenserver 6.1 as the host. Has anyone come across this before? dhcp lookups appear to be working fine. Is there a firewall rule in place on the router vm's (other than iptables), similar to the security groups which are applied by Xen, which is preventing these requests from hitting the routers? Many thanks for any help.
