Can you move the routerVM on the same hypervisor as guest VM (or guest VM to same hypervisor as routerVM)? If it works, then move the routerVM out to another hypervisor within the same cluster (but same switch).
Are you running vSphere? I've seen similar problem where ARPs would not make it thru due to some intricacy with VMware and Cisco (or perhaps Juniper) switch upstream. Solution was to console in to vRouter and ping a host 2 hops aways. That would fix ARP issue (albeit temporary). Let me know if it helps, Regards ilya On 11/7/16 12:35 PM, Cloud List wrote: > Hi Chiradeep and Wei, thanks for your reply. > > Wei, here's the result you requested: > > root@r-4155-VM:/var/log2# cat /etc/dnsmasq.conf |grep -v "^#" |grep -v "^$" > domain-needed > bogus-priv > resolv-file=/etc/dnsmasq-resolv.conf > local=/cs1cloud.internal/ > except-interface=eth1 > except-interface=eth2 > except-interface=lo > no-dhcp-interface=eth1 > no-dhcp-interface=eth2 > expand-hosts > domain=cs1cloud.internal > domain=cs1cloud.internal > domain=cs1cloud.internal > dhcp-range=X.Y.202.1,static > dhcp-hostsfile=/etc/dhcphosts.txt > dhcp-ignore=tag:!known > dhcp-option=15,"cs1cloud.internal" > dhcp-option=vendor:MSFT,2,1i > dhcp-boot=pxelinux.0 > enable-tftp > tftp-root=/opt/tftpboot > dhcp-lease-max=2100 > domain=cs1cloud.internal > log-dhcp > log-facility=/var/log2/dnsmasq.log > conf-dir=/etc/dnsmasq.d > dhcp-optsfile=/etc/dhcpopts.txt > dhcp-option=option:router,X.Y.202.1 > dhcp-option=6,X.Y.202.2,8.8.8.8,8.8.4.4 > dhcp-client-update > > We actually have 3 class-C subnets assigned to the network. > > X.Y.202.0/24 > X.Y.203.0/24 > Z.A.107.0/24 > > After enabling log-dhcp as per Chiradeep Vittal, I can see that the > available DHCP subnet is only the first subnet. > > Nov 7 20:27:49 dnsmasq-dhcp[22462]: 1979424125 available DHCP subnet: > X.Y.202.2/255.255.255.0 > Nov 7 20:27:49 dnsmasq-dhcp[22462]: 1979424125 available DHCP subnet: > X.Y.202.1/255.255.255.0 > Nov 7 20:27:49 dnsmasq-dhcp[22462]: 1979424125 client provides name: > Debian-81-64b > Nov 7 20:27:49 dnsmasq-dhcp[22462]: 1979424125 DHCPDISCOVER(eth0) > 06:c5:38:01:13:40 ignored > Nov 7 20:27:51 dnsmasq-dhcp[22462]: 2670909966 available DHCP subnet: > X.Y.202.2/255.255.255.0 > Nov 7 20:27:51 dnsmasq-dhcp[22462]: 2670909966 available DHCP subnet: > X.Y.202.1/255.255.255.0 > Nov 7 20:27:51 dnsmasq-dhcp[22462]: 2670909966 client provides name: > WIN-H4INMOBBRJA > Nov 7 20:27:51 dnsmasq-dhcp[22462]: 2670909966 vendor class: MSFT 5.0 > Nov 7 20:27:51 dnsmasq-dhcp[22462]: 2670909966 DHCPDISCOVER(eth0) > 06:31:ac:01:13:f0 ignored > > Coincidentally, all affected VMs are coming from the second subnet: > X.Y.203.0/24, while the third subnet is rarely used. > > Do we need to specifically include the second and third subnets into the > dnsmasq.conf file? I tried adding below line: > > dhcp-range=X.Y.203.1,static > > so it will become: > > dhcp-range=X.Y.202.1,static > dhcp-range=X.Y.203.1,static > > but it doesn't seem to work. > > Any advice is highly appreciated. > > Thank you. > > On Tue, Nov 8, 2016 at 4:19 AM, Wei ZHOU <ustcweiz...@gmail.com> wrote: > >> can you paste the result of following command? >> >> cat /etc/dnsmasq.conf |grep -v "^#" |grep -v "^$" >> >> -Wei >> >> >> 2016-11-07 20:27 GMT+01:00 Cloud List <cloud-l...@sg.or.id>: >> >>> Hi Wei, >>> >>> In addition, >>> >>> The VR is serving a shared not isolated network, meaning the IP it serves >>> is 'guest' not 'public' IP. Will that make a difference on the iptables >>> command we need to execute? >>> >>> Looking forward to your reply, thank you. >>> >>> Cheers. >>> >>> >>> On Tue, Nov 8, 2016 at 3:19 AM, Cloud List <cloud-l...@sg.or.id> wrote: >>> >>>> Hi Wei and Ozhan, >>>> >>>> Thanks for your reply. >>>> >>>> The problem doesn't affect only Debian-based guest VMs, but also >> affected >>>> some Windows and Ubuntu-based VMs as well. I have executed the command >> on >>>> the VR and reset the NIC of the guest VM, but unfortunately the issue >>> still >>>> persists. >>>> >>>> iptables -t mangle -A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM >>>> --checksum-fill >>>> >>>> After issuing the above command on VR and reset the NIC on guest vm >>>> (ifdown eth0, ifup eth0): >>>> >>>> On VR's /var/log/dnsmasq.log: >>>> >>>> Nov 7 19:09:22 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >> 06:b1:22:01:13:17 >>>> ignored >>>> Nov 7 19:09:25 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >> 06:b1:22:01:13:17 >>>> ignored >>>> Nov 7 19:09:30 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >> 06:b1:22:01:13:17 >>>> ignored >>>> Nov 7 19:09:36 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >> 06:b1:22:01:13:17 >>>> ignored >>>> >>>> On the guest VM: >>>> >>>> root@vm:~# ifup eth0 >>>> Internet Systems Consortium DHCP Client 4.2.2 >>>> Copyright 2004-2011 Internet Systems Consortium. >>>> All rights reserved. >>>> For info, please visit https://www.isc.org/software/dhcp >>>> >>>> Listening on LPF/eth0/06:b1:22:01:13:17 >>>> Sending on LPF/eth0/06:b1:22:01:13:17 >>>> Sending on Socket/fallback >>>> DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4 >>>> DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5 >>>> DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 64DHCPDISCOVER >>> on >>>> eth0 to 255.255.255.255 port 67 interval 14 >>>> DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9 >>>> DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10 >>>> DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13 >>>> No DHCOFFERS received. >>>> No working leases in persistent database - sleeping. >>>> >>>> I also tried to execute similar hotfix as mentioned on the bug report: >>>> >>>> iptables -A POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM >>>> --checksum-fill >>>> >>>> The problem still persists. Any further advice on this is highly >>>> appreciated. >>>> >>>> Looking forward to your reply, thank you. >>>> >>>> Cheers. >>>> >>>> >>>> On Tue, Nov 8, 2016 at 2:41 AM, Wei ZHOU <ustcweiz...@gmail.com> >> wrote: >>>> >>>>> GOOD point. >>>>> >>>>> @CloudList, can you try again after executing the following command in >>> VR >>>>> ? >>>>> >>>>> iptables -t mangle -A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM >>>>> --checksum-fill >>>>> >>>>> -Wei >>>>> >>>>> 2016-11-07 14:46 GMT+01:00 Özhan Rüzgar Karaman < >>> oruzgarkara...@gmail.com >>>>>> : >>>>> >>>>>> Hi; >>>>>> Whats your problematic vm's type is it Debian? Maybe you are >> affected >>>>> from >>>>>> the bug CLOUDSTACK-8326? I do not know if this bug has effected on >> ACS >>>>> 4.2 >>>>>> release but i know that it effects release 4.8.x 4.9.x >>>>>> >>>>>> Thanks >>>>>> Özhan >>>>>> >>>>>> On Mon, Nov 7, 2016 at 4:36 PM, Cloud List <cloud-l...@sg.or.id> >>> wrote: >>>>>> >>>>>>> Hi Wei, >>>>>>> >>>>>>> Thanks for your reply. >>>>>>> >>>>>>> I checked and I can confirm that the mac address is listed on >>>>>>> /etc/dhcphosts.txt in VR. >>>>>>> >>>>>>> For example: >>>>>>> >>>>>>> Nov 7 13:30:19 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >>>>> 06:31:ac:01:13:YY >>>>>>> ignored >>>>>>> Nov 7 13:30:24 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >>>>> 06:31:ac:01:13:YY >>>>>>> ignored >>>>>>> Nov 7 13:30:33 dnsmasq-dhcp[18986]: DHCPDISCOVER(eth0) >>>>> 06:31:ac:01:13:YY >>>>>>> ignored >>>>>>> >>>>>>> root@r-4155-VM:/var/log# grep 06:31:ac:01:13:f0 >> /etc/dhcphosts.txt >>>>>>> 06:31:ac:01:13:YY,X.X.X.X,vm-name,infinite >>>>>>> >>>>>>> YY - last two digits of the mac address >>>>>>> X.X.X.X - ip address which is supposed to be allocated to the VM >>>>>>> >>>>>>> Any advice how can I troubleshoot this further? >>>>>>> >>>>>>> Looking forward to your reply, thank you. >>>>>>> >>>>>>> Cheers. >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 7, 2016 at 9:22 PM, Wei ZHOU <ustcweiz...@gmail.com> >>>>> wrote: >>>>>>> >>>>>>>> If the mac address is not listed in /etc/dhcphosts.txt in VR, >> the >>>>>> request >>>>>>>> will be ignored. >>>>>>>> >>>>>>>> Can you give more details so we can reproduce it and fix it ? >>>>>>>> >>>>>>>> -Wei >>>>>>>> >>>>>>>> 2016-11-07 13:44 GMT+01:00 Cloud List <cloud-l...@sg.or.id>: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> After our upgrade from CloudStack 4.2 to 4.8.1.1, we noted >> that >>>>> some >>>>>>> (but >>>>>>>>> not all) of our VMs are not able to get DHCP from the VR. This >>>>> gives >>>>>>>>> problem when the VM is restarted and cannot get up and running >>>>>> because >>>>>>>>> unable to get IP. >>>>>>>>> >>>>>>>>> I logged in to the VR and found below messages showing that >> the >>>>> DHCP >>>>>>>> server >>>>>>>>> is ignoring the request from the VM. >>>>>>>>> >>>>>>>>> Nov 7 12:19:43 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:19:52 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:05:64:01:13:d3 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:19:53 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:20:04 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:20:11 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:05:64:01:13:d3 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:20:22 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:20:30 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:20:42 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:21:02 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> Nov 7 12:21:19 dnsmasq-dhcp[29938]: DHCPDISCOVER(eth0) >>>>>>> 06:44:da:01:13:98 >>>>>>>>> ignored >>>>>>>>> >>>>>>>>> Any reason why the dnsmasq service ignored the request from >> the >>> VM >>>>>> and >>>>>>>> how >>>>>>>>> can we fix that? >>>>>>>>> >>>>>>>>> At the moment, the only workaround we can do is to configure >> the >>>>> IP >>>>>>>> address >>>>>>>>> statically to the servelet, which is not practical. >>>>>>>>> >>>>>>>>> Any advice is greatly appreciated. >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >
