Hi Rohit; Today i am short in time but tomorrow i will create a new network and test your fix over this fresh VR.
I have one more question, with your current code fix do we still continue reloading dnsmasq on normal operations and we only flush leases on start/restart operations or after this fix we start to use restart the dnsmasq instead of reloading it on all our operations? Thanks for all your help. Özhan On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi Ozhan, > > > During the 4.11-systemvmtemplate migration work (to debian9 based > template), I refactored the code to reload dnsmasq instead of restart it. > Based on your feedback, I've created a fix that will remove the leases file > everytime dnsmasq needs to be restarted. > > > Can you help test/verify it: > > https://github.com/apache/cloudstack/pull/2427/files > > > > - Rohit > > <https://cloudstack.apache.org> > > > > ________________________________ > From: Özhan Rüzgar Karaman <oruzgarkara...@gmail.com> > Sent: Tuesday, January 23, 2018 1:01:16 PM > To: dev@cloudstack.apache.org > Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from > dnsmasq.leases file > > Hi; > We made some more tests to find the root cause of the problem. The problem > occur because we crashed(power off) VR for a 4.11 HA test. After rebooting > VR, active VM's dhcp lease datas are stucked > in /var/lib/misc/dnsmasq.leases file and this creates problems. > > Our solution is to clear/flush dnsmasq file using dhcp_release command. > After dhcp_release command problem disappears for new VMs with same ip > address. > > So i think we need to add some code on VR startup to flush dnsmasq.leases > file before starting dnsmasq. > > Thanks > Özhan > > On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman < > oruzgarkara...@gmail.com> wrote: > > > Hi Ivan; > > I am not sure PR 2393 directly points to my findings, i only tested this > > scenario on 4.11rc1. > > > > I am not a developer so i will not submit a fix, i am only testing 4.11rc > > because its a LTS release and its quality is very important. > > > > Please check the issue on your environment, all details and issue > > reproducing steps are written on my first email, but if you want i will > > create a PR only to report & record the situation, just send me message > if > > you want. > > > > Thanks > > Özhan > > > > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev < > > kudryavtsev...@bw-sw.com> wrote: > > > >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't > >> have > >> patched code? Also, describe testing scenario, I'll try to look at it in > >> my > >> patched 4.10. > >> > >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" < > >> oruzgarkara...@gmail.com> написал: > >> > >> > Hi Ivan; > >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists > >> on my > >> > environment. When a new vm uses IP from old expunged vm then leases > file > >> > creates problem. Please check the logs that i submitted on first > email, > >> > issue is clear there and in my opinion it still exists on 4.11rc1. > >> > > >> > By the way 2393 is about VM's IP Changing progress, maybe it does not > >> cover > >> > my scenario. > >> > > >> > Thanks > >> > Özhan > >> > > >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman < > >> > oruzgarkara...@gmail.com> wrote: > >> > > >> > > Hi Ivan; > >> > > I made several tests with same scenario on 4.11rc1 and got same > >> results, > >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing > or > >> it > >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a > >> > problem > >> > > > >> > > Thanks > >> > > Özhan > >> > > > >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev < > >> > > kudryavtsev...@bw-sw.com> wrote: > >> > > > >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are > >> overriden > >> > >> upon vm creation with same ip (or same hostname). It should work > >> fine, > >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested > the > >> > case > >> > >> when IP is reused. > >> > >> > >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" < > >> > >> oruzgarkara...@gmail.com> написал: > >> > >> > >> > >> Hi; > >> > >> Today we noticed that one of our new provisioned instance did not > >> get IP > >> > >> from VR. When we dig into the issue we find that one different mac > is > >> > >> written in dnsmasq.leases file holds new instances IP address. > >> > >> > >> > >> We checked this mac address from db and we noticed that this mac is > >> used > >> > >> for old expunged instance. > >> > >> > >> > >> So from this point we realised that when we destroy an instance its > >> mac > >> > >> did > >> > >> not removed from dnsmasq.leases file so if we use this ip for a new > >> > >> instance then we have a problem, our instance could not get IP from > >> VR. > >> > >> > >> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM. > >> > Today > >> > >> we made a HA test and we crashed the host so VR and SystemVM's are > >> > >> rebooted > >> > >> after we boot host back. I do not think this issue is related to VR > >> > reboot > >> > >> but i like to give information about our environment. > >> > >> > >> > >> We need to manage dnsmasq.leases file when we expunge an instance. > >> > >> > >> > >> Thanks > >> > >> Özhan > >> > >> > >> > >> Logs are below: > >> > >> > >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address > >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9 > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) > >> 1e:00:00:00:00:b9 > >> > no > >> > >> address available > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address > >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) > >> 1e:00:9f:00:00:ba > >> > no > >> > >> address available > >> > >> > >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt > >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h > >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h > >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h > >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h > >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h > >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases > >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip * > >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 * > >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac * > >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest * > >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 * > >> > >> > >> > >> mysql> select name,state,private_mac_address,private_ip_address > from > >> > >> vm_instance; > >> > >> +---------------+-----------+---------------------+--------- > >> > -----------+ > >> > >> | name | state | private_mac_address | > >> private_ip_address | > >> > >> +---------------+-----------+---------------------+--------- > >> > -----------+ > >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 > >> | > >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 > >> | > >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 > >> | > >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 > >> | > >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 > >> | > >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 > >> | > >> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 > >> | > >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 > >> | > >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 > >> | > >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 > >> | > >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 > >> | > >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 > >> | > >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 > >> | > >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 > >> | > >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 > >> | > >> > >> +---------------+-----------+---------------------+--------- > >> > -----------+ > >> > >> 15 rows in set (0.00 sec) > >> > >> > >> > > > >> > > > >> > > >> > > > > > > rohit.ya...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > >
