Thanks Ozhan for confirming, the fix will make its way to RC2, which we'll hopefully cut by EOD tomorrow.
- Rohit <https://cloudstack.apache.org> ________________________________ From: Özhan Rüzgar Karaman <oruzgarkara...@gmail.com> Sent: Thursday, January 25, 2018 1:12:07 PM To: dev@cloudstack.apache.org Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file Hi Rohit; I made the test again on a fresh VR and your solution fixed the issue. Thanks Özhan On Wed, Jan 24, 2018 at 11:26 AM, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi Ozhan, > > > With my fix, whenever dnsmasq needs to be restarted the leases file will > be removed, and dnsmasq will be restarted whenever /etc/dhcphosts.txt or > /etc/dnsmasq.d/cloud.conf change, otherwise it will be reloaded. > > > - Rohit > > <https://cloudstack.apache.org> > > > > ________________________________ > From: Özhan Rüzgar Karaman <oruzgarkara...@gmail.com> > Sent: Wednesday, January 24, 2018 7:40:18 AM > To: dev@cloudstack.apache.org > Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from > dnsmasq.leases file > > Hi Rohit; > Today i am short in time but tomorrow i will create a new network and test > your fix over this fresh VR. > > I have one more question, with your current code fix do we still continue > reloading dnsmasq on normal operations and we only flush leases on > start/restart operations or after this fix we start to use restart the > dnsmasq instead of reloading it on all our operations? Thanks for all your > help. > > Özhan > > On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > Hi Ozhan, > > > > > > During the 4.11-systemvmtemplate migration work (to debian9 based > > template), I refactored the code to reload dnsmasq instead of restart it. > > Based on your feedback, I've created a fix that will remove the leases > file > > everytime dnsmasq needs to be restarted. > > > > > > Can you help test/verify it: > > > > https://github.com/apache/cloudstack/pull/2427/files > > > > > > > > - Rohit > > > > <https://cloudstack.apache.org> > > > > > > > > ________________________________ > > From: Özhan Rüzgar Karaman <oruzgarkara...@gmail.com> > > Sent: Tuesday, January 23, 2018 1:01:16 PM > > To: dev@cloudstack.apache.org > > Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from > > dnsmasq.leases file > > > > Hi; > > We made some more tests to find the root cause of the problem. The > problem > > occur because we crashed(power off) VR for a 4.11 HA test. After > rebooting > > VR, active VM's dhcp lease datas are stucked > > in /var/lib/misc/dnsmasq.leases file and this creates problems. > > > > Our solution is to clear/flush dnsmasq file using dhcp_release command. > > After dhcp_release command problem disappears for new VMs with same ip > > address. > > > > So i think we need to add some code on VR startup to flush dnsmasq.leases > > file before starting dnsmasq. > > > > Thanks > > Özhan > > > > On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman < > > oruzgarkara...@gmail.com> wrote: > > > > > Hi Ivan; > > > I am not sure PR 2393 directly points to my findings, i only tested > this > > > scenario on 4.11rc1. > > > > > > I am not a developer so i will not submit a fix, i am only testing > 4.11rc > > > because its a LTS release and its quality is very important. > > > > > > Please check the issue on your environment, all details and issue > > > reproducing steps are written on my first email, but if you want i will > > > create a PR only to report & record the situation, just send me message > > if > > > you want. > > > > > > Thanks > > > Özhan > > > > > > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev < > > > kudryavtsev...@bw-sw.com> wrote: > > > > > >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't > > >> have > > >> patched code? Also, describe testing scenario, I'll try to look at it > in > > >> my > > >> patched 4.10. > > >> > > >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" < > > >> oruzgarkara...@gmail.com> написал: > > >> > > >> > Hi Ivan; > > >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still > exists > > >> on my > > >> > environment. When a new vm uses IP from old expunged vm then leases > > file > > >> > creates problem. Please check the logs that i submitted on first > > email, > > >> > issue is clear there and in my opinion it still exists on 4.11rc1. > > >> > > > >> > By the way 2393 is about VM's IP Changing progress, maybe it does > not > > >> cover > > >> > my scenario. > > >> > > > >> > Thanks > > >> > Özhan > > >> > > > >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman < > > >> > oruzgarkara...@gmail.com> wrote: > > >> > > > >> > > Hi Ivan; > > >> > > I made several tests with same scenario on 4.11rc1 and got same > > >> results, > > >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing > > or > > >> it > > >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a > > >> > problem > > >> > > > > >> > > Thanks > > >> > > Özhan > > >> > > > > >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev < > > >> > > kudryavtsev...@bw-sw.com> wrote: > > >> > > > > >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are > > >> overriden > > >> > >> upon vm creation with same ip (or same hostname). It should work > > >> fine, > > >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested > > the > > >> > case > > >> > >> when IP is reused. > > >> > >> > > >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" < > > >> > >> oruzgarkara...@gmail.com> написал: > > >> > >> > > >> > >> Hi; > > >> > >> Today we noticed that one of our new provisioned instance did not > > >> get IP > > >> > >> from VR. When we dig into the issue we find that one different > mac > > is > > >> > >> written in dnsmasq.leases file holds new instances IP address. > > >> > >> > > >> > >> We checked this mac address from db and we noticed that this mac > is > > >> used > > >> > >> for old expunged instance. > > >> > >> > > >> > >> So from this point we realised that when we destroy an instance > its > > >> mac > > >> > >> did > > >> > >> not removed from dnsmasq.leases file so if we use this ip for a > new > > >> > >> instance then we have a problem, our instance could not get IP > from > > >> VR. > > >> > >> > > >> > >> We have one host on our lab environment and its Ubuntu 16.04.3 > KVM. > > >> > Today > > >> > >> we made a HA test and we crashed the host so VR and SystemVM's > are > > >> > >> rebooted > > >> > >> after we boot host back. I do not think this issue is related to > VR > > >> > reboot > > >> > >> but i like to give information about our environment. > > >> > >> > > >> > >> We need to manage dnsmasq.leases file when we expunge an > instance. > > >> > >> > > >> > >> Thanks > > >> > >> Özhan > > >> > >> > > >> > >> Logs are below: > > >> > >> > > >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log > > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address > > >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9 > > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) > > >> 1e:00:00:00:00:b9 > > >> > no > > >> > >> address available > > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address > > >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba > > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) > > >> 1e:00:9f:00:00:ba > > >> > no > > >> > >> address available > > >> > >> > > >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt > > >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h > > >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h > > >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h > > >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h > > >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h > > >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases > > >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip * > > >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 * > > >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac * > > >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest * > > >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 * > > >> > >> > > >> > >> mysql> select name,state,private_mac_address,private_ip_address > > from > > >> > >> vm_instance; > > >> > >> +---------------+-----------+---------------------+--------- > > >> > -----------+ > > >> > >> | name | state | private_mac_address | > > >> private_ip_address | > > >> > >> +---------------+-----------+---------------------+--------- > > >> > -----------+ > > >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 > > >> | > > >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 > > >> | > > >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | > 192.168.18.189 > > >> | > > >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 > > >> | > > >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | > 192.168.18.192 > > >> | > > >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | > 192.168.18.188 > > >> | > > >> > >> | deneme | Expunging | 1e:00:80:00:00:ba | > 192.168.18.187 > > >> | > > >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | > 192.168.18.186 > > >> | > > >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | > 192.168.18.189 > > >> | > > >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | > 192.168.18.192 > > >> | > > >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | > 192.168.18.188 > > >> | > > >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | > 192.168.18.187 > > >> | > > >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | > 192.168.18.186 > > >> | > > >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | > 192.168.18.186 > > >> | > > >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | > 192.168.18.186 > > >> | > > >> > >> +---------------+-----------+---------------------+--------- > > >> > -----------+ > > >> > >> 15 rows in set (0.00 sec) > > >> > >> > > >> > > > > >> > > > > >> > > > >> > > > > > > > > > > rohit.ya...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > rohit.ya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
