Hi Ozhan,
During the 4.11-systemvmtemplate migration work (to debian9 based template), I refactored the code to reload dnsmasq instead of restart it. Based on your feedback, I've created a fix that will remove the leases file everytime dnsmasq needs to be restarted. Can you help test/verify it: https://github.com/apache/cloudstack/pull/2427/files - Rohit <https://cloudstack.apache.org> ________________________________ From: Özhan Rüzgar Karaman <oruzgarkara...@gmail.com> Sent: Tuesday, January 23, 2018 1:01:16 PM To: dev@cloudstack.apache.org Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file Hi; We made some more tests to find the root cause of the problem. The problem occur because we crashed(power off) VR for a 4.11 HA test. After rebooting VR, active VM's dhcp lease datas are stucked in /var/lib/misc/dnsmasq.leases file and this creates problems. Our solution is to clear/flush dnsmasq file using dhcp_release command. After dhcp_release command problem disappears for new VMs with same ip address. So i think we need to add some code on VR startup to flush dnsmasq.leases file before starting dnsmasq. Thanks Özhan On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman < oruzgarkara...@gmail.com> wrote: > Hi Ivan; > I am not sure PR 2393 directly points to my findings, i only tested this > scenario on 4.11rc1. > > I am not a developer so i will not submit a fix, i am only testing 4.11rc > because its a LTS release and its quality is very important. > > Please check the issue on your environment, all details and issue > reproducing steps are written on my first email, but if you want i will > create a PR only to report & record the situation, just send me message if > you want. > > Thanks > Özhan > > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev < > kudryavtsev...@bw-sw.com> wrote: > >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't >> have >> patched code? Also, describe testing scenario, I'll try to look at it in >> my >> patched 4.10. >> >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" < >> oruzgarkara...@gmail.com> написал: >> >> > Hi Ivan; >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists >> on my >> > environment. When a new vm uses IP from old expunged vm then leases file >> > creates problem. Please check the logs that i submitted on first email, >> > issue is clear there and in my opinion it still exists on 4.11rc1. >> > >> > By the way 2393 is about VM's IP Changing progress, maybe it does not >> cover >> > my scenario. >> > >> > Thanks >> > Özhan >> > >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman < >> > oruzgarkara...@gmail.com> wrote: >> > >> > > Hi Ivan; >> > > I made several tests with same scenario on 4.11rc1 and got same >> results, >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing or >> it >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a >> > problem >> > > >> > > Thanks >> > > Özhan >> > > >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev < >> > > kudryavtsev...@bw-sw.com> wrote: >> > > >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are >> overriden >> > >> upon vm creation with same ip (or same hostname). It should work >> fine, >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the >> > case >> > >> when IP is reused. >> > >> >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" < >> > >> oruzgarkara...@gmail.com> написал: >> > >> >> > >> Hi; >> > >> Today we noticed that one of our new provisioned instance did not >> get IP >> > >> from VR. When we dig into the issue we find that one different mac is >> > >> written in dnsmasq.leases file holds new instances IP address. >> > >> >> > >> We checked this mac address from db and we noticed that this mac is >> used >> > >> for old expunged instance. >> > >> >> > >> So from this point we realised that when we destroy an instance its >> mac >> > >> did >> > >> not removed from dnsmasq.leases file so if we use this ip for a new >> > >> instance then we have a problem, our instance could not get IP from >> VR. >> > >> >> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM. >> > Today >> > >> we made a HA test and we crashed the host so VR and SystemVM's are >> > >> rebooted >> > >> after we boot host back. I do not think this issue is related to VR >> > reboot >> > >> but i like to give information about our environment. >> > >> >> > >> We need to manage dnsmasq.leases file when we expunge an instance. >> > >> >> > >> Thanks >> > >> Özhan >> > >> >> > >> Logs are below: >> > >> >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9 >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) >> 1e:00:00:00:00:b9 >> > no >> > >> address available >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) >> 1e:00:9f:00:00:ba >> > no >> > >> address available >> > >> >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip * >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 * >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac * >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest * >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 * >> > >> >> > >> mysql> select name,state,private_mac_address,private_ip_address from >> > >> vm_instance; >> > >> +---------------+-----------+---------------------+--------- >> > -----------+ >> > >> | name | state | private_mac_address | >> private_ip_address | >> > >> +---------------+-----------+---------------------+--------- >> > -----------+ >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 >> | >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 >> | >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 >> | >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 >> | >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 >> | >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 >> | >> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 >> | >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 >> | >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 >> | >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 >> | >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 >> | >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 >> | >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 >> | >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 >> | >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 >> | >> > >> +---------------+-----------+---------------------+--------- >> > -----------+ >> > >> 15 rows in set (0.00 sec) >> > >> >> > > >> > > >> > >> > > rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
