Hi Wido, When we had similar issues last year, we found that for example comparing the iptables rules one-by-one is 1000x slower than simply loading them all at once. Boris rewrote this part in our Cosmic fork, may be worth looking into this again. The PR to CloudStack was merged, but reverted later, can't remember why. We run it in production ever since. Also feeding passwords to the passwd server is very inefficient (it operates like a snowball and gets slower once you have more VMs). That we also fixed in Cosmic, not sure if that patch made it upstream. Wrote it about a year ago already.
We tested applying 10K iptables rules in just a couple of seconds. 1000 VMs takes a few minutes to deploy. Generally speaking I'd suggest looking at the logs to find what takes long or is executed a lot of times. Iptables and passwd are two to look at. If you want I can lookup the patches. Not handy on my phone now ;-) Regards, Remi ________________________________ From: Wido den Hollander <w...@widodh.nl> Sent: Tuesday, May 2, 2017 7:57:08 PM To: dev@cloudstack.apache.org Subject: Very slow Virtual Router provisioning with 4.9.2.0 Hi, Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well, but the VR provisioning is terribly slow which causes all kinds of problems. The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq, add metadata, etc. But for just 1800 hosts this can take up to 2 hours and that causes timeouts in the management server and other problems. 2 hours is just very, very slow. So I am starting to wonder if something is wrong here. Did anybody else see this? Running Basic Networking with CloudStack 4.9.2.0 Wido
