Hi Wido,
A simple improvement is, donot wait while restarting dnsmasq service in VR.
'''
diff --git a/systemvm/patches/debian/config/opt/cloud/bin/cs/CsDhcp.py
b/systemvm/patches/debian/config/opt/cloud/bin/cs/CsDhcp.py
index 95d2eff..999be8f 100755
--- a/systemvm/patches/debian/config/opt/cloud/bin/cs/CsDhcp.py
+++ b/systemvm/patches/debian/config/opt/cloud/bin/cs/CsDhcp.py
@@ -59,7 +59,7 @@ class CsDhcp(CsDataBag):
# We restart DNSMASQ every time the configure.py is called in
order to avoid lease problems.
if not self.cl.is_redundant() or self.cl.is_master():
- CsHelper.service("dnsmasq", "restart")
+ CsHelper.execute3("service dnsmasq restart")
def configure_server(self):
# self.conf.addeq("dhcp-hostsfile=%s" % DHCP_HOSTS)
diff --git a/systemvm/patches/debian/config/opt/cloud/bin/cs/CsHelper.py
b/systemvm/patches/debian/config/opt/cloud/bin/cs/CsHelper.py
index a8ccea2..b06bde3 100755
--- a/systemvm/patches/debian/config/opt/cloud/bin/cs/CsHelper.py
+++ b/systemvm/patches/debian/config/opt/cloud/bin/cs/CsHelper.py
@@ -191,6 +191,11 @@ def execute2(command):
p.wait()
return p
+def execute3(command):
+ """ Execute command """
+ logging.debug("Executing: %s" % command)
+ p = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
+ return p
def service(name, op):
execute("service %s %s" % (name, op))
'''
-Wei
2017-05-04 10:48 GMT+02:00 Wido den Hollander <w...@widodh.nl>:
> Thanks Daan, Remi.
>
> I found a additional bug where it seems that 'network.dns.basiczone.updates'
> isn't read when sending DHCP settings in Basic Networking.
>
> This means that the VR gets all DHCP setting for the whole zone instead of
> just for that POD.
>
> In this case some VRs we have get ~2k of DHCP offerings send to them which
> causes a large slowdown.
>
> Wido
>
> > Op 3 mei 2017 om 14:49 schreef Daan Hoogland <daan.hoogl...@gmail.com>:
> >
> >
> > Happy to pick this up, Remi. I'm travelling now but will look at both on
> > Friday.
> >
> > Biligual auto correct use. Read at your own risico
> >
> > On 3 May 2017 2:25 pm, "Remi Bergsma" <rberg...@schubergphilis.com>
> wrote:
> >
> > > Always happy to share, but I won’t have time to work on porting this to
> > > CloudStack any time soon.
> > >
> > > Regards, Remi
> > >
> > >
> > > On 03/05/2017, 13:44, "Rohit Yadav" <rohit.ya...@shapeblue.com> wrote:
> > >
> > > Hi Remi, thanks for sharing. We would love to have those changes
> (for
> > > 4.9+), looking forward to your pull requests.
> > >
> > >
> > > Regards.
> > >
> > > ________________________________
> > > From: Remi Bergsma <rberg...@schubergphilis.com>
> > > Sent: 03 May 2017 16:58:18
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: Very slow Virtual Router provisioning with 4.9.2.0
> > >
> > > Hi,
> > >
> > > The patches I talked about:
> > >
> > > 1) Iptables speed improvement
> > > https://github.com/apache/cloudstack/pull/1482
> > > Was reverted due to a licensing issue.
> > >
> > > 2) Passwd speed improvement
> > > https://github.com/MissionCriticalCloudOldRepos/
> cosmic-core/pull/138
> > >
> > > By now, these are rather old patches so they need some work before
> > > they apply to CloudStack again.
> > >
> > > Regards, Remi
> > >
> > >
> > >
> > > On 03/05/2017, 12:49, "Jeff Hair" <j...@greenqloud.com> wrote:
> > >
> > > Hi Remi,
> > >
> > > Do you have a link to the PR that was reverted? And also
> possibly
> > > the code
> > > that makes the password updating more efficient?
> > >
> > > Jeff
> > >
> > > On Wed, May 3, 2017 at 10:36 AM, Remi Bergsma <
> > > rberg...@schubergphilis.com>
> > > wrote:
> > >
> > > > Hi Wido,
> > > >
> > > > When we had similar issues last year, we found that for
> example
> > > comparing
> > > > the iptables rules one-by-one is 1000x slower than simply
> > > loading them all
> > > > at once. Boris rewrote this part in our Cosmic fork, may be
> > > worth looking
> > > > into this again. The PR to CloudStack was merged, but
> reverted
> > > later, can't
> > > > remember why. We run it in production ever since. Also
> feeding
> > > passwords to
> > > > the passwd server is very inefficient (it operates like a
> > > snowball and gets
> > > > slower once you have more VMs). That we also fixed in Cosmic,
> > > not sure if
> > > > that patch made it upstream. Wrote it about a year ago
> already.
> > > >
> > > > We tested applying 10K iptables rules in just a couple of
> > > seconds. 1000
> > > > VMs takes a few minutes to deploy.
> > > >
> > > > Generally speaking I'd suggest looking at the logs to find
> what
> > > takes long
> > > > or is executed a lot of times. Iptables and passwd are two to
> > > look at.
> > > >
> > > > If you want I can lookup the patches. Not handy on my phone
> now
> > > ;-)
> > > >
> > > > Regards, Remi
> > > > ________________________________
> > > > From: Wido den Hollander <w...@widodh.nl>
> > > > Sent: Tuesday, May 2, 2017 7:57:08 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Very slow Virtual Router provisioning with 4.9.2.0
> > > >
> > > > Hi,
> > > >
> > > > Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0.
> All
> > > went well,
> > > > but the VR provisioning is terribly slow which causes all
> kinds
> > > of problems.
> > > >
> > > > The vr_cfg.sh and update_config.py scripts start to run.
> Restart
> > > dnsmasq,
> > > > add metadata, etc.
> > > >
> > > > But for just 1800 hosts this can take up to 2 hours and that
> > > causes
> > > > timeouts in the management server and other problems.
> > > >
> > > > 2 hours is just very, very slow. So I am starting to wonder
> if
> > > something
> > > > is wrong here.
> > > >
> > > > Did anybody else see this?
> > > >
> > > > Running Basic Networking with CloudStack 4.9.2.0
> > > >
> > > > Wido
> > > >
> > >
> > >
> > >
> > >
> > > rohit.ya...@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > >
> > >
>
