Hi Rafael, These are clients of our support or consulting services, therefore I can only ever talk in very broad terms regarding their architectures.
One client had a single management server, the other had a pair which were load balanced behind a physical appliance. Regards, Paul Angus VP Technology/Cloud Architect S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus paul.an...@shapeblue.com -----Original Message----- From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] Sent: 19 November 2015 22:32 To: dev@cloudstack.apache.org Subject: Re: [STABILITY]} Large KVM Infrastructure with ACS How many MS do you have in your environment? On Thu, Nov 19, 2015 at 7:56 PM, Paul Angus <paul.an...@shapeblue.com> wrote: > Hi, > > In the past a couple of clients of our have had issues with indirect > agents (KVM hosts and system VMs) connecting over port 8250, > particularly if connectivity was lost to the management server(s). > They both had 300+ indirect agents active. > > In these circumstances we have found that running a netstat to see > connections to port 8250 on the mgmt server(s) revealed many open but > unused connections to port 8250. > > I recall at one time we found the agent connection code had been > altered to attempt to reconnect it the connection didn't complete with 10secs. > However the failed connection would take 60 seconds to time out. > > Another time we found that management server and mysql db were both > being starved of enough connections to the mysql db to process the > reconnections faster enough. The default from the mgmt server is 100 > connections and the documented setting for mysql is 350 connections. > However external connections (and additional mgmt servers) require these to > be adjusted. > > -- just some ideas... > > > Regards, > > Paul Angus > VP Technology/Cloud Architect > S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus > paul.an...@shapeblue.com > > -----Original Message----- > From: ilya [mailto:ilya.mailing.li...@gmail.com] > Sent: 19 November 2015 20:32 > To: dev@cloudstack.apache.org > Subject: Re: [STABILITY]} Large KVM Infrastructure with ACS > > Rafael, > > Please see response in-line: > > On 11/18/15 4:16 PM, Rafael Weingärtner wrote: > > When you say 250+, you mean 250+ host spread in lots of cluster, right? > > If I am not mistaken, ACS limits the number of KVM hosts in a > > cluster, something like 50? I do not remember now if that value can > > be configured, may it can be. > > Yes lots of clusters, way less than 50 per cluster. > > > I recall to have read something in a Red Hat doc about the KVM that > > it does not have limit of hosts in a cluster. Actually, it does not > > seem to have the figure of cluster at all. That is created solely in > > ACS, to facilitate the management. > > > > To debug the problem, I would start with the following questions: > > > > Is every single cluster of your environment is presenting that problem? > > No, few clusters with some nodes within the cluster - not all. > > > What is the size of physical hosts that you have in your environment? > > Do all of them have the same configuration? > Yes, all hosts have the same configuration. Cant go into details, but > its rather large. > > > Do you know the load (resource allocated and used) that is being > > imposed in those hosts that had shown those problems? > > What is your over commitment/provisioning factor that you are using? > Servers are not heavily taxed, we dont over commit memory, other > components could be over committed by 2 or less. Overall, we still > have capacity to accommodate more VMs if needed, we just don't max it out. > > ---- > > Both Marcus and myself are looking through this, it could be just our > specific implementation - hence, I wanted to see if anyone else in the > community with heavy KVM usage came across this issue. > > Maybe I need to ping LeaseWeb and ExtremePC folks.. > > Thanks, > ilya > > > > On Wed, Nov 18, 2015 at 8:19 PM, Daan Hoogland > > <daan.hoogl...@gmail.com> > > wrote: > > > >> sounds like a bad limit Ilya, i'll keep an eye out. > >> > >> On Wed, Nov 18, 2015 at 10:10 PM, ilya > >> <ilya.mailing.li...@gmail.com> > >> wrote: > >> > >>> I'm curious if anyone runs ACS with atleast 250+ KVM hosts. > >>> > >>> We've been noticing weird issues with KVM where occasionally lots > >>> of KVM agents get Nio connection closed issue followed by barrage > >>> of > alerts. > >>> > >>> In some instances the agent reconnects right away and in other - > >>> it attempts to reconnect but never receives an ACK from MS. > >>> > >>> Please let me know if you notice anything like it and if you found > >>> a solution. > >>> > >>> Also, it would help to know what global settings have been tuned > >>> to make things work better (aside from direct.agent.*) and how MS > >>> are > running. > >>> > >>> Thanks > >>> ilya > >>> > >> > >> > >> > >> -- > >> Daan > >> > > > > > > > Find out more about ShapeBlue and our range of CloudStack related > services > > IaaS Cloud Design & Build< > http://shapeblue.com/iaas-cloud-design-and-build//> > CSForge – rapid IaaS deployment > framework<http://shapeblue.com/csforge/> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > CloudStack Software Engineering< > http://shapeblue.com/cloudstack-software-engineering/> > CloudStack Infrastructure Support< > http://shapeblue.com/cloudstack-infrastructure-support/> > CloudStack Bootcamp Training Courses< > http://shapeblue.com/cloudstack-training/> > > This email and any attachments to it may be confidential and are > intended solely for the use of the individual to whom it is addressed. > Any views or opinions expressed are solely those of the author and do > not necessarily represent those of Shape Blue Ltd or related > companies. If you are not the intended recipient of this email, you > must neither take any action based upon its contents, nor copy or show > it to anyone. Please contact the sender if you believe you have > received this email in error. Shape Blue Ltd is a company incorporated > in England & Wales. ShapeBlue Services India LLP is a company > incorporated in India and is operated under license from Shape Blue > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA > Pty Ltd is a company registered by The Republic of South Africa and is traded > under license from Shape Blue Ltd. ShapeBlue is a registered trademark. > -- Rafael Weingärtner Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
