On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote: > Hi, > > Digging the issue with Galera, I found that our environments have very > high RAM utilization which leads to the problem during environment > deployment. For instance "HA deployment + neutron/GRE" requires almost > 2.6-2.7 GB during deployment > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I found > high swap in/swap out usage during deployment with very high load > average. This creates many sporadic issues with some services. They time > out in random place making our debugging very hard. I would like to > review our policy for CI environment and increase RAM (at least for bvt > tests) to 3 GB. > > -- > Best regards, > Sergii Golovatiuk, > Skype #golserge > IRC #holser > >
I believe we should do at least the following for our CI jobs and bvt tests: 1) Deployment shortcuts: stop deployment abruptly, if any deployment blocker has been met, such as something exceeded given # of retries. That could be done in puppet by overriding 'tries' behavior in exec provider, or at orchestration layer as well. 2) Load management: collect and automatically analyze atop stats (swap rates, load average, io waiters) from jenkins slaves and vm nodes while running the jobs, and stop or freeze some jobs, if some performance-stopper criteria has been met as well. 3) Do not use debug level logging for CI gates, use it only for bvt tests. -- Best regards, Bogdan Dobrelya, Skype #bogdando_at_yahoo.com Irc #bogdando -- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
