There is a bug: https://bugs.launchpad.net/fuel/+bug/1358345 and patch on review: https://review.openstack.org/#/c/115954/
Vladimir, please participate there. Thanks, On Mon, Aug 25, 2014 at 11:05 PM, Vladimir Kuklin <[email protected]> wrote: > Roman, we figured it out. The problem was that mysql 5.6 was turning > performance_schema parameter on by default. That lead to the shortage of > memory as MySQL allocated 1 gigabyte at once. Now we modified the template > and performance_schema is turned on only if we have 4 gigabytes of RAM on > the node. All the tests are passing now and we closed all the bugs related > to intermittent failures because of insufficient resources on the node. > > > On Mon, Aug 25, 2014 at 10:58 PM, Roman Alekseenkov < > [email protected]> wrote: > >> Team, >> >> Shall we settle on a conclusion? If we need more memory for a working HA, >> let's accept this fact and move on. >> >> I believe we should define a set of supported scenarios for the >> VirtualBox, update our scripts, update release notes and also change quick >> start on Mirantis portal. I filed >> https://bugs.launchpad.net/fuel/+bug/1361331 to track that. >> >> My suggestion is to introduce something along the lines of: >> >> 1. config_4GB.sh - non-HA, 1 controller, 1 compute >> 2. config_8GB.sh - non-HA, 1 controller, 3 compute nodes >> 3. config_16GB.sh - HA, 3 controllers, 1 compute >> >> If we want to go beyond that, there should be an answer in form of >> deploying OVA/OVF into vCenter and utilizing more compute power from a set >> of ESX hosts. >> >> Thanks, >> Roman >> >> >> On Fri, Aug 22, 2014 at 1:09 AM, Meg McRoberts <[email protected]> >> wrote: >> >>> We should check with our training people to see what they need. For >>> example, any >>> reasonable evaluation of MongoDB (used for Ceilometer) or Zabbix >>> probably requires >>> a lot more resources than people are going to have on their personal >>> machine, but it >>> would be nice if they could configure some sort of minimally-functional >>> version of these >>> on a smaller system just for training purposes. >>> >>> >>> On Fri, Aug 22, 2014 at 12:12 AM, Sergii Golovatiuk < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> I would say we have to update release notes. Even now it's barely >>>> possible to unleash Fuel power on laptop with 8GB of RAM. With 16Gb you may >>>> have up to 5 VMs simulating HA environment. If we limit VM RAM to 2.5 user >>>> may have 5x2.5+1(master node)=13.5 GB which should be enough for demo >>>> purposes. >>>> >>>> >>>> -- >>>> Best regards, >>>> Sergii Golovatiuk, >>>> Skype #golserge >>>> IRC #holser >>>> >>>> >>>> On Fri, Aug 22, 2014 at 3:06 AM, Christopher Aedo <[email protected]> >>>> wrote: >>>> >>>>> In my opinion it's reasonable to limit expectations on what you can >>>>> evaluate on virtualbox, and provide appropriate caveats. The truth is >>>>> that making a virtual machine do everything it needs to do to stay >>>>> synced in real-time with two or more other virtual machines requires >>>>> resources. Evaluating Fuel as a deployment tool under virtualbox >>>>> makes sense, but expecting to get a legitimate evaluation of working >>>>> (and performant) HA under a severely restricted environment doesn't >>>>> seem reasonable. >>>>> >>>>> Would a legitimate customer expect to be able to do a thorough >>>>> evaluation of MOS without using anything more than a single machine >>>>> with a moderate amount of ram? >>>>> >>>>> -Christopher >>>>> >>>>> On Thu, Aug 21, 2014 at 4:59 PM, David Easter <[email protected]> >>>>> wrote: >>>>> > It has already became nearly impossible to run the default virtualbox >>>>> > environment (with a cluster size of 3 nodes) on an 8GB system. It >>>>> would >>>>> > become challenging if we begin to tax a 16GB system to do this now… >>>>> > >>>>> > Are there alternatives or specific tunings that can be done so that >>>>> the >>>>> > sizing for a virtualbox trial is different than what is needed for >>>>> > production or physical server testing? >>>>> > >>>>> > Thanks, >>>>> > >>>>> > - David J. Easter >>>>> > Director of Product Management, Mirantis, Inc. >>>>> > >>>>> > From: "[email protected]" <[email protected]> >>>>> > Date: Thursday, August 21, 2014 at 4:29 PM >>>>> > To: Sergii Golovatiuk <[email protected]>, David Easter >>>>> > <[email protected]> >>>>> > Cc: Bogdan Dobrelya <[email protected]>, " >>>>> [email protected]" >>>>> > <[email protected]> >>>>> > Subject: Re: [Fuel-dev] RAM issues with environments >>>>> > >>>>> > David, >>>>> > >>>>> > FYI - it will impact VirtualBox trials. 3 controllers x 3GB = 9 GB >>>>> RAM only >>>>> > for controllers if you run HA. >>>>> > >>>>> > Roman >>>>> > >>>>> > >>>>> > On Thu, Aug 21, 2014 at 5:22 AM, Sergii Golovatiuk >>>>> > <[email protected]> wrote: >>>>> >> >>>>> >> +1 for not using debug at CI gates (only BVT) >>>>> >> +1 for adding atop to our builds as it helps to understand what was >>>>> wrong >>>>> >> at that particular time >>>>> >> +1 for increasing RAM (though we'll try to tune rabbit and Galera). >>>>> 3GB >>>>> >> should be enough so we'll be able to run up to 2 environments on >>>>> 32GB RAM >>>>> >> servers. >>>>> >> >>>>> >> -- >>>>> >> Best regards, >>>>> >> Sergii Golovatiuk, >>>>> >> Skype #golserge >>>>> >> IRC #holser >>>>> >> >>>>> >> >>>>> >> On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya < >>>>> [email protected]> >>>>> >> wrote: >>>>> >>> >>>>> >>> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote: >>>>> >>> > Hi, >>>>> >>> > >>>>> >>> > Digging the issue with Galera, I found that our environments >>>>> have very >>>>> >>> > high RAM utilization which leads to the problem during >>>>> environment >>>>> >>> > deployment. For instance "HA deployment + neutron/GRE" requires >>>>> almost >>>>> >>> > 2.6-2.7 GB during deployment >>>>> >>> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). >>>>> I found >>>>> >>> > high swap in/swap out usage during deployment with very high load >>>>> >>> > average. This creates many sporadic issues with some services. >>>>> They >>>>> >>> > time >>>>> >>> > out in random place making our debugging very hard. I would like >>>>> to >>>>> >>> > review our policy for CI environment and increase RAM (at least >>>>> for bvt >>>>> >>> > tests) to 3 GB. >>>>> >>> > >>>>> >>> > -- >>>>> >>> > Best regards, >>>>> >>> > Sergii Golovatiuk, >>>>> >>> > Skype #golserge >>>>> >>> > IRC #holser >>>>> >>> > >>>>> >>> > >>>>> >>> >>>>> >>> I believe we should do at least the following for our CI jobs and >>>>> bvt >>>>> >>> tests: >>>>> >>> 1) Deployment shortcuts: stop deployment abruptly, if any >>>>> deployment >>>>> >>> blocker has been met, such as something exceeded given # of >>>>> retries. >>>>> >>> That could be done in puppet by overriding 'tries' behavior in exec >>>>> >>> provider, or at orchestration layer as well. >>>>> >>> 2) Load management: collect and automatically analyze atop stats >>>>> (swap >>>>> >>> rates, load average, io waiters) from jenkins slaves and vm nodes >>>>> while >>>>> >>> running the jobs, and stop or freeze some jobs, if some >>>>> >>> performance-stopper criteria has been met as well. >>>>> >>> 3) Do not use debug level logging for CI gates, use it only for bvt >>>>> >>> tests. >>>>> >>> >>>>> >>> -- >>>>> >>> Best regards, >>>>> >>> Bogdan Dobrelya, >>>>> >>> Skype #bogdando_at_yahoo.com >>>>> >>> Irc #bogdando >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Mailing list: https://launchpad.net/~fuel-dev >>>>> >> Post to : [email protected] >>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev >>>>> >> More help : https://help.launchpad.net/ListHelp >>>>> >> >>>>> > >>>>> > >>>>> > -- >>>>> > Mailing list: https://launchpad.net/~fuel-dev >>>>> > Post to : [email protected] >>>>> > Unsubscribe : https://launchpad.net/~fuel-dev >>>>> > More help : https://help.launchpad.net/ListHelp >>>>> > >>>>> >>>> >>>> >>>> -- >>>> Mailing list: https://launchpad.net/~fuel-dev >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~fuel-dev >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>> >>> -- >>> Mailing list: https://launchpad.net/~fuel-dev >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~fuel-dev >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >> -- >> Mailing list: https://launchpad.net/~fuel-dev >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~fuel-dev >> More help : https://help.launchpad.net/ListHelp >> >> > > > -- > Yours Faithfully, > Vladimir Kuklin, > Fuel Library Tech Lead, > Mirantis, Inc. > +7 (495) 640-49-04 > +7 (926) 702-39-68 > Skype kuklinvv > 45bk3, Vorontsovskaya Str. > Moscow, Russia, > www.mirantis.com <http://www.mirantis.ru/> > www.mirantis.ru > [email protected] > > -- > Mailing list: https://launchpad.net/~fuel-dev > Post to : [email protected] > Unsubscribe : https://launchpad.net/~fuel-dev > More help : https://help.launchpad.net/ListHelp > > -- Mike Scherbakov #mihgen
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
