On Thu, Sep 22, 2016 at 1:40 PM, Steven Hardy <[email protected]> wrote: > On Thu, Sep 22, 2016 at 04:36:30PM +0200, Gabriele Cerami wrote: >> Hi, >> >> As reported on this bug >> >> https://bugs.launchpad.net/tripleo/+bug/1626483 >> >> HA gate and periodic jobs for master and sometimes newton started to >> fail for errors related to memory shortage. Memory on undercloud >> instance was increased to 8G less than a month ago, so the problem >> needs a different approach to be solved. >> >> We have some solutions in store. However, with the release date so >> close, I don't think it's time for this kind of changes. So I thought >> it could be a good compromise to temporarily increase the undercloud >> instance memory to 12G, just for this week, unless there's a rapid way >> to reduce memory footprint for heat-engine (usually the biggest memory >> consumer on the undercloud instance) > > If we can avoid it, I'd rather we avoided increasing the ram again - I > suspect there is an issue with a heat regression as I'm seeing much higher > memory usage in my local test environment too. > > I did a quick re-test of some local monitoring I did earlier in the cycle > when we experienced some high memory usage: > > http://people.redhat.com/~shardy/heat/plots/heat_before_after_end_newton.png > > There are three plots there, one early in the cycle, one after some fixes > which reduced memory usage a lot, then the highest leaky plot is the one I > just did today. > > So I'm pretty sure we have another heat memory leak to track down. > > If anyone has any historical data of memory usage e.g from periodic CI > runs, that would be helpful, otherwise we'll have to bisect testing locally > or derive it from scraping our dstat data from CI run logs. > > Steve.
Steve, I dropped a comment in your Heat bug report, that might be related to our CI problem: https://bugs.launchpad.net/heat/+bug/1626675/comments/1 I hope it helps, -- Emilien Macchi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
