On Fri, Aug 19, 2016 at 11:41 AM, Derek Higgins <der...@redhat.com> wrote: > On 19 August 2016 at 00:07, Sagi Shnaidman <sshna...@redhat.com> wrote: >> Hi, >> >> we have a problem again with not enough memory in HA jobs, all of them >> constantly fails in CI: http://status-tripleoci.rhcloud.com/ > > Have we any idea why we need more memory all of a sudden? For months > the overcloud nodes have had 5G of RAM, then last week[1] we bumped it > too 5.5G now we need it bumped too 6G. > > If a new service has been added that is needed on the overcloud then > bumping to 6G is expected and probably the correct answer but I'd like > to see us avoiding blindly increasing the resources each time we see > out of memory errors without investigating if there was a regression > causing something to start hogging memory. > > Sorry if it seems like I'm being picky about this (I seem to resist > these bumps every time they come up) but there are two good reasons to > avoid this if possible > o at peak we are currently configured to run 75 simultaneous jobs > (although we probably don't reach that at the moment), and each HA job > has 5 baremetal nodes so bumping from 5G too 6G increases the amount > of RAM ci can use at peak by 375G > o When we bump the RAM usage of baremetal nodes from 5G too 6G what > we're actually doing is increasing the minimum requirements for > developers from 28G(or whatever the number is now) too 32G > > So before we bump the number can we just check first if its justified, > as I've watched this number increase from 2G since we started running > tripleo-ci > > thanks, > Derek. > > [1] - https://review.openstack.org/#/c/353655/
Wondering if it makes sense to enable any but the most basic overcloud services in TripleO CI. The idea of using some type of on-demand job for services other than the ones needed for the ping test has been proposed elsewhere -- maybe this should be our default mode for TripleO CI. Thoughts? --Hugh >> I've created a patch that will increase it[1], but we need to increase it >> right now on rh1. >> I can't do it now, because unfortunately I'll not be able to watch this if >> it works and no problems appear. >> TripleO CI cloud admins, please increase the memory for baremetal flavor on >> rh1 tomorrow (to 6144?). >> >> Thanks >> >> [1] https://review.openstack.org/#/c/357532/ >> -- >> Best regards >> Sagi Shnaidman > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- , , | Hugh Brock, hbr...@redhat.com )-_"""_-( | Director of Engineering, OpenStack Management ./ o\ /o \. | TripleO: Install, configure, and scale OpenStack. . \__/ \__/ . | http://rdoproject.org, http://tripleo.org ... V ... | ... - - - ... | "I know that you believe you understand what you . - - . | think I said, but I'm not sure you realize that what `-.....-ยด | you heard is not what I meant." --Robert McCloskey "TripleOwl" __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev