Hey all, So this took me a lot longer than I would have liked (the fact that I pushed this up in the middle of a conference should be evidence), but I believe I have managed to track this down: https://review.openstack.org/#/c/275483/
We also have only ever been clearing our internal cache when reloading. We should perhaps look at doing that periodically but that is a different issue to the newly introduced leak. Although I haven't checked, I suspect our cache is a lot larger recently (looking at cacti and other metrics) probably due to cross-repo-dependencies so there is more need to keep it clear. This may have contributed to the recent larger memory use. Cheers, Josh On Sat, Jan 9, 2016 at 7:20 AM, James E. Blair <[email protected]> wrote: > Hi, > > Some time before December 17th we merged a change to Zuul that causes a > memory leak. Prior to then, the amount of used memory on OpenStack's > Zuul server consumed a very steady 5GB. Now it continually increases. > We can see the behavior start very clearly with our December 17 restart > of Zuul. But it had been running for quite some time (perhaps a month > or more?) before then, so that is not much use in narrowing down the > change, and several changes were merged in that time period that could > have an effect. > > See this graph: > > > http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=4&view_type=&graph_start=1449499887&graph_end=1452197778&graph_height=120&graph_width=500&title_font_size=10 > > If anyone else running a Zuul from git master has observed increased > memory usage, would you please let us know any dates where you restarted > Zuul (with the latest commit) and either did or did not observe > increased memory usage? That may help us quickly narrow down what we > need to look at. > > Thanks, > > Jim > > _______________________________________________ > OpenStack-Infra mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
