Awesome detective work, Joshua, thank you! -jay
On 02/03/2016 03:56 AM, Joshua Hesketh wrote:
Hey all, So this took me a lot longer than I would have liked (the fact that I pushed this up in the middle of a conference should be evidence), but I believe I have managed to track this down: https://review.openstack.org/#/c/275483/ We also have only ever been clearing our internal cache when reloading. We should perhaps look at doing that periodically but that is a different issue to the newly introduced leak. Although I haven't checked, I suspect our cache is a lot larger recently (looking at cacti and other metrics) probably due to cross-repo-dependencies so there is more need to keep it clear. This may have contributed to the recent larger memory use. Cheers, Josh On Sat, Jan 9, 2016 at 7:20 AM, James E. Blair <[email protected] <mailto:[email protected]>> wrote: Hi, Some time before December 17th we merged a change to Zuul that causes a memory leak. Prior to then, the amount of used memory on OpenStack's Zuul server consumed a very steady 5GB. Now it continually increases. We can see the behavior start very clearly with our December 17 restart of Zuul. But it had been running for quite some time (perhaps a month or more?) before then, so that is not much use in narrowing down the change, and several changes were merged in that time period that could have an effect. See this graph: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=4&view_type=&graph_start=1449499887&graph_end=1452197778&graph_height=120&graph_width=500&title_font_size=10 If anyone else running a Zuul from git master has observed increased memory usage, would you please let us know any dates where you restarted Zuul (with the latest commit) and either did or did not observe increased memory usage? That may help us quickly narrow down what we need to look at. Thanks, Jim _______________________________________________ OpenStack-Infra mailing list [email protected] <mailto:[email protected]> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
