On Thu, Feb 4, 2016 at 2:44 AM, James E. Blair <[email protected]> wrote:
> Joshua Hesketh <[email protected]> writes: > > > Hey all, > > > > So this took me a lot longer than I would have liked (the fact that I > > pushed this up in the middle of a conference should be evidence), but I > > believe I have managed to track this down: > > https://review.openstack.org/#/c/275483/ > > > > We also have only ever been clearing our internal cache when reloading. > We > > should perhaps look at doing that periodically but that is a different > > issue to the newly introduced leak. Although I haven't checked, I suspect > > our cache is a lot larger recently (looking at cacti and other metrics) > > probably due to cross-repo-dependencies so there is more need to keep it > > clear. This may have contributed to the recent larger memory use. > > Ah, that explanation makes sense, thanks! > > On the subject of clearing the cache more often, I think we may not want > to wipe out the cache more often than we do now -- in fact, I think we > may want to look into ways to keep from doing even that, because > whenever we reload now, Zuul slows down considerably as it has to query > Gerrit again for all of the data previously in its cache. > I can see a lot of 3rd parties or simpler CI's not needing to reload zuul very often so this cache would never get cleared. Perhaps cached objects should have an expiry time (of a day or so) and can be cleaned up periodically? Additionally if clearing the cache on a reload is causing pain maybe we should move the cache into the scheduler and keep it between reloads? Cheers, Josh > > But we certainly don't want multiple copies of the entire cache. Thanks > again! > > -Jim >
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
