On Thursday 17 of January 2019 17:26:57 Michal Čihař wrote: > Hi > > Slávek Banko píše v Po 14. 01. 2019 v 16:42 +0100: > > I have another interesting note about memory consumption. > > > > As I mentioned, I see the large memory consumption when processing > > large > > repository with many linked components. Yesterday, I made a large > > update > > of POT files where most of them were updated => large repository > > were > > being processed. Surprisingly, the memory consumption remained at an > > acceptable level. > > > > It seems that when large repository is processed and there are no > > changes, > > memory is allocated but not released. While when the same repository > > is > > processed and there are changes, processing is done without huge > > memory > > consumption. > > > > Can this observation help to find the cause? > > This behavior really indicates that the problem is most likely in the > fulltext index. This can be separated into two parts: > > 1. Few days ago I've learned the hard way that Weblate used Whoosh in > quite unoptimal way. The problem is that we were using update_document > API which is not really suitable for batch updates as it loads the > whole index with every update. I've recently addressed this in several > commits, for example: > > https://github.com/WeblateOrg/weblate/commit/91cb3570805e4f8fb35c614a8f >6d27c6f2a978db > > You can try using current master or wait for upcoming 3.5 release to > get these fixes. > > 2. Too high memory usage caused by way Whoosh handles deleted entries > (deleted entries are also old ones when newer version is added). This > is documented in the issue tracker: > > https://github.com/whoosh-community/whoosh/issues/492 > > I've tried to address part of the problem here, but it still waits for > deeper review and merging: > > https://github.com/whoosh-community/whoosh/pull/522 > > You're apparently hit by this quite a lot, so it would be great if you > could give this patch a try and check if that improves the situation > for you. Providing feedback on the pull request is welcome.
Right now, I wanted to write in the pull-request comments that it looks good. I tried to apply a patch from pull/522 to our Weblate instance. And today was a huge update of repositories that did not contain changes in the translation files or templates. Unlike the former state, today no celery consumes a huge amount of memory. The highest consumption is 240 MiB, which is acceptable. As I can see, pull/522 has already been merged, so my comment is no longer needed. Anyway, thank you very much for your help, it now looks good. Cheers -- Slávek
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Weblate mailing list [email protected] https://lists.cihar.com/cgi-bin/mailman/listinfo/weblate
