On Monday 28 of January 2019 18:47:46 Slávek Banko wrote: > On Thursday 17 of January 2019 17:26:57 Michal Čihař wrote: > > Hi > > > > Slávek Banko píše v Po 14. 01. 2019 v 16:42 +0100: > > > I have another interesting note about memory consumption. > > > > > > As I mentioned, I see the large memory consumption when processing > > > large > > > repository with many linked components. Yesterday, I made a large > > > update > > > of POT files where most of them were updated => large repository > > > were > > > being processed. Surprisingly, the memory consumption remained at > > > an acceptable level. > > > > > > It seems that when large repository is processed and there are no > > > changes, > > > memory is allocated but not released. While when the same > > > repository is > > > processed and there are changes, processing is done without huge > > > memory > > > consumption. > > > > > > Can this observation help to find the cause? > > > > This behavior really indicates that the problem is most likely in the > > fulltext index. This can be separated into two parts: > > > > 1. Few days ago I've learned the hard way that Weblate used Whoosh in > > quite unoptimal way. The problem is that we were using > > update_document API which is not really suitable for batch updates as > > it loads the whole index with every update. I've recently addressed > > this in several commits, for example: > > > > https://github.com/WeblateOrg/weblate/commit/91cb3570805e4f8fb35c614a > >8f 6d27c6f2a978db > > > > You can try using current master or wait for upcoming 3.5 release to > > get these fixes. > > > > 2. Too high memory usage caused by way Whoosh handles deleted entries > > (deleted entries are also old ones when newer version is added). This > > is documented in the issue tracker: > > > > https://github.com/whoosh-community/whoosh/issues/492 > > > > I've tried to address part of the problem here, but it still waits > > for deeper review and merging: > > > > https://github.com/whoosh-community/whoosh/pull/522 > > > > You're apparently hit by this quite a lot, so it would be great if > > you could give this patch a try and check if that improves the > > situation for you. Providing feedback on the pull request is welcome. > > Right now, I wanted to write in the pull-request comments that it looks > good. I tried to apply a patch from pull/522 to our Weblate instance. > And today was a huge update of repositories that did not contain > changes in the translation files or templates. Unlike the former state, > today no celery consumes a huge amount of memory. The highest > consumption is 240 MiB, which is acceptable. > > As I can see, pull/522 has already been merged, so my comment is no > longer needed. Anyway, thank you very much for your help, it now looks > good. > > Cheers
Hi, bad news. Yesterday, I first found that searching in Weblate will cause an unprecedented temporary increase memory consumption in wsgi process - up to 1.4 GiB => mostly on my VPS causing OOM Killer activation. That's why I tried to upgrade Weblate from 3.3 to 3.4. After updating the celery daemon settings, I now see the constant activity of worker for -queues=search == 100% of one core for several hours, and the memory consumption of this one worker is now 1.7 GiB and is still growing. I expect to activate OOM Killer soon. Please, some idea? Cheers -- Slávek
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Weblate mailing list [email protected] https://lists.cihar.com/cgi-bin/mailman/listinfo/weblate
