Hi Slávek Banko píše v Po 14. 01. 2019 v 16:42 +0100: > I have another interesting note about memory consumption. > > As I mentioned, I see the large memory consumption when processing > large > repository with many linked components. Yesterday, I made a large > update > of POT files where most of them were updated => large repository > were > being processed. Surprisingly, the memory consumption remained at an > acceptable level. > > It seems that when large repository is processed and there are no > changes, > memory is allocated but not released. While when the same repository > is > processed and there are changes, processing is done without huge > memory > consumption. > > Can this observation help to find the cause?
This behavior really indicates that the problem is most likely in the fulltext index. This can be separated into two parts: 1. Few days ago I've learned the hard way that Weblate used Whoosh in quite unoptimal way. The problem is that we were using update_document API which is not really suitable for batch updates as it loads the whole index with every update. I've recently addressed this in several commits, for example: https://github.com/WeblateOrg/weblate/commit/91cb3570805e4f8fb35c614a8f6d27c6f2a978db You can try using current master or wait for upcoming 3.5 release to get these fixes. 2. Too high memory usage caused by way Whoosh handles deleted entries (deleted entries are also old ones when newer version is added). This is documented in the issue tracker: https://github.com/whoosh-community/whoosh/issues/492 I've tried to address part of the problem here, but it still waits for deeper review and merging: https://github.com/whoosh-community/whoosh/pull/522 You're apparently hit by this quite a lot, so it would be great if you could give this patch a try and check if that improves the situation for you. Providing feedback on the pull request is welcome. -- Michal Čihař | https://cihar.com/ | https://weblate.org/
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Weblate mailing list [email protected] https://lists.cihar.com/cgi-bin/mailman/listinfo/weblate
