Hi On Thu, 2018-12-06 at 19:40 +0100, Slávek Banko wrote: > a few days ago I updated Weblate on our server to version 3.3.0 and > the > behavior was "interesting". > > During a time when the celery started, everything was all right. The > memory consumption was at a good level - main process of about 100 > MiB, a > worker of about 150 MiB. The suspect was only a constant CPU usage. > > I courageously increased the number of workers from 1 to 2. At begin > everything looked good. But after a while, workers began to increase > their memory consumption - sometimes significantly, however after > some > time gradually releasing them. Each worker then consumed about 350 - > 450 > MiB. Later, however, the main process, which typically consumed 100 > MiB, > began to significantly increase memory consumption - reaching 900 > MiB. > Each worker still showed a continuous CPU usage. > > I preventively stopped the celery and started again. To my surprise, > since > then behaviour of celery has been totally in order. The control > process > has a consumption of about 100 MiB, each worker of about 150 MiB. And > the > CPU consumption is low. > > I do not know what it was like before, but now for two days, the > behavior > is absolutely seamless. Thank you for your support!
The way you describe it, it seems to be related to Celery task prefetching [1]. The problem is that for fulltext updates, we need to fetch all (or lot of) tasks. This has caused some problems and might lead to situation you observe - the memory consumption is growing in case more workers are fighting for locks and the queue is not emptying (you can see that in /api/metrics/). Recently I've adjusted the recommended setup to have dedicated worker to do the fulltext index updates, what should improve performance. As this is just matter of configuration, you can apply that to the 3.3 release as well: Settings part: https://github.com/WeblateOrg/weblate/blob/cc65e9f305221d723fe52c75ce83a2545c685dac/weblate/settings_example.py#L827-L831 Updated Celery startup configuration: https://github.com/WeblateOrg/weblate/blob/cc65e9f305221d723fe52c75ce83a2545c685dac/examples/celery-weblate.conf#L13 [1]: http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-worker_prefetch_multiplier -- Michal Čihař | https://cihar.com/ | https://weblate.org/
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Weblate mailing list [email protected] https://lists.cihar.com/cgi-bin/mailman/listinfo/weblate
