On Saturday 08 of December 2018 13:37:13 Michal Čihař wrote: > Hi > > On Thu, 2018-12-06 at 19:40 +0100, Slávek Banko wrote: > > a few days ago I updated Weblate on our server to version 3.3.0 and > > the > > behavior was "interesting". > > > > During a time when the celery started, everything was all right. The > > memory consumption was at a good level - main process of about 100 > > MiB, a > > worker of about 150 MiB. The suspect was only a constant CPU usage. > > > > I courageously increased the number of workers from 1 to 2. At begin > > everything looked good. But after a while, workers began to increase > > their memory consumption - sometimes significantly, however after > > some > > time gradually releasing them. Each worker then consumed about 350 - > > 450 > > MiB. Later, however, the main process, which typically consumed 100 > > MiB, > > began to significantly increase memory consumption - reaching 900 > > MiB. > > Each worker still showed a continuous CPU usage. > > > > I preventively stopped the celery and started again. To my surprise, > > since > > then behaviour of celery has been totally in order. The control > > process > > has a consumption of about 100 MiB, each worker of about 150 MiB. And > > the > > CPU consumption is low. > > > > I do not know what it was like before, but now for two days, the > > behavior > > is absolutely seamless. Thank you for your support! > > The way you describe it, it seems to be related to Celery task > prefetching [1]. The problem is that for fulltext updates, we need to > fetch all (or lot of) tasks. This has caused some problems and might > lead to situation you observe - the memory consumption is growing in > case more workers are fighting for locks and the queue is not emptying > (you can see that in /api/metrics/). > > Recently I've adjusted the recommended setup to have dedicated worker > to do the fulltext index updates, what should improve performance. > > As this is just matter of configuration, you can apply that to the 3.3 > release as well: > > Settings part: > > https://github.com/WeblateOrg/weblate/blob/cc65e9f305221d723fe52c75ce83 >a2545c685dac/weblate/settings_example.py#L827-L831 > > Updated Celery startup configuration: > > https://github.com/WeblateOrg/weblate/blob/cc65e9f305221d723fe52c75ce83 >a2545c685dac/examples/celery-weblate.conf#L13 > > > [1]: > http://docs.celeryproject.org/en/latest/userguide/configuration.html#st >d:setting-worker_prefetch_multiplier
Hi, in the past month, I watched celery behave. There is still a problem. I notice that when is pushed commit to a repository that has in Weblate a large number of linked components, the celery worker allocate a lot of memory during processing and does not release it. Never. Then, if I do not restart the celery daemon in sufficient time, this can result in OOM killer initialization. Cheers -- Slávek
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Weblate mailing list [email protected] https://lists.cihar.com/cgi-bin/mailman/listinfo/weblate
