Hi,
We are using kallithea with a postgres database on another server
connected via ssl.
It seems that celery in version 4 uses prefork in its worker pool model
now, which leads to issues with such a database connection:
```
Task kallithea.lib.celerylib.whoosh_index[9c
4dbbcc-b9e9-48be-9455-0bbcd8d251a2] raised unexpected:
OperationalError('(psycopg2.Operational
Error) SSL SYSCALL error: EOF detected\n')
Traceback (most recent call last):
File
"/mnt/webservices/virtualenv/kallithea-0.6.2/lib/python3.7/site-packages/sqlalchemy/eng
ine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File
"/mnt/webservices/virtualenv/kallithea-0.6.2/lib/python3.7/site-packages/sqlalchemy/eng
ine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected
The above exception was the direct cause of the following exception:
(...)
```
We know that issues like this may arise from using prefork in uwsgi
which we work around by setting lazy:true
The reason for this is that when a database connection with SSL (as we
use) is already established with a valid SSL context before a process is
forked, the connection will cease to work as such a connection is
stateful on both sides and cannot cope well with a fork.
The only workaround to this we found is to setup celery to only use a
single worker in production.ini:
```
celery.worker_concurrency = 1
celery.worker_max_tasks_per_child = 5
```
But this has the drawback that long running tasks like indexing will
block short running tasks like the creation of a new repository. Thus,
this setting is undesired and worker_concurrency should be at least 2 on
a production system.
I tried to find an easy fix for this but it seems to need some changes
to kallithea as long as celery and sqalchemy don't change to cope with
this problem.
I hope someone can help with this.
Cheers,
Valentin
_______________________________________________
kallithea-general mailing list
[email protected]
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
