El sáb., 29 dic. 2018 a las 18:35, Mads Kiilerich (<[email protected]>) escribió: > > On 12/29/18 3:51 PM, Thomas De Schampheleire wrote: > >> (Also, "celery" is just an implementation detail. And a quite complex > >> and fragile one - especially if using rabbitmq. But at our scale, it > >> would probably be better to just use a sql database. And most > >> immportant: Our commands and documentation should perhaps just talk > >> about "worker process".) > > Can you elaborate on the following, please? > > > > - "especially if using rabbitmq" -- in which way is rabbitmq a bigger > > problem than other brokers? > > > It is my experience that rabbitmq can be tricky to configure. It is a > separate and very different server process and different environment > that can be hard to manage. No doubt it is great for what it is, but it > might also be overkill.
Ok. In our case we have started it with the default settings and it seems to do enough for Kallithea usage. > > > > - what do you mean with 'use a sql database'? To replace what, > > rabbitmq? And how exactly? If you meant to replace celery, then I'm > > totally lost. > > > > Celery is a 'distributed task queue', while in Kallithea we probably > > just need something to asynchronously handle tasks. Unless Kallithea > > would be deployed at Github scale, maybe? > > Do you suggest a different approach? > > > Since all tasks operate on the database anyway, there is no point in > making these async tasks scalable beyond what the database can handle. > Celery *can* apparently use a plain database as backend ... with the > caveat that it needs inefficient polling. That could perhaps be an > alternative to rabbitmq. Are you referring to this: http://docs.celeryproject.org/en/3.1/getting-started/brokers/sqlalchemy.html This page mentions: "Experimental Status The SQLAlchemy transport is unstable in many areas and there are several issues open. Unfortunately we don’t have the resources or funds required to improve the situation, so we’re looking for contributors and partners willing to help." In Celery 4 it is no longer listed: http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html > > (But also, in our usecase we try to abstract and embed celery. I'm not > sure how much value the celery features add. It would perhaps be easy > and simple to just implement the few things we need: a database table > with pending commands, and a long-running cli command that pop commands > and execute them. But now we have a working celery intergration, so as > long as it is low maintenance, it is probably better to keep it. > Depending on how tricky it is to upgrade.) Ok, we'll see later. > > > > > >> Yes. If we assess that an area has good test coverage, then a safe > >> upgrade is easy. > > The problem is to make a statement about test coverage about external > > components. What is our coverage of 'Paste' ? > > > The Paste coverage is certainly better than our coverage of celery ;-) > > Perhaps take a look at where we use paste. Try to hack the integration > points and introduce some errors in the source and see if they are hit > when running the test suite. Points without coverage need some manual > testing. And it is also nice to do a bit of manual "does it really work" > testing for areas with some test coverage. Especially if it is close to > the front-end where there are different "right" ways to do it ... or > where something might be "wrong" in unknown ways. I looked what exactly we are using from Paste, at least directly (not indirectly through other packages). And then I identified some uses that could be removed, I'll send patches shortly. What is left is: kallithea/bin/kallithea_cli_iis.py: from paste.deploy import loadapp kallithea/bin/kallithea_cli_base.py:import paste.deploy kallithea/bin/kallithea_cli_base.py: kallithea.CONFIG = paste.deploy.appconfig('config:' + path_to_ini_file) kallithea/lib/base.py:import paste.httpexceptions kallithea/lib/base.py:import paste.auth.basic kallithea/lib/base.py:import paste.httpheaders kallithea/lib/base.py:class BasicAuth(paste.auth.basic.AuthBasicAuthenticator): kallithea/lib/base.py: head = paste.httpheaders.WWW_AUTHENTICATE.tuples('Basic realm="%s"' % self.realm) kallithea/lib/base.py: return paste.httpexceptions.HTTPForbidden(headers=head) kallithea/lib/base.py: return paste.httpexceptions.HTTPUnauthorized(headers=head) kallithea/lib/base.py: authorization = paste.httpheaders.AUTHORIZATION(environ) kallithea/lib/base.py: paste.httpheaders.AUTH_TYPE.update(environ, 'basic') kallithea/lib/base.py: paste.httpheaders.REMOTE_USER.update(environ, result) kallithea/lib/hooks.py: from paste.deploy import appconfig kallithea/tests/conftest.py:from paste.deploy import loadwsgi kallithea/lib/middleware/sessionmiddleware.py: if environ.get('paste.registry'): kallithea/lib/middleware/sessionmiddleware.py: if environ['paste.registry'].reglist: kallithea/lib/middleware/sessionmiddleware.py: environ['paste.registry'].register(self.session, session) kallithea/lib/middleware/sessionmiddleware.py: if 'paste.testing_variables' in environ and 'webtest_varname' in options: kallithea/lib/middleware/sessionmiddleware.py: environ['paste.testing_variables'][options['webtest_varname']] = session kallithea/tests/scripts/manual_test_concurrency.py:from paste.deploy import appconfig So mostly paste.deploy and some paste.httpheaders, paste.httpexceptions, paste.auth. Via pytest-cov on Kallithea itself, I could see that the stuff in kallithea/lib/base.py is mostly covered by the tests, not all but looks good to me. /Thomas _______________________________________________ kallithea-general mailing list [email protected] https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
