ok, I think I got it ..... most of it is cumbersome and fruit of multiple
reiteration (e.g. lots of trials and errors). There's no way in hell to put
up celery docs pertaining the particular usecase in web2py.
Pleeeeeease, watch it carefully, may burn your house to the ground.
@Bruno: maybe the same thing can be applied to rq (I still have problems
with running that one with functions not defined in models outside web2py
environment)
Chosen broker and result backend --> redis (absolutely no time to install
rabbitmq, sorry)
1st issue: models are executed at every request, so a celery instance
defined in models continues to instantiate new connections to Redis.
Although redis is fast as hell, it just mean wasting resources (a single
redis instance makes use by default of a connection pool, but if the object
is recycled and istantiated at every request the pool doesn't survice). So,
celery instance must be defined in a *module* and imported
1st step, hence --> create modules/w2p_celery.py with
from celery import Celery
mycelery = Celery('tasks', broker='redis://localhost:6379/0', backend=
'redis://localhost:6379/0')
2nd issue unresolved: the default task decorator computes a name based on
the current execution instance (for newcomers to web2py, __restricted__)
Celery has a task decorator that takes a name parameter, so it's quite easy
to patch (even so, I'd really like to be able to customize that.
Suggestions ?)
However, if you put this in models/thecelerymodel.py
from w2p_celery import mycelery
celery = mycelery
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL('rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user():
try:
db.auth_user.insert(first_name='John')
db.commit()
except:
db.rollback()
you'll have two tasks registered as 'tasks.add' and 'tasks.add_user'
3rd step: f*******k the world. having something runnable by a celery
worker. This part was the absolute nightmare of the last two evenings. I
really didn't get the problems in it, but seems that task "code" is
somewhat-where-when shipped around (or, my nightmares were in fact the
reality....)
Given that, let's assume that copying the tasks defined before elsewhere
won't get you killed in action (i.e. yes, I'd like too to have celery
workers figuring out where to pick something from, but I just couldn't get
there)
3rd step --> create a tasks.py in the same folder as web2py.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import gluon.widget #forgot: why this is apparently the only way to
fix custom imports ?
from gluon.shell import env
from gluon import current
from celery import Celery
def make_celery(app):
celery = Celery('tasks', broker='redis://localhost:6379/0', backend=
'redis://localhost:6379/0')
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
_env = env(a=app, import_models=True)
globals().update(_env)
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
celery = make_celery('yourappname') #be sure that you write this correctly
@celery.task(name='tasks.gen_url')
def gen_url(x):
return A(x, _href=URL(x, 'rule_the_world'))
@celery.task(name='tasks.add_user')
def add_user(): #yes, for the love of your apps, wrap all db operations!
try:
db.auth_user.insert(first_name='miao')
db.commit()
except:
db.rollback()
now, this **seems** to work without issues, e.g. in a controller you can do
without harm
def myfunction():
async = add_user.delay()
return dict(async=async)
and start an instance of celery, cding into web2py.py folder and doing
>>> celery worker -A tasks
Again, this is not the recommended way nor the "documented somewhere" one.
It is just the first **iteration** that didn't lend in exceptions all over
the place.
If someone wants to pick this up, go to ask (solem) and get it
corrected/revised, I'll be glad to have spent 6 hours for the web2py
community :P
@PS on the robust-widely-adopted argument: celery vs web2py's scheduler is
like comparing web2py to web.py. They have VERY DIFFERENT goals in mind and
celery is by far the most complete task queue solution out there (even
comparing other programming languages task queues). With that in mind (i.e.
even Niphlod in the need of a gigantic solution for out-of-the-band
processes chooses celery), celery broke a few times backward compatibility.
Web2py's scheduler got new features every iteration and just broke code
when it had no API whatsoever to retrieve task results (has just changed
the db schema once, to accomodate all engines reserved keywords)
--
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
