I've not done much heavy lifting with the taskqueue, but when i have
long running tasks i have broken them up into chunks that run in 30
seconds or less. i call the task with something like:
for user in db(db.auth_user.id>0).select():
for question in db(db.question.id>0).select():
taskqueue.add(url=URL(r=request,
f='aggregate_user_by_month'),
params={'user': user.id,
'q': question.id})
and then the method processing some things. if it fails because it
tried to process too many, i requeue with a param to process 1/2 as
many objects, otherwise i requeue with the next set to process:
try
...
except DeadlineExceededError:
#reprocess with 1/2 as many responses.
taskqueue.add(url=URL(r=request), params={'user':
request.vars.user,
'q': request.vars.q,
'limit':limit/2})
return
if responses and \
responses[-1].updated_on != last_processed_timestamp:
#we had some responses, so schedule more processing until no
responses
# are found
taskqueue.add(url=URL(r=request), params={'user':
request.vars.user,
'q':
request.vars.q})
this is all using the default queue on GAE. let me know if you have
more specific questions about how i'm using it.
cfh
On Aug 28, 7:34 am, Michael Ellis <[email protected]> wrote:
> At the moment, I'm mostly interested in knowing how to use taskqueue to set
> up a background email notification system. My web2py app is for group
> collaboration and there are situations where all members of a group needed
> to be notified of changes.
>
> The taskqueue API seems straightforward, but it would help to see a working
> example of its use and configuration in web2py. Can it be really simple,
> e.g. enqueueing a controller URL that accesses a table in my app
> containing email addresses and messages to be sent?
>
> On Sat, Aug 28, 2010 at 10:19 AM, Kevin <[email protected]> wrote:
> > Yeah, the gae cron is a lot different than the web2py cron (which is
> > based on traditional crons), because web2py supports any process or
> > internal controller, while GAE only support calling relative URLs in
> > their cron.
>
> > In any case, it'd be nice to have a compatibility layer -- if you
> > don't need anything more than what GAE's method provides (calling some
> > internal controller's function), then perhaps we can make web2py
> > support cron.yaml in non-gae deployments, to where it essentially
> > loads the GAE cron jobs as though they were ** tasks?
>
> > On Aug 28, 7:21 am, mdipierro <[email protected]> wrote:
> > > No but cron does not work on gae either. On gae you can use taskque.
>
> > > On Aug 28, 7:25 am, Michael Ellis <[email protected]> wrote:
>
> > > > Massimo, can the technique described in the book be used on GAE?
> > > > Thx,
> > > > Mike
>
> > > > On Aug 27, 10:23 pm, Bruno Rocha <[email protected]> wrote:
>
> > > > > Thanks Massimo.
> > > > > ,
>
> > > > > 2010/8/27 mdipierro <[email protected]>
>
> > > > > > Run the web server with -N (no cron) and run a separate backrgound
> > > > > > process for cron. Anyway, there is no way to control how much
> > memory
> > > > > > cron consumes if a cron task takes longer than expected. It is
> > safer
> > > > > > not to use cron (-N) and use this instead:
>
> >http://www.web2py.com/book/default/chapter/04#Background-Processes-an...
>
> > > > > > Massimo
>
> > > > > > On Aug 27, 7:18 pm, Bruno Rocha <[email protected]> wrote:
> > > > > > > Hi,
>
> > > > > > > I am running a web2py application at my home server,
> > > > > > > that is published and I have some clients using for accurracy
> > tests.
>
> > > > > > > But , every 3 hours+- the server fails with the message
>
> > > > > > > Exception in thread Thread-20:
>
> > > > > > > Traceback (most recent call last):
>
> > > > > > > File "/usr/lib/python2.6/threading.py", line 532, in
> > __bootstrap_inner
>
> > > > > > > self.run()
>
> > > > > > > File "/home/bruno/web2py/gluon/newcron.py", line 206, in run
>
> > > > > > > shell=self.shell)
>
> > > > > > > File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
>
> > > > > > > errread, errwrite)
>
> > > > > > > File "/usr/lib/python2.6/subprocess.py", line 1049, in
> > _execute_child
>
> > > > > > > self.pid = os.fork()
>
> > > > > > > OSError: [Errno 12] Do not possible to allocate memory
>
> > > > > > > I can see newcron.py is the problem there, but I need to use
> > cron,
> > > > > > > this could be a problem with my OS or machine, or it is a Rocket
> > problem?
>
> > > > > > > --
>
> > > > > > >http://rochacbruno.com.br
>
> > > > > --
>
> > > > >http://rochacbruno.com.br
