Or you can just run the long running task on a separate thread.
def my_long_running_function():
...
t = threading.Thread(target=my_long_running_function)
t.setDaemon(True)
t.start()
Then make a view that has access to the result calculated on this
thread, and shows the result when it's done running. The drawback with
this is that you will occupy one Django processes working on this, so
it's not particularly suited for lots of concurrent users.
Reserve queues for when you have more concurrency, or more tasks with
this behaviour. There's a whole new layer of complexity you have to
manage once you implement queues.
On Aug 15, 10:30 pm, Javier Guerra Giraldez <[email protected]>
wrote:
> On Sat, Aug 14, 2010 at 1:28 PM, ydjango <[email protected]> wrote:
> > I have a online user initiated synchronous process which runs anywhere
> > between 1-5 minutes and gives user status message at the end. It is a
> > very DB intensive process that reads and updates lots of mysql rows
> > and does many calculations. The process is run as part of a view
> > method.
>
> long-running tasks don't belong to the request-responce cycle, send
> them to a background process.
>
> one of the easiest solutions is to use "Ghetto Queues". it can look like this:
>
> - create a new 'task' table, with fields to specify how to process a
> task and a status field (preparing, ready, processing, done, failed),
> if needed a 'result' field and if possible a 'percent done' field too.
>
> - when the user initiates a task, don't process it in the view
> function. just add a new record to the task table, with all the info
> needed to process it and set in 'ready' state. then return. don't
> forget to note the task record ID somewhere you can relate to the
> user's action.
>
> - another process, separate from the Django server (can be a cron task
> in many cases) checks the task table and picks any record in 'ready'
> state to do all the needed processing, first changing it to
> 'processing' state. it should update the 'percent done' field, and
> set to 'done' or 'failed' when finished.
>
> - make some way for the user to get the status of the task, either
> revisiting the same URL (which should show the status if there's a
> task in 'processing'), or via some AJAX query.
>
> there are many variations of this strategy: you can have several
> 'workers' if you pay attention to the way it picks a task and sets to
> 'processing' (transactions make it easy to atomically pick one), if
> you need low latency between queuing a task and picking it by the
> worker process, you can replace the task table with a queue manager
> (check carrot and/or celery)
>
> --
> Javier
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
