Ian Bicking wrote: > On Tue, Mar 3, 2009 at 2:57 PM, kmw <[email protected]> wrote: >> Hi everyone, >> >> I'm trying to find some docs or perhaps old discussions about >> implementing a task queue within a pylons application. The scenario >> I'm trying to support involves a request coming into the app server to >> perform an action which takes a long time to complete, such as >> rebuilding an index or updating a value across hundreds of thousands >> of objects. >> >> My thought was to create a processing thread when the app is loaded. >> When I get the request I add the item to a synchronized queue (which >> the processing thread blocks on) and return a HTTP 201 Accepted to the >> client. The processing thread picks up tasks from the queue and they >> are completed in the order received. The 201 response also has an >> additional Location header to poll the status of the task. >> >> The question that remained was how to create and manage processing >> thread. I've read a couple of threads on this subject, and hunted >> around google a bit and found a couple of options: >> - >> http://groups.google.com/group/pylons-discuss/browse_thread/thread/e30fb912ca79b000/7cc1d4a6b1d9919d?lnk=gst&q=background#7cc1d4a6b1d9919d >> - >> http://groups.google.com/group/pylons-discuss/browse_thread/thread/3e9dfda05af50634/bc914b96e2b96a1b?lnk=gst&q=background#bc914b96e2b96a1b >> >> Now I'm leaning towards creating a process using the python >> multiprocessing module which interfaces like a thread but skips issues >> with the GIL and pylons thread management. However, I didn't find any >> information about how to manage the process lifecycle and allow it to >> shutdown gracefully when the server is stopped. >> >> I'd appreciate feedback on this approach and any pointers to resources >> that will allow me to hook into the app lifecycle and manage my >> subprocess as well. Hopefully I can get a working recipe out of this >> and put it all together in the pylons cook book for future reference. > > > If you are thinking about user-visible long running tasks, maybe give > a look at: http://pythonpaste.org/waitforit/ -- it seems like you are > more thinking about APIs, but at least similar. > > FYI, I think there's actually an HTTP header to indicate when the > client should poll next.
Thanks Ian, that's pretty useful to know about. It doesn't fit my case because I'm working on server-side APIs, but I'll take a look at the implementation to see if I can glean some ideas. - Kochhar --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
