Hi everyone, I'm trying to find some docs or perhaps old discussions about implementing a task queue within a pylons application. The scenario I'm trying to support involves a request coming into the app server to perform an action which takes a long time to complete, such as rebuilding an index or updating a value across hundreds of thousands of objects.
My thought was to create a processing thread when the app is loaded. When I get the request I add the item to a synchronized queue (which the processing thread blocks on) and return a HTTP 201 Accepted to the client. The processing thread picks up tasks from the queue and they are completed in the order received. The 201 response also has an additional Location header to poll the status of the task. The question that remained was how to create and manage processing thread. I've read a couple of threads on this subject, and hunted around google a bit and found a couple of options: - http://groups.google.com/group/pylons-discuss/browse_thread/thread/e30fb912ca79b000/7cc1d4a6b1d9919d?lnk=gst&q=background#7cc1d4a6b1d9919d - http://groups.google.com/group/pylons-discuss/browse_thread/thread/3e9dfda05af50634/bc914b96e2b96a1b?lnk=gst&q=background#bc914b96e2b96a1b Now I'm leaning towards creating a process using the python multiprocessing module which interfaces like a thread but skips issues with the GIL and pylons thread management. However, I didn't find any information about how to manage the process lifecycle and allow it to shutdown gracefully when the server is stopped. I'd appreciate feedback on this approach and any pointers to resources that will allow me to hook into the app lifecycle and manage my subprocess as well. Hopefully I can get a working recipe out of this and put it all together in the pylons cook book for future reference. Thanks. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
