On Mon, May 31, 2010 at 10:04 AM, Eugueny Kontsevoy <[email protected]> wrote: > We have an unusual server application that is essentially a task queue: it > takes job requests via HTTP and sticks them into a thread/process pool of > workers. It works well. > > However, I have always felt uncomfortable having all this process/thread > logic in our app. It looks suspiciously similar to a cluster of Pasters it > runs on. This got me thinking: what if we just do everything "in-place"? Is > it possible to do the following: > > A request comes into Pylons controller via HTTP > The controller immediately tells the client "200 OK" - so it won't wait. > ... and proceeds working on that request right there, synchronously - > Paster&Nginx will take care of finding another available process/thread to > handle the next request. > > This would make our code much leaner and cleaner, but I am not sure how to > implement #2. It seems like the only way to end the request is to return > something (or throw an exception) from the controller. > > Thoughs?
You're trying to turn Pylons into something it isn't. Pylons/PasteHTTPServer/Nginx are designed for *web applications*, where the HTTP response is the only goal, and any side effects (like a database update) are done before the response. Separation is always good, in this case the user interface (web server) from the actual task processing. This suggests a real IPC queue. The task engine and web app would be separate processes, and the web app would send messages via inter-process communication to start tasks. You didn't mention querying the status of tasks later, but that could be done with job IDs, which the task engine would generate when creating a task. If you want to have it all in one process, I suppose you could extend PasteHTTPServer and modify the "worker thread" code to accept a WSGI request (calling the Pylons app) and then process it. PasteHTTPServer would see the thread as busy until it finished. You'd have to disable that setting for hung threads, because any thead that took a long time to process would appear hung. But I don't see that you'd gain much by forcing the WSGI request and the task handling into the same thread (or process). The only advantage would be in preserving local variables from the controller action. But that sounds really convoluted. Better to send a message including all necessary arguments to a proper queue. You might also want to look at Twisted. It has a very different programming paradigm, but it's designed more for a "long-running program that has an HTTP service on the side" rather than a web application per se, And it can supposedly run WSGI applications in a thread (note: again a separate thread from the task processing), although you may have to disable things like the SOPs. (I.e., use 'self._py_object.request' in the controller instead of 'pylons.request'). I've heard that's a requirement for asynchronous usage, but I don't know if it's actually true. -- Mike Orr <[email protected]> -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
