On Mon, May 31, 2010 at 2:09 PM, Eugueny Kontsevoy <[email protected]> wrote: > Thanks for all your replies guys, yes you're right - there are better tools > for this job (we're looking at celery) but that only increases the number of > moving parts for something fundamentally very simple. > > Using a message broker like RabbitMQ makes our chain look like: > > nginx queue -> paster queue -> celery queue -> worker process > > Why so many queues and so many processes (and config files) to manage? All > we really need is just to receive a request, return "200 OK" and start > working on it. There is no "GUI", no browser, just a series of HTTP POSTs > that don't expect anything in return.
There are ways to make it much simpler, without using celery or Pylons. If you're only queuing a job and returning a quick dummy response, you can use a Queue and BaseHTTPServer, both in the Python standard library. Have one thread accept requests, put the job in the queue, and return 204 No Content on success, or 4xx or 5xx on error. BaseHTTPServer is synchronous, but that won't matter unless you're getting several requests a second. Going up a level in complexity, you can use the wsgiref server, and write a basic WSGI application, optionally using WebOb and Routes. Or you can even plug in a Pylons application, with all the optional middlewares turned off, and skipping the Paste/INI stuff. (Specifically, you'd call make_app() in middleware.py, and pass that to wsgiref.simple_server.make_server().). You could also use Routes manually, by defining a mapper and calling mapper.match() with the WSGI environ. Or instead of a queue, the request handler could spawn a thread to handle the task, while the parent thread returns the dummy response. You may want to keep a thread count in this case, to prevent too many worker threads from running simultaneously. You could keep an integer count, or a set of thread IDs. It may be possible to use weakrefs to put the thread itself in a set and have it automatically disappear when the thread exits; I'm not sure about that. Otherwise you'd have to consider: runaway threads (too many requests), deadlocks (two threads waiting for each other to do something), hung threads (a thread gets stuck for some reason), and zombie threads (thread won't exit for some reason). You'd also have to consider whether it's OK for jobs to disappear when the server exists, and for threads to be killed in progress. -- Mike Orr <[email protected]> -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
