we have a similar situation. `pyramid` is the main web application. when a user uploads a file, it is handed off to `celery` via `redis`, which handles all the image resizing and uploading onto S3.
the solution we went with: • short polling on a configurable delay. We default to 2500ms, and half or double it (manually) depending on average server load. • we use a VERY STRIPPED DOWN pyramid view. It's part of the normal app, but inherits from a different base class (we use class based views) and has a very lean integration with pyramid. it basically has 20% the overhead of our other views. this allows for the short polling to even be 100s of ms if we want. • just to be safe and bust aggressive caching, we send a nonce with each poll. (originally we put the nonce in the url, but that had a wee bit of overhead in pyramid) Since the "answer" is going to be in redis, you'll have a blazing fast lookup. • a streamlined view/route gets it done rather perfomant. • if things slow down, you can pull it out and build a dedicated pyramid service (it won't be serving other routes, so the memory footprint will be smaller and you can run way more instances) • if things still don't scale well, then you can think about nodejs. we have some twisted services and would probably choose nodejs for this task over twisted -- it's better suited and you'll be abandoning enough of your codebase to work in Twisted, that it's worth starting such a small project from scratch in nodejs. You could definitely do long polling instead. short polling is just super simple to get up and running. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/pylons-discuss. For more options, visit https://groups.google.com/d/optout.
