Mikko, thanks a lot for the suggestion. We indeed use this type of architecture for most of our _really_ slow operations, but in this case - maybe I should have been more specific - the slow operations is in the matter of seconds, because if fetches a file from a distributed file-storage that may be a bit slow. The download takes a few seconds max, so having an asynchronous model sounds like overkill here. Since we receive thousands of "fast" requests a minute, this can hurt...
Thanks again, Clemens On Mon, Oct 19, 2015 at 5:25 PM, Mikko Ohtamaa <[email protected]> wrote: > > > My current thinking is: one way I could simulate this is to divide the >> workers into two uwsgi services - one only serving processing intensive >> tasks, and the other the rest (this is easy, since we use nginx in front of >> uwsgi). The problem is that whether a request is going to be processing >> intensive does not [only] depend on the request URL, but also the state of >> the DB, so it's only a partial solution. >> > > What I would probably do, without knowing the specifics of the task, is > that I'd move CPU intensive tasks totally away from a web server. I'd use > Celery or similar task queue system with a worker poll to run the tasks > > - uWSGI gets a request to start intensive task > > - A task is posted to Celery queue > > - uWSGI returns > > - Celery worker process reads the queue, runs the task, writes result to > database. Task can be even run on a different physical machine if needed. > > - If the client must know the task is finished Celery can use pubsub > (PostgreSQL / Redis) to inform about this > > - Client makes another request to uWSGI and gets results > > This way CPU intensive tasks don't count against with HTTP timeouts or > similar more HTTP'ish settings. > > Celery has more tools to fine-tune intensive tasks, distribute them across > process, monitor them, queue them, etc. It's pretty heavy with initial > switching cost, but makes certainly sense with some workloads. Right tool > for the right job (though without knowing details I would not claim Celery > is the right tool here). > > Cheers, > -Mikko > > > >> >> Thanks for a great server and hopefully some insights ;) >> -Clemens >> >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> >> > > > -- > Mikko Ohtamaa > http://opensourcehacker.com > http://twitter.com/moo9000 > > > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > -- Clemens Kolbitsch Security Researcher [email protected] 805-456-7075 Lastline, Inc. 6950 Hollister Avenue, Suite 101 Goleta, CA 93117 www.lastline.com
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
