Hi Carl The only motivation to use multiple processes is to be able to parallelize blocking calls (since event driven handling is not useful) In my case forward proxy is not a bottleneck, however when my code spawns multiple processes as per http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multiprocessor
and I run multiple concurrent clients, I see that the server's processes are not efficiently utilized. Specifically, even if there are outstanding requests, one of the processes accepts requests lot more while other processes are idling. I was able to confirm this by printing the Process IDs of the processes which are handling request This sort of imbalance could lead to underutilization of CPU - also discussed here https://lwn.net/Articles/542629/ Hence I was curious to know how Twisted multiprocessing works. It seems SO_REUSEPORT is not used within Twisted and as per Gavin's reply can be used "outside" of Twisted. On Wed, Jun 10, 2015 at 5:42 PM, Carl Waldbieser <cwaldbie...@gmail.com> wrote: > There is a twisted developer guide for communicating with child > processes: > http://twistedmatrix.com/documents/current/core/howto/process.html > > I'm not sure I understand why having multiple processes listen on the same > socket is desirable in your case. From reading the articles you linked to, > it seems like it is only useful in the case where a forward proxy becomes a > bottleneck. Is that the case wirh your application? > > Thanks, > Carl Waldbieser > On Jun 9, 2015 10:03 PM, "Sagar Dixit" <sagar.di...@gmail.com> wrote: > >> Hi, >> >> I am exploring Twisted Web for my RESTful application. My application is >> stateless and involves storing and retrieving objects based on Object-ID. >> This application will run on beefy (multicore, lots of memory) machine. >> However, not all APIs that the application issues to underlying storage are >> async and hence I cannot fully utilize Deferreds >> Which means, there will some blocking calls and hence my primary interest >> is to use Twisted Web in multiprocessing mode >> >> I came across >> http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multiprocessor >> >> However, I am not sure if it is the "correct" way of doing things. >> >> Hence I had some questions around it: >> >> 1. Is there an interface (similar to defertoThread) which allows me to >> execute a blocking call in a separate process ? >> >> 2. Does reactor synchronize access of all processes to the shared listen >> socket ? >> >> 3. Is there a sample code I can refer to where the application is >> spawning subprocesses to handle HTTP requests ? >> >> Thanks in advance! >> >> >> _______________________________________________ >> Twisted-web mailing list >> Twisted-web@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web >> >> > _______________________________________________ > Twisted-web mailing list > Twisted-web@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web > > -- ssdixit
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
