I think I got around the second problem (thanks for sharing the react documentation)
How do I use more multiple CPU cores? I guess atleast the fact that twisted app wont be blocking is good, but what would be a good way to use multiple cores. I do see that meejah suggested using multiple python processes while setting up the reactor? My question is that.. say.. I set up multiple twisted based python processes by leveraging the spawnProcess.. And then we setup a resource 'foo'.. When the request is hit to `foo` end point... will the parallelization happen here.. say.. I have spawned 2 processes and one request came and hit the 'foo' end point.. I am guessing.. the fact that 2 processes is running would help responding to multiple requests simultaneously. I think that level of parallelization is good and I will take it.. But here.. what I want is... when you hit "foo".. then there is a function that is parallelizable.. Like.. you hit "foo" endpoint.. I fetch data from somewhere.. (yay treq).. and then there is a function that is called when we hit "foo" end point and that function is extremely parallelizable over for loop. Now, what is great is that with all your help, I am able to make this flow non-blocking. But, would definitely like to exploit parallellism wherever feasible. On Mon, Jun 24, 2019 at 7:37 PM Moshe Zadka <mos...@zadka.club> wrote: > On Mon, Jun 24, 2019, at 16:27, Chengi Liu wrote: > > cpu_res *= yield *defer.*gatherResults*(cpus) > > > Remember: This will not block the reactor (Good!) but will still limit you > to one CPU core (in general, some caveats, but true.) If you are CPU bound, > this is woefully underutilizing modern CPU resources (again, some caveats > here). > > if __name__ == '__main__': > > > In general, the best thing to do is to to use twisted.internet.task.react > for main. This would also solve the other problem you point out below. > > > https://twistedmatrix.com/documents/current/api/twisted.internet.task.html#react > has > a good example. > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
