Take a look into gevent. For most workloads you just need to monkeypatch all and it'll just magically thread your network requests.
On Sun, Mar 29, 2015 at 9:37 PM, Anthony Di Franco <[email protected]> wrote: > The intent of lots of jobs is to have lots of redis queue workers making > network requests in parallel, so it would probably be significant work with > python's threading or twisted or the like to consolidate multiple requests > into a single OS process, but I'll start looking into it, and the resource > increase seems to have us covered in the meantime. Thanks! > > On Sun, Mar 29, 2015 at 7:49 PM, Yuvi Panda <[email protected]> wrote: > >> Part of the reason was that they asked for trusty and we only had one >> trusty node working for exec hosts, apparently. I have just added 5 >> more and that should help. >> >> However, +1 to what Coren said. Can you simplify your code to combine >> the tasks together? >> >> _______________________________________________ >> Labs-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/labs-l >> > > > _______________________________________________ > Labs-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/labs-l > >
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
