Actually python is fairly easy to get muti-threaded, and with a little more work you can even get muti-processing working. All you really need is a master worker that monitors the number of workers and limits the spawn of additional workers. I use a 6 worker multi-process when doing my database dump scans, each worker processes 500 pages and then dies. When the master process sees a sub-process die it grabs another 500 pages from the work queue and launches a replacement process. With a little monitoring and tweaking you can adjust both the amount of work and number of workers to work withing a reasonable resource limit. This way the work can still get done fairly quickly and you are only using one job queue slot. (Each slot isnt limited to a single process, it just calculates total resource usage and compares that to the limits set, and if a job exceeds the max set it kills it.) Depending on what your bounding factor is either mut-threading or muti-processing should work for you without too much headache in the conversion.
On Mon, Mar 30, 2015 at 12:37 AM, Anthony Di Franco <[email protected]> wrote: > The intent of lots of jobs is to have lots of redis queue workers making > network requests in parallel, so it would probably be significant work with > python's threading or twisted or the like to consolidate multiple requests > into a single OS process, but I'll start looking into it, and the resource > increase seems to have us covered in the meantime. Thanks! > > On Sun, Mar 29, 2015 at 7:49 PM, Yuvi Panda <[email protected]> wrote: > >> Part of the reason was that they asked for trusty and we only had one >> trusty node working for exec hosts, apparently. I have just added 5 >> more and that should help. >> >> However, +1 to what Coren said. Can you simplify your code to combine >> the tasks together? >> >> _______________________________________________ >> Labs-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/labs-l >> > > > _______________________________________________ > Labs-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/labs-l > >
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
