You could use zyre ( https://github.com/zeromq/zyre ) to create a network of worker nodes and then distribute tasks and collate results with a central server.
On Thu, Aug 27, 2015 at 12:58 PM, Keith Brown <[email protected]> wrote: > at the moment I am using Python multiprocessing to process requests > but it won't scale well for over 30+ processes since I don't have that > many cores. However, I have many desktops in our lab which are idle. I > was wondering if its possible to use python multiprocessing module > with zero mq to distribute my tasks and get results quickly. > > BTW, each process takes about 1 second to run but there are thousands > I need to run. > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
