Even simpler, check out the task pipeline example in the Guide. You can run as many task workers on the idle boxes you like, each connecting to a fixed IP address/port on your main machine. It's not elegant yet it'll work in a few minutes and all exists in Python.
On Thu, Aug 27, 2015 at 2:42 PM, Joss Gray <[email protected]> wrote: > You could use zyre ( https://github.com/zeromq/zyre ) to create a network > of worker nodes and then distribute tasks and collate results with a central > server. > > On Thu, Aug 27, 2015 at 12:58 PM, Keith Brown <[email protected]> wrote: >> >> at the moment I am using Python multiprocessing to process requests >> but it won't scale well for over 30+ processes since I don't have that >> many cores. However, I have many desktops in our lab which are idle. I >> was wondering if its possible to use python multiprocessing module >> with zero mq to distribute my tasks and get results quickly. >> >> BTW, each process takes about 1 second to run but there are thousands >> I need to run. >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
