indeed. the task pipeline looks great. I will start a new thread to ask about details on it.
On Sat, Aug 29, 2015 at 5:42 AM, Pieter Hintjens <[email protected]> wrote: > Even simpler, check out the task pipeline example in the Guide. You > can run as many task workers on the idle boxes you like, each > connecting to a fixed IP address/port on your main machine. It's not > elegant yet it'll work in a few minutes and all exists in Python. > > On Thu, Aug 27, 2015 at 2:42 PM, Joss Gray <[email protected]> wrote: >> You could use zyre ( https://github.com/zeromq/zyre ) to create a network >> of worker nodes and then distribute tasks and collate results with a central >> server. >> >> On Thu, Aug 27, 2015 at 12:58 PM, Keith Brown <[email protected]> wrote: >>> >>> at the moment I am using Python multiprocessing to process requests >>> but it won't scale well for over 30+ processes since I don't have that >>> many cores. However, I have many desktops in our lab which are idle. I >>> was wondering if its possible to use python multiprocessing module >>> with zero mq to distribute my tasks and get results quickly. >>> >>> BTW, each process takes about 1 second to run but there are thousands >>> I need to run. >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
