Hi Brian, > Great article. Our main usage case for 0MQ + Python is a parallel > computing framework that we have been working on for years now. > Originally we were targeting traditional Gb/Infiniband based clusters, > but that entire landscape is really changing. The new AMD 6100 CPUs > that just came out are a huge step towards massively multicore. You > can now get a 48 core machine for not too much $. But my word, what > software can use 48 cores (other than using it as a virtualization > platform)? Other than Erlang, I can't think of anything better than > 0MQ at this point. > > Some other thoughts and comments: > > * On a massively multicore machine, what transport layer makes to most > sense? IPC? tcp:localhost? What would be the fastest transport > possible?
Obviously, inproc transport. However, in your case where Python is not able to run in parallel, it's presumably ipc. IPC is faster than tcp:localhost on Linux, but exactly the same on Solaris. Other systems would have their own perf characteristics. > * When using multiple massively multicore machines, which transport > layer makes the most sense? It would be nice if processes on > different hosts could use tcp or infiniband, but processes on the same > host could use something faster. That's exactly the use case behind 0MQ being able to bind to multiple endpoints. So you can for example bind a socket to TCP port _and_ to inproc endpoint. That way threads in the same process can connect via extra-fast inproc transport while processes on different boxes can use TCP. > * Python still has the Global interpreter lock that prevents more than > one thread from executing Python code at a time. This means that > thread based concurrency is not an option for Python. IOW, we can't > use multiple threads that talk using OMQ. So, we have to use > processes. I don't like that, but maybe it is OK... Well, it's deficiency of Python. Hopefuly Python developers would address the issue in the future. Btw, maybe it would be worth poking them a little by asking about the behaviour on python dev mailing list... Martin _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
