I think this would be a significant undertaking — my suggestion would be to make the network emulate IP (I believe that option exists for many networks) and then only optimize individual packages where it makes sense. Lots of the communication is just “control plane” (small messages) and only the block manager does more expensive transfers.
Matei On Jan 3, 2014, at 2:03 AM, Kai Backman <k...@airstonelabs.com> wrote: > dev, > > I would be interesting in understanding how to make core Spark run on a non > IP network stack like MPI. The main dependencies seem to be > in org.apache.spark.network but I also see some other dependencies > sprinkled in auxiliary functions. > > Pointers to code, mailing list discussions or people to talk to appreciated. > > Take care, > > Kai > > -- > Kai Backman, CEO > http://airstonelabs.com
