There are two main uses of network in Spark (ignoring Spark Streaming) through 1. spark.network stuff for bulk data transfer 2. Akka <http://akka.io/> actor library for control plane messaging
Besides porting the spark.network, you will also have to port Akka to run on your stack. You will find most of the control layer class like DAGScheduler, BlockManager, etc uses actor to communicate as well as to process events in a single-threaded fashion. For example, the BlockManagerMaster (driver) and the BlockManager (worker) communicate (control messages only) using the BlockManagerMasterActor and BlockManagerSlaveActor, respectively. A good place to start would be to first read up on Akka from online docs<http://akka.io/docs/> and look at the code to see how we use it. TD On Thu, Jan 2, 2014 at 11:03 PM, Kai Backman <k...@airstonelabs.com> wrote: > dev, > > I would be interesting in understanding how to make core Spark run on a non > IP network stack like MPI. The main dependencies seem to be > in org.apache.spark.network but I also see some other dependencies > sprinkled in auxiliary functions. > > Pointers to code, mailing list discussions or people to talk to > appreciated. > > Take care, > > Kai > > -- > Kai Backman, CEO > http://airstonelabs.com >
