Re: feasibility of ignite and alluxio for interfacing MPI and Spark

2016-09-19 Thread Calvin Jia
Hi, Alluxio allows for data sharing between applications through a File System API (Native Java Alluxio client, Hadoop FileSystem, or POSIX through fuse). If your MPI applications can use any of these interfaces, you should be able to use Alluxio for data sharing out of the box. In terms of

feasibility of ignite and alluxio for interfacing MPI and Spark

2016-09-16 Thread AlexG
Do Ignite and Alluxio offer reasonable means of transferring data, in memory, from Spark to MPI? A straightforward way to transfer data is use piping, but unless you have MPI processes running in a one-to-one mapping to the Spark partitions, this will require some complicated logic to get working