On Jun 27, 2014, at 8:53 AM, Brock Palen <bro...@umich.edu> wrote: > Is there a way to import/map memory from a process (data acquisition) such > that an MPI program could 'take' or see that memory? > > We have a need to do data acquisition at the rate of .7TB/s and need todo > some shuffles/computation on these data, some of the nodes are directly > connected to the device, and some will do processing. > > Here is the proposed flow: > > * Data collector nodes runs process collecting data from device > * Those nodes somehow pass the data to an MPI job running on these nodes and > a number of other nodes (cpu need for filterting is greater than what the 16 > data nodes can provide).
For a non-MPI solution for intranode data transfer in this case, take a look at vmsplice(2): http://man7.org/linux/man-pages/man2/vmsplice.2.html Pay particular attention to the SPLICE_F_GIFT flag, which will allow you to simply give memory pages away to the MPI process, avoiding unnecessary data copies. You would just need a pipe shared between the data collector process and the MPI process (and to be a bit careful with your memory allocation/management, since any page you gift away should probably come from mmap(2) directly). Otherwise, as George mentioned, I would investigate converting your current data collector processes to also be MPI processes so that they can simply communicate the data to the rest of the cluster. -Dave
