Re: [OMPI users] importing to MPI data already in memory from another process
On Jun 27, 2014, at 8:53 AM, Brock Palenwrote: > Is there a way to import/map memory from a process (data acquisition) such > that an MPI program could 'take' or see that memory? > > We have a need to do data acquisition at the rate of .7TB/s and need todo > some shuffles/computation on these data, some of the nodes are directly > connected to the device, and some will do processing. > > Here is the proposed flow: > > * Data collector nodes runs process collecting data from device > * Those nodes somehow pass the data to an MPI job running on these nodes and > a number of other nodes (cpu need for filterting is greater than what the 16 > data nodes can provide). For a non-MPI solution for intranode data transfer in this case, take a look at vmsplice(2): http://man7.org/linux/man-pages/man2/vmsplice.2.html Pay particular attention to the SPLICE_F_GIFT flag, which will allow you to simply give memory pages away to the MPI process, avoiding unnecessary data copies. You would just need a pipe shared between the data collector process and the MPI process (and to be a bit careful with your memory allocation/management, since any page you gift away should probably come from mmap(2) directly). Otherwise, as George mentioned, I would investigate converting your current data collector processes to also be MPI processes so that they can simply communicate the data to the rest of the cluster. -Dave
Re: [OMPI users] importing to MPI data already in memory from another process
MPI "universe" yes, but not necessarily MPI "world". You could have the two worlds connect/accept or join (https://www.open-mpi.org/doc/v1.8/man3/MPI_Comm_join.3.php) and then you should be able to take advantage of the RMA. At least, that is what is written in the book ... George. On Fri, Jun 27, 2014 at 10:30 AM, Brock Palenwrote: > But this is within the same MPI "universe" right? > > Brock Palen > www.umich.edu/~brockp > CAEN Advanced Computing > XSEDE Campus Champion > bro...@umich.edu > (734)936-1985 > > > > On Jun 27, 2014, at 10:19 AM, George Bosilca wrote: > >> The One-Sided Communications from the Chapter 11 of the MPI standard? >> For processes on the same node you might want to look at >> MPI_WIN_ALLOCATE_SHARED. >> >> George. >> >> On Fri, Jun 27, 2014 at 9:53 AM, Brock Palen wrote: >>> Is there a way to import/map memory from a process (data acquisition) such >>> that an MPI program could 'take' or see that memory? >>> >>> We have a need to do data acquisition at the rate of .7TB/s and need todo >>> some shuffles/computation on these data, some of the nodes are directly >>> connected to the device, and some will do processing. >>> >>> Here is the proposed flow: >>> >>> * Data collector nodes runs process collecting data from device >>> * Those nodes somehow pass the data to an MPI job running on these nodes >>> and a number of other nodes (cpu need for filterting is greater than what >>> the 16 data nodes can provide). >>> >>> One thought is to have the data collector processes be threads inside the >>> MPI job running across all nodes, but was curious is there is a way to pass >>> data still in memory (to much to hit disk) to the running MPI filter job. >>> >>> Thanks! >>> >>> Brock Palen >>> www.umich.edu/~brockp >>> CAEN Advanced Computing >>> XSEDE Campus Champion >>> bro...@umich.edu >>> (734)936-1985 >>> >>> >>> >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/06/24716.php >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/06/24717.php > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/06/24718.php
Re: [OMPI users] importing to MPI data already in memory from another process
The One-Sided Communications from the Chapter 11 of the MPI standard? For processes on the same node you might want to look at MPI_WIN_ALLOCATE_SHARED. George. On Fri, Jun 27, 2014 at 9:53 AM, Brock Palenwrote: > Is there a way to import/map memory from a process (data acquisition) such > that an MPI program could 'take' or see that memory? > > We have a need to do data acquisition at the rate of .7TB/s and need todo > some shuffles/computation on these data, some of the nodes are directly > connected to the device, and some will do processing. > > Here is the proposed flow: > > * Data collector nodes runs process collecting data from device > * Those nodes somehow pass the data to an MPI job running on these nodes and > a number of other nodes (cpu need for filterting is greater than what the 16 > data nodes can provide). > > One thought is to have the data collector processes be threads inside the MPI > job running across all nodes, but was curious is there is a way to pass data > still in memory (to much to hit disk) to the running MPI filter job. > > Thanks! > > Brock Palen > www.umich.edu/~brockp > CAEN Advanced Computing > XSEDE Campus Champion > bro...@umich.edu > (734)936-1985 > > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/06/24716.php
[OMPI users] importing to MPI data already in memory from another process
Is there a way to import/map memory from a process (data acquisition) such that an MPI program could 'take' or see that memory? We have a need to do data acquisition at the rate of .7TB/s and need todo some shuffles/computation on these data, some of the nodes are directly connected to the device, and some will do processing. Here is the proposed flow: * Data collector nodes runs process collecting data from device * Those nodes somehow pass the data to an MPI job running on these nodes and a number of other nodes (cpu need for filterting is greater than what the 16 data nodes can provide). One thought is to have the data collector processes be threads inside the MPI job running across all nodes, but was curious is there is a way to pass data still in memory (to much to hit disk) to the running MPI filter job. Thanks! Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion bro...@umich.edu (734)936-1985 signature.asc Description: Message signed with OpenPGP using GPGMail