Re: [OMPI users] importing to MPI data already in memory from another process

[Dave Goodell (dgoodell)]

On Jun 27, 2014, at 8:53 AM, Brock Palen  wrote:

> Is there a way to import/map memory from a process (data acquisition) such 
> that an MPI program could 'take' or see that memory?
> 
> We have a need to do data acquisition at the rate of .7TB/s and need todo 
> some shuffles/computation on these data,  some of the nodes are directly 
> connected to the device, and some will do processing. 
> 
> Here is the proposed flow:
> 
> * Data collector nodes runs process collecting data from device
> * Those nodes somehow pass the data to an MPI job running on these nodes and 
> a number of other nodes (cpu need for filterting is greater than what the 16 
> data nodes can provide).

For a non-MPI solution for intranode data transfer in this case, take a look at 
vmsplice(2):

http://man7.org/linux/man-pages/man2/vmsplice.2.html

Pay particular attention to the SPLICE_F_GIFT flag, which will allow you to 
simply give memory pages away to the MPI process, avoiding unnecessary data 
copies.  You would just need a pipe shared between the data collector process 
and the MPI process (and to be a bit careful with your memory 
allocation/management, since any page you gift away should probably come from 
mmap(2) directly).


Otherwise, as George mentioned, I would investigate converting your current 
data collector processes to also be MPI processes so that they can simply 
communicate the data to the rest of the cluster.

-Dave

Re: [OMPI users] importing to MPI data already in memory from another process

[George Bosilca]

MPI "universe" yes, but not necessarily MPI "world". You could have
the two worlds connect/accept or join
(https://www.open-mpi.org/doc/v1.8/man3/MPI_Comm_join.3.php) and then
you should be able to take advantage of the RMA. At least, that is
what is written in the book ...

  George.

On Fri, Jun 27, 2014 at 10:30 AM, Brock Palen  wrote:
> But this is within the same MPI "universe" right?
>
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> XSEDE Campus Champion
> bro...@umich.edu
> (734)936-1985
>
>
>
> On Jun 27, 2014, at 10:19 AM, George Bosilca  wrote:
>
>> The One-Sided Communications from the Chapter 11 of the MPI standard?
>> For processes on the same node you might want to look at
>> MPI_WIN_ALLOCATE_SHARED.
>>
>>  George.
>>
>> On Fri, Jun 27, 2014 at 9:53 AM, Brock Palen  wrote:
>>> Is there a way to import/map memory from a process (data acquisition) such 
>>> that an MPI program could 'take' or see that memory?
>>>
>>> We have a need to do data acquisition at the rate of .7TB/s and need todo 
>>> some shuffles/computation on these data,  some of the nodes are directly 
>>> connected to the device, and some will do processing.
>>>
>>> Here is the proposed flow:
>>>
>>> * Data collector nodes runs process collecting data from device
>>> * Those nodes somehow pass the data to an MPI job running on these nodes 
>>> and a number of other nodes (cpu need for filterting is greater than what 
>>> the 16 data nodes can provide).
>>>
>>> One thought is to have the data collector processes be threads inside the 
>>> MPI job running across all nodes, but was curious is there is a way to pass 
>>> data still in memory (to much to hit disk) to the running MPI filter job.
>>>
>>> Thanks!
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> CAEN Advanced Computing
>>> XSEDE Campus Champion
>>> bro...@umich.edu
>>> (734)936-1985
>>>
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2014/06/24716.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/06/24717.php
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24718.php

Re: [OMPI users] importing to MPI data already in memory from another process

[George Bosilca]

The One-Sided Communications from the Chapter 11 of the MPI standard?
For processes on the same node you might want to look at
MPI_WIN_ALLOCATE_SHARED.

  George.

On Fri, Jun 27, 2014 at 9:53 AM, Brock Palen  wrote:
> Is there a way to import/map memory from a process (data acquisition) such 
> that an MPI program could 'take' or see that memory?
>
> We have a need to do data acquisition at the rate of .7TB/s and need todo 
> some shuffles/computation on these data,  some of the nodes are directly 
> connected to the device, and some will do processing.
>
> Here is the proposed flow:
>
> * Data collector nodes runs process collecting data from device
> * Those nodes somehow pass the data to an MPI job running on these nodes and 
> a number of other nodes (cpu need for filterting is greater than what the 16 
> data nodes can provide).
>
> One thought is to have the data collector processes be threads inside the MPI 
> job running across all nodes, but was curious is there is a way to pass data 
> still in memory (to much to hit disk) to the running MPI filter job.
>
> Thanks!
>
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> XSEDE Campus Champion
> bro...@umich.edu
> (734)936-1985
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24716.php

[OMPI users] importing to MPI data already in memory from another process

[Brock Palen]

Is there a way to import/map memory from a process (data acquisition) such that 
an MPI program could 'take' or see that memory?

We have a need to do data acquisition at the rate of .7TB/s and need todo some 
shuffles/computation on these data,  some of the nodes are directly connected 
to the device, and some will do processing. 

Here is the proposed flow:

* Data collector nodes runs process collecting data from device
* Those nodes somehow pass the data to an MPI job running on these nodes and a 
number of other nodes (cpu need for filterting is greater than what the 16 data 
nodes can provide).

One thought is to have the data collector processes be threads inside the MPI 
job running across all nodes, but was curious is there is a way to pass data 
still in memory (to much to hit disk) to the running MPI filter job.

Thanks! 

Brock Palen
www.umich.edu/~brockp
CAEN Advanced Computing
XSEDE Campus Champion
bro...@umich.edu
(734)936-1985





signature.asc
Description: Message signed with OpenPGP using GPGMail