Re: [OMPI users] Problem moving from 1.4 to 1.6

[Ralph Castain]

Let me steer you on a different course. Can you run "ompi_info" and paste the 
output here? It looks to me like someone installed a version that includes 
uDAPL support, so you may have to disable some additional things to get it to 
run.


On Jun 27, 2014, at 9:53 AM, Jeffrey A Cummings  
wrote:

> We have recently upgraded our cluster to a version of Linux which comes with 
> openMPI version 1.6.2. 
> 
> An application which ran previously (using some version of 1.4) now errors 
> out with the following messages: 
> 
> librdmacm: Fatal: no RDMA devices found 
> librdmacm: Fatal: no RDMA devices found 
> librdmacm: Fatal: no RDMA devices found 
> 
> -- 
> WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:]. 
> This may be a real error or it may be an invalid entry in the uDAPL 
> Registry which is contained in the dat.conf file. Contact your local 
> System Administrator to confirm the availability of the interfaces in 
> the dat.conf file. 
> 
> -- 
> [tupile:25363] 2 more processes have sent help message 
> help-mpi-btl-udapl.txt / dat_ia_open fail 
> [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to 0 to 
> see all help / error messages 
> 
> The mpirun command line contains the argument '--mca btl ^openib', which I 
> thought told mpi to not look for the ib interface. 
> 
> Can anyone suggest what the problem might be?  Did the relevant syntax change 
> between versions 1.4 and 1.6? 
> 
> 
> Jeffrey A. Cummings
> Engineering Specialist
> Performance Modeling and Analysis Department
> Systems Analysis and Simulation Subdivision
> Systems Engineering Division
> Engineering and Technology Group
> The Aerospace Corporation
> 571-307-4220
> jeffrey.a.cummi...@aero.org 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24721.php

Re: [OMPI users] Problem moving from 1.4 to 1.6

[Gus Correa]

If you don't have control over the MPI version/versions/implementations 
installed, you probably can still verify if your environment is 
consistently pointing to the same MPI implementation and version.


It is not uncommon to have more than one implementation and version
installed on a computer, on a cluster, or worse, different 
versions+implementations on different cluster nodes.

Mixed-up environment variables can produce very confusing results.

Commands such as:

which mpiexec
which mpicc
which mpif90

and also

mpiexec --version
mpicc --show
etc

may help diagnose that.

Likewise,

env |grep PATH

and

env |grep LD_LIBRARY_PATH

may hint if you have a mixed environment and mixed MPI implementations 
and versions.


I hope this helps,
Gus Correa

PS - BTW, unless your company's policies forbid,
you can install OpenMPI on a user directory, say, your /home directory. 
 This will work if that directory is shared across the cluster (e.g. 
via NFS), and as long as you set your PATH and LD_LIBRARY_PATH to point 
to its bin and lib subdirectories.


https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path

On 06/27/2014 01:56 PM, Jeffrey A Cummings wrote:

I appreciate your response and I understand the logic behind your
suggestion, but you and the other regular expert contributors to this
list are frequently working under a misapprehension.  Many of your
openMPI users don't have any control over what version of openMPI is
available on their system.  I'm stuck with whatever version my IT people
choose to bless, which in general is the (possibly old and/or moldy)
version that is bundled with some larger package (i.e., Rocks, Linux).
  The fact that I'm only now seeing this 1.4 to 1.6 problem illustrates
the situation I'm in.  I really need someone to did into their memory
archives to see if they can come up with a clue for me.

Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org



From: Gus Correa 
To: Open MPI Users ,
Date: 06/27/2014 01:45 PM
Subject: Re: [OMPI users] Problem moving from 1.4 to 1.6
Sent by: "users" 




It may be easier to install the latest OMPI from the tarball,
rather than trying to sort out the error.

http://www.open-mpi.org/software/ompi/v1.8/

The packaged built of (somewhat old) OMPI 1.6.2 that came with
Linux may not have built against the same IB libraries, hardware,
and configuration you have.
[The error message reference to udapl is ominous.]

 > The mpirun command line contains the argument '--mca btl ^openib', which
 > I thought told mpi to not look for the ib interface.

As you said, the mca parameter above tells OMPI not to use openib,
although it may not be the only cause of the problem.
If you want to use openib switch to
--mca btl openib,sm,self

Another thing to check is whether there is a mixup of enviroment
variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI
version you may have installed.

My two cents,
Gus Correa

On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:
 > We have recently upgraded our cluster to a version of Linux which comes
 > with openMPI version 1.6.2.
 >
 > An application which ran previously (using some version of 1.4) now
 > errors out with the following messages:
 >
 >  librdmacm: Fatal: no RDMA devices found
 >  librdmacm: Fatal: no RDMA devices found
 >  librdmacm: Fatal: no RDMA devices found
 >
 >
--
 >  WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
 >  This may be a real error or it may be an invalid entry in the
 > uDAPL
 >  Registry which is contained in the dat.conf file. Contact your
 > local
 >  System Administrator to confirm the availability of the
 > interfaces in
 >  the dat.conf file.
 >
 >
--
 >  [tupile:25363] 2 more processes have sent help message
 > help-mpi-btl-udapl.txt / dat_ia_open fail
 >  [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to
 > 0 to see all help / error messages
 >
 > The mpirun command line contains the argument '--mca btl ^openib', which
 > I thought told mpi to not look for the ib interface.
 >
 > Can anyone suggest what the problem might be?  Did the relevant syntax
 > change between versions 1.4 and 1.6?
 >
 >
 > Jeffrey A. Cummings
 > Engineering Specialist
 > Performance Modeling and Analysis Department
 > Systems Analysis and Simulation Subdivision
 > Systems Engineering Division
 > Engineering and Technology Group
 > The Aerospace Corporation
 > 571-307-4220
 > 

Re: [OMPI users] Problem moving from 1.4 to 1.6

[Jeffrey A Cummings]

Once again, you guys are assuming (incorrectly) that all your users are 
working in an environment where everyone is free (based on corporate IT 
policies) to do things like that.  As an aside, you're also assuming that 
all your users are Unix/Linux experts.  I've been following this list for 
several years and couldn't even begin to count the number of questions 
from the non-experts who are struggling with something which is trivial 
for you but not for them.

Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org



From:   Reuti 
To: Open MPI Users , 
List-Post: users@lists.open-mpi.org
Date:   06/27/2014 02:03 PM
Subject:Re: [OMPI users] Problem moving from 1.4 to 1.6
Sent by:"users" 



Hi,

Am 27.06.2014 um 19:56 schrieb Jeffrey A Cummings:

> I appreciate your response and I understand the logic behind your 
suggestion, but you and the other regular expert contributors to this list 
are frequently working under a misapprehension.  Many of your openMPI 
users don't have any control over what version of openMPI is available on 
their system.  I'm stuck with whatever version my IT people choose to 
bless, which in general is the (possibly old and/or moldy) version that is 
bundled with some larger package (i.e., Rocks, Linux).  The fact that I'm 
only now seeing this 1.4 to 1.6 problem illustrates the situation I'm in. 
I really need someone to did into their memory archives to see if they can 
come up with a clue for me.

You can freely download the Open MPI source and install it for example in 
your personal ~/local/openmpi-1.8 or alike. Pointing your $PATH and 
$LD_LIBRARY_PATH to your own version will supersede installed system one.

-- Reuti


> Jeffrey A. Cummings
> Engineering Specialist
> Performance Modeling and Analysis Department
> Systems Analysis and Simulation Subdivision
> Systems Engineering Division
> Engineering and Technology Group
> The Aerospace Corporation
> 571-307-4220
> jeffrey.a.cummi...@aero.org 
> 
> 
> 
> From:Gus Correa  
> To:Open MPI Users , 
> Date:06/27/2014 01:45 PM 
> Subject:Re: [OMPI users] Problem moving from 1.4 to 1.6 
> Sent by:"users"  
> 
> 
> 
> It may be easier to install the latest OMPI from the tarball,
> rather than trying to sort out the error.
> 
> http://www.open-mpi.org/software/ompi/v1.8/
> 
> The packaged built of (somewhat old) OMPI 1.6.2 that came with
> Linux may not have built against the same IB libraries, hardware,
> and configuration you have.
> [The error message reference to udapl is ominous.]
> 
> > The mpirun command line contains the argument '--mca btl ^openib', 
which
> > I thought told mpi to not look for the ib interface.
> 
> As you said, the mca parameter above tells OMPI not to use openib,
> although it may not be the only cause of the problem.
> If you want to use openib switch to
> --mca btl openib,sm,self
> 
> Another thing to check is whether there is a mixup of enviroment 
> variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI 
> version you may have installed.
> 
> My two cents,
> Gus Correa
> 
> On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:
> > We have recently upgraded our cluster to a version of Linux which 
comes
> > with openMPI version 1.6.2.
> >
> > An application which ran previously (using some version of 1.4) now
> > errors out with the following messages:
> >
> >  librdmacm: Fatal: no RDMA devices found
> >  librdmacm: Fatal: no RDMA devices found
> >  librdmacm: Fatal: no RDMA devices found
> >
> > 
--
> >  WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
> >  This may be a real error or it may be an invalid entry in the
> > uDAPL
> >  Registry which is contained in the dat.conf file. Contact 
your
> > local
> >  System Administrator to confirm the availability of the
> > interfaces in
> >  the dat.conf file.
> >
> > 
--
> >  [tupile:25363] 2 more processes have sent help message
> > help-mpi-btl-udapl.txt / dat_ia_open fail
> >  [tupile:25363] Set MCA parameter "orte_base_help_aggregate" 
to
> > 0 to see all help / error messages
> >
> > The mpirun command line contains the argument '--mca btl ^openib', 
which
> > I thought told mpi to not look for the ib interface.
> >
> > Can anyone suggest what the problem might be?  Did the relevant syntax
> > change between versions 1.4 and 1.6?
> >
> >
> > Jeffrey A. Cummings
> > Engineering Specialist
> > 

Re: [OMPI users] Problem moving from 1.4 to 1.6

[Reuti]

Hi,

Am 27.06.2014 um 19:56 schrieb Jeffrey A Cummings:

> I appreciate your response and I understand the logic behind your suggestion, 
> but you and the other regular expert contributors to this list are frequently 
> working under a misapprehension.  Many of your openMPI users don't have any 
> control over what version of openMPI is available on their system.  I'm stuck 
> with whatever version my IT people choose to bless, which in general is the 
> (possibly old and/or moldy) version that is bundled with some larger package 
> (i.e., Rocks, Linux).  The fact that I'm only now seeing this 1.4 to 1.6 
> problem illustrates the situation I'm in.  I really need someone to did into 
> their memory archives to see if they can come up with a clue for me.

You can freely download the Open MPI source and install it for example in your 
personal ~/local/openmpi-1.8 or alike. Pointing your $PATH and $LD_LIBRARY_PATH 
to your own version will supersede installed system one.

-- Reuti


> Jeffrey A. Cummings
> Engineering Specialist
> Performance Modeling and Analysis Department
> Systems Analysis and Simulation Subdivision
> Systems Engineering Division
> Engineering and Technology Group
> The Aerospace Corporation
> 571-307-4220
> jeffrey.a.cummi...@aero.org 
> 
> 
> 
> From:Gus Correa  
> To:Open MPI Users , 
> Date:06/27/2014 01:45 PM 
> Subject:Re: [OMPI users] Problem moving from 1.4 to 1.6 
> Sent by:"users"  
> 
> 
> 
> It may be easier to install the latest OMPI from the tarball,
> rather than trying to sort out the error.
> 
> http://www.open-mpi.org/software/ompi/v1.8/
> 
> The packaged built of (somewhat old) OMPI 1.6.2 that came with
> Linux may not have built against the same IB libraries, hardware,
> and configuration you have.
> [The error message reference to udapl is ominous.]
> 
> > The mpirun command line contains the argument '--mca btl ^openib', which
> > I thought told mpi to not look for the ib interface.
> 
> As you said, the mca parameter above tells OMPI not to use openib,
> although it may not be the only cause of the problem.
> If you want to use openib switch to
> --mca btl openib,sm,self
> 
> Another thing to check is whether there is a mixup of enviroment 
> variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI 
> version you may have installed.
> 
> My two cents,
> Gus Correa
> 
> On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:
> > We have recently upgraded our cluster to a version of Linux which comes
> > with openMPI version 1.6.2.
> >
> > An application which ran previously (using some version of 1.4) now
> > errors out with the following messages:
> >
> >  librdmacm: Fatal: no RDMA devices found
> >  librdmacm: Fatal: no RDMA devices found
> >  librdmacm: Fatal: no RDMA devices found
> >
> > --
> >  WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
> >  This may be a real error or it may be an invalid entry in the
> > uDAPL
> >  Registry which is contained in the dat.conf file. Contact your
> > local
> >  System Administrator to confirm the availability of the
> > interfaces in
> >  the dat.conf file.
> >
> > --
> >  [tupile:25363] 2 more processes have sent help message
> > help-mpi-btl-udapl.txt / dat_ia_open fail
> >  [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to
> > 0 to see all help / error messages
> >
> > The mpirun command line contains the argument '--mca btl ^openib', which
> > I thought told mpi to not look for the ib interface.
> >
> > Can anyone suggest what the problem might be?  Did the relevant syntax
> > change between versions 1.4 and 1.6?
> >
> >
> > Jeffrey A. Cummings
> > Engineering Specialist
> > Performance Modeling and Analysis Department
> > Systems Analysis and Simulation Subdivision
> > Systems Engineering Division
> > Engineering and Technology Group
> > The Aerospace Corporation
> > 571-307-4220
> > jeffrey.a.cummi...@aero.org
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post: 
> > http://www.open-mpi.org/community/lists/users/2014/06/24721.php
> >
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24722.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> 

Re: [OMPI users] Problem moving from 1.4 to 1.6

[Jeffrey A Cummings]

I appreciate your response and I understand the logic behind your 
suggestion, but you and the other regular expert contributors to this list 
are frequently working under a misapprehension.  Many of your openMPI 
users don't have any control over what version of openMPI is available on 
their system.  I'm stuck with whatever version my IT people choose to 
bless, which in general is the (possibly old and/or moldy) version that is 
bundled with some larger package (i.e., Rocks, Linux).  The fact that I'm 
only now seeing this 1.4 to 1.6 problem illustrates the situation I'm in. 
I really need someone to did into their memory archives to see if they can 
come up with a clue for me.

Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org



From:   Gus Correa 
To: Open MPI Users , 
List-Post: users@lists.open-mpi.org
Date:   06/27/2014 01:45 PM
Subject:Re: [OMPI users] Problem moving from 1.4 to 1.6
Sent by:"users" 



It may be easier to install the latest OMPI from the tarball,
rather than trying to sort out the error.

http://www.open-mpi.org/software/ompi/v1.8/

The packaged built of (somewhat old) OMPI 1.6.2 that came with
Linux may not have built against the same IB libraries, hardware,
and configuration you have.
[The error message reference to udapl is ominous.]

 > The mpirun command line contains the argument '--mca btl ^openib', 
which
 > I thought told mpi to not look for the ib interface.

As you said, the mca parameter above tells OMPI not to use openib,
although it may not be the only cause of the problem.
If you want to use openib switch to
--mca btl openib,sm,self

Another thing to check is whether there is a mixup of enviroment 
variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI 
version you may have installed.

My two cents,
Gus Correa

On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:
> We have recently upgraded our cluster to a version of Linux which comes
> with openMPI version 1.6.2.
>
> An application which ran previously (using some version of 1.4) now
> errors out with the following messages:
>
>  librdmacm: Fatal: no RDMA devices found
>  librdmacm: Fatal: no RDMA devices found
>  librdmacm: Fatal: no RDMA devices found
>
> 
--
>  WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
>  This may be a real error or it may be an invalid entry in the
> uDAPL
>  Registry which is contained in the dat.conf file. Contact your
> local
>  System Administrator to confirm the availability of the
> interfaces in
>  the dat.conf file.
>
> 
--
>  [tupile:25363] 2 more processes have sent help message
> help-mpi-btl-udapl.txt / dat_ia_open fail
>  [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to
> 0 to see all help / error messages
>
> The mpirun command line contains the argument '--mca btl ^openib', which
> I thought told mpi to not look for the ib interface.
>
> Can anyone suggest what the problem might be?  Did the relevant syntax
> change between versions 1.4 and 1.6?
>
>
> Jeffrey A. Cummings
> Engineering Specialist
> Performance Modeling and Analysis Department
> Systems Analysis and Simulation Subdivision
> Systems Engineering Division
> Engineering and Technology Group
> The Aerospace Corporation
> 571-307-4220
> jeffrey.a.cummi...@aero.org
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/06/24721.php
>

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/06/24722.php

Re: [OMPI users] Problem moving from 1.4 to 1.6

[Gus Correa]

It may be easier to install the latest OMPI from the tarball,
rather than trying to sort out the error.

http://www.open-mpi.org/software/ompi/v1.8/

The packaged built of (somewhat old) OMPI 1.6.2 that came with
Linux may not have built against the same IB libraries, hardware,
and configuration you have.
[The error message reference to udapl is ominous.]

> The mpirun command line contains the argument '--mca btl ^openib', which
> I thought told mpi to not look for the ib interface.

As you said, the mca parameter above tells OMPI not to use openib,
although it may not be the only cause of the problem.
If you want to use openib switch to
--mca btl openib,sm,self

Another thing to check is whether there is a mixup of enviroment 
variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI 
version you may have installed.


My two cents,
Gus Correa

On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:

We have recently upgraded our cluster to a version of Linux which comes
with openMPI version 1.6.2.

An application which ran previously (using some version of 1.4) now
errors out with the following messages:

 librdmacm: Fatal: no RDMA devices found
 librdmacm: Fatal: no RDMA devices found
 librdmacm: Fatal: no RDMA devices found

--
 WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
 This may be a real error or it may be an invalid entry in the
uDAPL
 Registry which is contained in the dat.conf file. Contact your
local
 System Administrator to confirm the availability of the
interfaces in
 the dat.conf file.

--
 [tupile:25363] 2 more processes have sent help message
help-mpi-btl-udapl.txt / dat_ia_open fail
 [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to
0 to see all help / error messages

The mpirun command line contains the argument '--mca btl ^openib', which
I thought told mpi to not look for the ib interface.

Can anyone suggest what the problem might be?  Did the relevant syntax
change between versions 1.4 and 1.6?


Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/06/24721.php

[OMPI users] Problem moving from 1.4 to 1.6

[Jeffrey A Cummings]

We have recently upgraded our cluster to a version of Linux which comes 
with openMPI version 1.6.2.

An application which ran previously (using some version of 1.4) now errors 
out with the following messages:

librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found

--
WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
This may be a real error or it may be an invalid entry in the 
uDAPL
Registry which is contained in the dat.conf file. Contact your 
local
System Administrator to confirm the availability of the interfaces 
in
the dat.conf file.

--
[tupile:25363] 2 more processes have sent help message 
help-mpi-btl-udapl.txt / dat_ia_open fail
[tupile:25363] Set MCA parameter "orte_base_help_aggregate" to 0 
to see all help / error messages

The mpirun command line contains the argument '--mca btl ^openib', which I 
thought told mpi to not look for the ib interface.

Can anyone suggest what the problem might be?  Did the relevant syntax 
change between versions 1.4 and 1.6?


Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org

Re: [OMPI users] importing to MPI data already in memory from another process

[Dave Goodell (dgoodell)]

On Jun 27, 2014, at 8:53 AM, Brock Palen  wrote:

> Is there a way to import/map memory from a process (data acquisition) such 
> that an MPI program could 'take' or see that memory?
> 
> We have a need to do data acquisition at the rate of .7TB/s and need todo 
> some shuffles/computation on these data,  some of the nodes are directly 
> connected to the device, and some will do processing. 
> 
> Here is the proposed flow:
> 
> * Data collector nodes runs process collecting data from device
> * Those nodes somehow pass the data to an MPI job running on these nodes and 
> a number of other nodes (cpu need for filterting is greater than what the 16 
> data nodes can provide).

For a non-MPI solution for intranode data transfer in this case, take a look at 
vmsplice(2):

http://man7.org/linux/man-pages/man2/vmsplice.2.html

Pay particular attention to the SPLICE_F_GIFT flag, which will allow you to 
simply give memory pages away to the MPI process, avoiding unnecessary data 
copies.  You would just need a pipe shared between the data collector process 
and the MPI process (and to be a bit careful with your memory 
allocation/management, since any page you gift away should probably come from 
mmap(2) directly).


Otherwise, as George mentioned, I would investigate converting your current 
data collector processes to also be MPI processes so that they can simply 
communicate the data to the rest of the cluster.

-Dave

Re: [OMPI users] importing to MPI data already in memory from another process

[George Bosilca]

MPI "universe" yes, but not necessarily MPI "world". You could have
the two worlds connect/accept or join
(https://www.open-mpi.org/doc/v1.8/man3/MPI_Comm_join.3.php) and then
you should be able to take advantage of the RMA. At least, that is
what is written in the book ...

  George.

On Fri, Jun 27, 2014 at 10:30 AM, Brock Palen  wrote:
> But this is within the same MPI "universe" right?
>
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> XSEDE Campus Champion
> bro...@umich.edu
> (734)936-1985
>
>
>
> On Jun 27, 2014, at 10:19 AM, George Bosilca  wrote:
>
>> The One-Sided Communications from the Chapter 11 of the MPI standard?
>> For processes on the same node you might want to look at
>> MPI_WIN_ALLOCATE_SHARED.
>>
>>  George.
>>
>> On Fri, Jun 27, 2014 at 9:53 AM, Brock Palen  wrote:
>>> Is there a way to import/map memory from a process (data acquisition) such 
>>> that an MPI program could 'take' or see that memory?
>>>
>>> We have a need to do data acquisition at the rate of .7TB/s and need todo 
>>> some shuffles/computation on these data,  some of the nodes are directly 
>>> connected to the device, and some will do processing.
>>>
>>> Here is the proposed flow:
>>>
>>> * Data collector nodes runs process collecting data from device
>>> * Those nodes somehow pass the data to an MPI job running on these nodes 
>>> and a number of other nodes (cpu need for filterting is greater than what 
>>> the 16 data nodes can provide).
>>>
>>> One thought is to have the data collector processes be threads inside the 
>>> MPI job running across all nodes, but was curious is there is a way to pass 
>>> data still in memory (to much to hit disk) to the running MPI filter job.
>>>
>>> Thanks!
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> CAEN Advanced Computing
>>> XSEDE Campus Champion
>>> bro...@umich.edu
>>> (734)936-1985
>>>
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2014/06/24716.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/06/24717.php
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24718.php

Re: [OMPI users] importing to MPI data already in memory from another process

[George Bosilca]

The One-Sided Communications from the Chapter 11 of the MPI standard?
For processes on the same node you might want to look at
MPI_WIN_ALLOCATE_SHARED.

  George.

On Fri, Jun 27, 2014 at 9:53 AM, Brock Palen  wrote:
> Is there a way to import/map memory from a process (data acquisition) such 
> that an MPI program could 'take' or see that memory?
>
> We have a need to do data acquisition at the rate of .7TB/s and need todo 
> some shuffles/computation on these data,  some of the nodes are directly 
> connected to the device, and some will do processing.
>
> Here is the proposed flow:
>
> * Data collector nodes runs process collecting data from device
> * Those nodes somehow pass the data to an MPI job running on these nodes and 
> a number of other nodes (cpu need for filterting is greater than what the 16 
> data nodes can provide).
>
> One thought is to have the data collector processes be threads inside the MPI 
> job running across all nodes, but was curious is there is a way to pass data 
> still in memory (to much to hit disk) to the running MPI filter job.
>
> Thanks!
>
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> XSEDE Campus Champion
> bro...@umich.edu
> (734)936-1985
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24716.php

[OMPI users] importing to MPI data already in memory from another process

[Brock Palen]

Is there a way to import/map memory from a process (data acquisition) such that 
an MPI program could 'take' or see that memory?

We have a need to do data acquisition at the rate of .7TB/s and need todo some 
shuffles/computation on these data,  some of the nodes are directly connected 
to the device, and some will do processing. 

Here is the proposed flow:

* Data collector nodes runs process collecting data from device
* Those nodes somehow pass the data to an MPI job running on these nodes and a 
number of other nodes (cpu need for filterting is greater than what the 16 data 
nodes can provide).

One thought is to have the data collector processes be threads inside the MPI 
job running across all nodes, but was curious is there is a way to pass data 
still in memory (to much to hit disk) to the running MPI filter job.

Thanks! 

Brock Palen
www.umich.edu/~brockp
CAEN Advanced Computing
XSEDE Campus Champion
bro...@umich.edu
(734)936-1985





signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [OMPI users] OpenMPI 1.8.1 runs more OpenMP Threads on the same core

[Luigi Santangelo]

:) Thanks to both
I'll try you solution and I'll give you a feedback

Thanks


2014-06-27 15:01 GMT+02:00 :

>
>
> Hi Luigi,
>
> Please try:
>
> --map-by slot:pe=4
>
> Probably Ralph is very busy, so something sliped his memory...
>
> Regards,
> Tetsuya
>
> > Hi all,
> > My system is a 64 core, with Debian 3.2.57 64 bit, GNU gcc 4.7, kernel
> Linux 3.2.0 and OpenMPI 1.8.1.
> > I developed an application to matching proteins files using OpenMP
> +OpenMPI. I compiled souce code with -fopenmp flag, I set OMP_NUM_THREADS=4
> then I ran binary with mpiexec -np 16.
> > When the program runs, the top command show me that only 16 core are
> used. If I use the previous of OpenMPI (1.6.5) it works fine. Must I set
> anything in the new version? Where I wrong?
> >
> > Thanks in advance
> >
> > Luigi___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/usersLink to
> this post: http://www.open-mpi.org/community/lists/users/2014/06/24712.php
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/06/24714.php
>

Re: [OMPI users] OpenMPI 1.8.1 runs more OpenMP Threads on the same core

[tmishima]

Hi Luigi,

Please try:

--map-by slot:pe=4

Probably Ralph is very busy, so something sliped his memory...

Regards,
Tetsuya

> Hi all,
> My system is a 64 core, with Debian 3.2.57 64 bit, GNU gcc 4.7, kernel
Linux 3.2.0 and OpenMPI 1.8.1.
> I developed an application to matching proteins files using OpenMP
+OpenMPI. I compiled souce code with -fopenmp flag, I set OMP_NUM_THREADS=4
then I ran binary with mpiexec -np 16.
> When the program runs, the top command show me that only 16 core are
used. If I use the previous of OpenMPI (1.6.5) it works fine. Must I set
anything in the new version? Where I wrong?
>
> Thanks in advance
>
> Luigi___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/usersLink to
this post: http://www.open-mpi.org/community/lists/users/2014/06/24712.php

Re: [OMPI users] OpenMPI 1.8.1 runs more OpenMP Threads on the same core

[Ralph Castain]

You should add this to your cmd line:

--map-by core:pe=4

This will bind each process to 4 cores

Sent from my iPhone

> On Jun 27, 2014, at 5:22 AM, Luigi Santangelo  
> wrote:
> 
> Hi all,
> My system is a 64 core, with Debian 3.2.57 64 bit, GNU gcc 4.7, kernel Linux 
> 3.2.0 and OpenMPI 1.8.1.
> I developed an application to matching proteins files using OpenMP+OpenMPI. I 
> compiled souce code with -fopenmp flag, I set OMP_NUM_THREADS=4 then I ran 
> binary with mpiexec -np 16.
> When the program runs, the top command show me that only 16 core are used. If 
> I use the previous of OpenMPI (1.6.5) it works fine. Must I set anything in 
> the new version? Where I wrong?
> 
> Thanks in advance
> 
> Luigi
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/06/24712.php

[OMPI users] OpenMPI 1.8.1 runs more OpenMP Threads on the same core

[Luigi Santangelo]

Hi all,
My system is a 64 core, with Debian 3.2.57 64 bit, GNU gcc 4.7, kernel
Linux 3.2.0 and OpenMPI 1.8.1.
I developed an application to matching proteins files using OpenMP+OpenMPI.
I compiled souce code with -fopenmp flag, I set OMP_NUM_THREADS=4 then I
ran binary with mpiexec -np 16.
When the program runs, the top command show me that only 16 core are used.
If I use the previous of OpenMPI (1.6.5) it works fine. Must I set anything
in the new version? Where I wrong?

Thanks in advance

Luigi