Re: [OMPI users] Ubuntu and LD_LIBRARY_PATH

[Jeff Squyres (jsquyres)]

In addition to what Ralph said, check out these 3 FAQ items:

https://www.open-mpi.org/faq/?category=running#run-prereqs
https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path
https://www.open-mpi.org/faq/?category=running#mpirun-prefix



> On Apr 25, 2016, at 2:02 PM, Ralph Castain  wrote:
> 
> Have you tried configuring OMPI with -enable-orterun-prefix-by-default?
> 
>> On Apr 25, 2016, at 10:34 AM, Rob Malpass  wrote:
>> 
>> Hi 
>>  
>> Sorry if this isn’t 100% relevant to this list but I’m at my wits end…
>>  
>> After a lot of hacking, I’ve finally configured openmpi on my Ubuntu 
>> cluster.   I had been having awful problems with not being able to find the 
>> libraries on the remote nodes but apparently the workaround is to use 
>> ld.conf.so.d/*.conf
>>  
>> I have compiled and run an example and it has worked but as it is not 
>> recommended practice, can I please ask if there is a better / more elegant 
>> workaround?   It is still the case that LD_LIBRARY_PATH is not set (because 
>> no dot files like .profile, .bashrc etc) get executed when a non-interactive 
>> ssh session takes place).   Is this going to cause me problems later?
>>  
>> Thanks
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2016/04/29030.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/04/29031.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Ubuntu and LD_LIBRARY_PATH

[Ralph Castain]

Have you tried configuring OMPI with -enable-orterun-prefix-by-default?

> On Apr 25, 2016, at 10:34 AM, Rob Malpass  wrote:
> 
> Hi 
>  
> Sorry if this isn’t 100% relevant to this list but I’m at my wits end…
>  
> After a lot of hacking, I’ve finally configured openmpi on my Ubuntu cluster. 
>   I had been having awful problems with not being able to find the libraries 
> on the remote nodes but apparently the workaround is to use 
> ld.conf.so.d/*.conf
>  
> I have compiled and run an example and it has worked but as it is not 
> recommended practice, can I please ask if there is a better / more elegant 
> workaround?   It is still the case that LD_LIBRARY_PATH is not set (because 
> no dot files like .profile, .bashrc etc) get executed when a non-interactive 
> ssh session takes place).   Is this going to cause me problems later?
>  
> Thanks
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
> 
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/04/29030.php 
> 

[OMPI users] Ubuntu and LD_LIBRARY_PATH

[Rob Malpass]

Hi 



Sorry if this isn't 100% relevant to this list but I'm at my wits end.



After a lot of hacking, I've finally configured openmpi on my Ubuntu
cluster.   I had been having awful problems with not being able to find the
libraries on the remote nodes but apparently the workaround is to use
ld.conf.so.d/*.conf



I have compiled and run an example and it has worked but as it is not
recommended practice, can I please ask if there is a better / more elegant
workaround?   It is still the case that LD_LIBRARY_PATH is not set (because
no dot files like .profile, .bashrc etc) get executed when a non-interactive
ssh session takes place).   Is this going to cause me problems later?



Thanks

Re: [OMPI users] MPIRUN SEGMENTATION FAULT

[Gus Correa]

Hi Elio

As Gilles said, if you change the integer size to -i8 in the 
application, and MPI was built with the default-sized integers

(4 bytes), things will get really ugly and mismatched.
Better avoid flags such as -i8, -r8, etc, when compiling MPI programs.

Have you tried to compile the code with the -traceback flag?
This at least should tell you where the code is failing (source file
and line).
As Ralph said, most likely the program is trying to
go beyond array boundaries,
or accessing non-allocated memory.
That can happen even with innocent strings
(say a file name or an informative message that are too big).

A better approach, as suggested by Ralph,
is to open the core file with gdb.
It should be named something like "core.98765", where the "98765" is the 
process number.
However, many Linux distributions set the core file size to zero by 
default, which prevents the core file to be created when the program 
crashes, but on the upside also prevents disk to fill up with big core 
files that are forgotten and hang around forever.

[ulimit -a will tell.]

I hope this helps,
Gus Correa


On 04/23/2016 07:06 PM, Gilles Gouaillardet wrote:

If you build your application with intel compilers and -i8, then openmpi
must also be built with intel compilers and -i8.

Cheers,

Gilles

On Sunday, April 24, 2016, Elio Physics mailto:elio-phys...@live.com>> wrote:

Well, I changed the compiler from mpif90 to mpiifort with
corresponding flags -i8 -g and recompiled. i am not getting the
segmentation fault problem anymore and the program runs but later
stops with no errors (except that the Fermi energy was not found!)
and with some strange empty files that are produced something like:
   fortDgcQe3  fortechvF2  fortMaN6a1  fortnxoYy1  fortvR5F8q.  i
still feel something is wrong.. Does anybody know what are these files?


Regards




*From:* users > on
behalf of Ralph Castain >
*Sent:* Saturday, April 23, 2016 1:38 PM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] MPIRUN SEGMENTATION FAULT
I don’t see any way this could be compilation related - I suspect
there is simply some error in the program (e.g., forgetting to
initialize some memory region).



On Apr 23, 2016, at 8:03 AM, Elio Physics > wrote:

Hello Andy,

the program is not mine. I have got it from a group upon request.
It might be program related because I run other codes such as
quantum espresso and work perfectly fine although it is the
cluster people who compiled it. Since I have compiled the program
I am having problems with, I am thinking that it might be
"compilation" related. This is why i wanted some experts' opinion
on this




*From:*users > on
behalf of Andy Riebs >
*Sent:*Saturday, April 23, 2016 12:49 PM
*To:*us...@open-mpi.org

*Subject:*Re: [OMPI users] MPIRUN SEGMENTATION FAULT
The challenge for the MPI experts here (of which I am NOT one!) is
that the problem appears to be in your program; MPI is simply
reporting that your program failed. If you got the program from
someone else, you will need to solicit their help. If you wrote
it, well, it is never a bad time to learn to use gdb!

Best regards
Andy

On 04/23/2016 10:41 AM, Elio Physics wrote:

I am not really an expert with gdb. What is the core file? and
how to use gdb? I have got three files as an output when the
executable is used. One is the actual output which stops and the
other two are error files (from which I knew about the
segmentation fault).


thanks



*From:*users
on
behalf of Ralph Castain

*Sent:*Saturday, April 23, 2016 11:39 AM
*To:*Open MPI Users
*Subject:*Re: [OMPI users] MPIRUN SEGMENTATION FAULT
valgrind isn’t going to help here - there are multiple reasons
why your application could be segfaulting. Take a look at the
core file with gdb and find out where it is failing.


On Apr 22, 2016, at 10:20 PM, Elio Physics
> wrote:

One more thing i forgot to mention in my previous e-mail. In the
output file I get the following message:


2 total processes killed (some possibly by mpirun during cleanup)

Thanks




*From:*users > on
behalf of Elio Physics


*Sent:*Saturday, April 23, 2016 3:07 AM
*To:*Open MPI Users
*Subject:*Re: [OMPI users] MPIRUN SEGMENTATION FAULT
I have used valgrind and this is what i got:

valgrind mpirun ~/Elie/SPRKKR/bin/kkrscf6.3MPI Fe_SCF.inp >
scf-51551.jlborges.fisica.ufmg.br.out
==8135== Memcheck, a memory error detector
==8135== Copy

Re: [OMPI users] MPI_Bcast implementations in OpenMPI

[George Bosilca]

> On Apr 25, 2016, at 11:33 , Dave Love  wrote:
> 
> George Bosilca  writes:
> 
>> Dave,
>> 
>> You are absolutely right, the parameters are now 6-7 years old,
>> gathered on interconnects long gone. Moreover, several discussions in
>> this mailing list indicated that they do not match current network
>> capabilities.
>> 
>> I have recently reshuffled the tuned module to move all the algorithms
>> in the base and therefore make them available to other collective
>> modules (the code is available in master and 1.10 and the future
>> 2.0). This move has the potential for allowing different decision
>> schemes to coexists, and be dynamically selected at runtime based on
>> network properties, network topology, or even applications needs. I
>> continue to have hopes that network vendors will eventually get
>> interested in tailoring the collective selection to match their
>> network capabilities, and provide their users with a performance boost
>> by allowing for network specific algorithm selection.
> 
> That sounds useful, assuming the speed is generally dominated by the
> basic fabric.  What's involved in making the relevant measurements and
> plugging them in?  I did look at using OTPO(?) to check this sort of
> thing once.  I couldn't make it work in the time I had, but Periscope
> might be a good alternative now.

It is a multidimensional space optimization problem. The critical point is 
identifying the switching points between different algorithms based on their 
performance (taking in account, at least, physical topology, number of 
processes and amount of data). The paper I sent on one of my previous email 
discusses how we did the decision functions on the current implementation. 
There are certainly better ways, but the one we took at least did not involve 
any extra software, and was done using simple scripts.

> If it's fairly mechanical -- maybe even if not -- it seems like
> something that should just be done regardless of vendors.  I'm sure
> plenty of people could measure QDR fat tree, for a start (at least where
> measurement isn’t frowned upon).

Based on feedback from the user mailing list, several users did such 
optimizations for their specific applications. This makes the optimization 
problem much simpler, as some of the parameters have discrete values (message 
size). If we assume a symmetric network, and have a small number of message 
sizes of interest, it is enough to run few benchmarks (skampi, to the IMB test 
on the collective of interest), and manually finding the switch point is a 
relatively simple process.

  George.



> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/04/29024.php

Re: [OMPI users] Cannot run a simple MPI program

[Jeff Squyres (jsquyres)]

On Apr 24, 2016, at 8:12 PM, Gilles Gouaillardet  wrote:
> 
>   fwiw, once in a while, i
>   rm -rf /.../ompi_install_dir/lib/openmpi
>   to get rid of the removed modules

If it helps, I usually install Open MPI into a tree all by itself, and then I 
can "rm -rf $ompi_install_dir" in its entirety to avoid any complications.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] track progress of a mpi gather

[George Bosilca]

We just switched the default compile-mode to MPI_THREAD_MULTIPLE (with
prior testing). Thus, we are relatively confident it should work on all
BTLs. If not we would be happy to hear about.

  George.


On Sun, Apr 24, 2016 at 11:02 AM, dpchoudh .  wrote:

> Hello Gilles
>
> That idea crossed my mind as well, but I was under the impression that
> MPI_THREAD_MULTIPLE is not very well supported on OpenMPI? I believe it is
> not supported on OpenIB, but the original poster seems to be using TCP.
> Does it work for TCP?
>
> Thanks
> Durga
>
> 1% of the executables have 99% of CPU privilege!
> Userspace code! Unite!! Occupy the kernel!!!
>
> On Sun, Apr 24, 2016 at 10:48 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> an other option is to use MPI_THREAD_MULTIPLE, and MPI_Recv() on the
>> master task in a dedicated thread, and use a unique tag (or MPI_Comm_dup()
>> MPI_COMM_WORLD) to separate the traffic.
>>
>> If this is not the desired design, then the master task has to post
>> MPI_Irecv() and "poll" with MPI_Probe() / MPI_Test() and friends.
>> Note it is possible to use non blocking collective (MPI_Ibcast(),
>> MPI_Iscatter() and MPI_Igather()) and "both" collective and the progress
>> statuses
>>
>> Cheers,
>>
>> Gilles
>>
>> On Sunday, April 24, 2016, dpchoudh .  wrote:
>>
>>> Hello
>>>
>>>
>>> I am not sure I am understanding your requirements correctly, but base
>>> on what I think it is, how about this: you do an MPI_Send() from all the
>>> non-root nodes to the root node and pack all the progress related data into
>>> this send. Use a special tag for this message to make it stand out from
>>> 'regular' sends. The root node does a non-blocking receive on this tag from
>>> all the nodes that it is expecting this message from.
>>>
>>> Would that work?
>>>
>>> Durga
>>>
>>>
>>> 1% of the executables have 99% of CPU privilege!
>>> Userspace code! Unite!! Occupy the kernel!!!
>>>
>>> On Sun, Apr 24, 2016 at 7:05 AM, MM  wrote:
>>>
 Hello,

 With a miniature case of 3 linux quadcore boxes, linked via 1Gbit
 Ethernet, I have a UI that runs on 1 of the 3 boxes, and that is the root
 of the communicator.
 I have a 1-second-running function on up to 10 parameters, my parameter
 space fits in the memory of the root, the space of it is N ~~ 1 million.

 I use broadcast/scatter/gather to collect the value of my function on
 each of the 1million points, dividing 1million by the number of nodes (11:
 rootnode has 1 core/thread assigned to the UI, 1 core/thread for its
 evaluation of the function on its own part of the parameter space and 2
 other cores run non-root nodes, and the 2 other boxes all run non-root
 nodes)

 the rootnode does:
 1. broadcast needed data
 2. scatter param space
 3. evaluate function locally
 4. gather results from this and all other nodes

 How would I go about having the non-root nodes send a sort of progress
 status to the root node? that's used for plotting results on the root box
 as soon as they are available?

 Rds,





 ___
 users mailing list
 us...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
 Link to this post:
 http://www.open-mpi.org/community/lists/users/2016/04/29013.php

>>>
>>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/04/29015.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/04/29016.php
>

Re: [OMPI users] Porting MPI-3 C-program to Fortran

[Dave Love]

Tom Rosmond  writes:

> Thanks for replying, but the difference between what can be done in C
> vs fortran is still my problem.  I apologize for my rudimentary
> understanding of C, but here is a brief summary:

I'm not an expert on this stuff, just cautioning about Fortran semantics
where I could imagine it's important.

I guess you'll need to read what the standard says about the routine,
and use a compiler that supports iso_c_binding, which is the right way
to interface to C.  (The OMPI man pages don't document the modern
Fortran interface, unfortunately.)

If it's not clear how to use it, perhaps one of the Committee can clarify.

Re: [OMPI users] MPI_Bcast implementations in OpenMPI

[Dave Love]

George Bosilca  writes:

> Dave,
>
> You are absolutely right, the parameters are now 6-7 years old,
> gathered on interconnects long gone. Moreover, several discussions in
> this mailing list indicated that they do not match current network
> capabilities.
>
> I have recently reshuffled the tuned module to move all the algorithms
> in the base and therefore make them available to other collective
> modules (the code is available in master and 1.10 and the future
> 2.0). This move has the potential for allowing different decision
> schemes to coexists, and be dynamically selected at runtime based on
> network properties, network topology, or even applications needs. I
> continue to have hopes that network vendors will eventually get
> interested in tailoring the collective selection to match their
> network capabilities, and provide their users with a performance boost
> by allowing for network specific algorithm selection.

That sounds useful, assuming the speed is generally dominated by the
basic fabric.  What's involved in making the relevant measurements and
plugging them in?  I did look at using OTPO(?) to check this sort of
thing once.  I couldn't make it work in the time I had, but Periscope
might be a good alternative now.

If it's fairly mechanical -- maybe even if not -- it seems like
something that should just be done regardless of vendors.  I'm sure
plenty of people could measure QDR fat tree, for a start (at least where
measurement isn't frowned upon).