Re: [OMPI users] know which CPU has the maximum value

2018-08-10 Thread Gus Correa

Hi Jeff S.

OK, then I misunderstood Jeff H.
Sorry about that, Jeff H..

Nevertheless, Diego Avesani has certainly a point.
And it is the point of view of an user,
something that hopefully matters.
I'd add to Diego's arguments
that maxloc, minloc, and friends are part of
Fortran, Matlab, etc.
A science/engineering programmer expects it to be available,
not to have to be reinvented from scratch,
both on the baseline language, as well as in MPI.

In addition,
MPI developers cannot expect the typical MPI user to keep track
of what goes on in the MPI Forum.
I certainly don't have either the skill or the time for it.
However, developers can make an effort to listen to the chatter in the
various MPI user's list, before making any decision of stripping off 
functionality, specially such a basic one as minloc, maxloc.


My two cents from a pedestrian MPI user,
who thinks minloc and maxloc are great,
knows nothing about the MPI Forum protocols and activities,
but hopes the Forum pays attention to users' needs.

Gus Correa

PS - Jeff S.: Please, bring Diego's request to the Forum! Add my vote 
too.  :)



On 08/10/2018 02:19 PM, Jeff Squyres (jsquyres) via users wrote:

Jeff H. was referring to Nathan's offhand remark about his desire to kill the 
MPI_MINLOC / MPI_MAXLOC operations.  I think Jeff H's point is that this is 
just Nathan's opinion -- as far as I know, there is no proposal in front of the 
MPI Forum to actively deprecate MPI_MINLOC or MPI_MAXLOC.  Speaking this 
opinion on a public mailing list with no other context created a bit of 
confusion.

The Forum is quite transparent in what it does -- e.g., anyone is allowed to 
come to its meetings and hear (and participate in!) all the deliberations, etc. 
 But speaking off-the-cuff about something that *might* happen *someday* that 
would have impact on real users and real codes -- that might have caused a 
little needless confusion.




On Aug 10, 2018, at 2:11 PM, Gus Correa  wrote:

Hmmm ... no, no, no!
Keep it secret why!?!?

Diego Avesani's questions and questioning
may have saved us users from getting a
useful feature deprecated in the name of code elegance.
Code elegance may be very cherished by developers,
but it is not necessarily helpful to users,
specially if it strips off useful functionality.

My cheap 2 cents from a user.
Gus Correa


On 08/10/2018 01:52 PM, Jeff Hammond wrote:

This thread is a perfect illustration of why MPI Forum participants should not 
flippantly discuss feature deprecation in discussion with users.  Users who are 
not familiar with the MPI Forum process are not able to evaluate whether such 
proposals are serious or have any hope of succeeding and therefore may be 
unnecessarily worried about their code breaking in the future, when that future 
is 5 to infinity years away.
If someone wants to deprecate MPI_{MIN,MAX}LOC, they should start that 
discussion on https://github.com/mpi-forum/mpi-issues/issues or 
https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll.
Jeff
On Fri, Aug 10, 2018 at 10:27 AM, Jeff Squyres (jsquyres) via users 
mailto:users@lists.open-mpi.org>> wrote:
It is unlikely that MPI_MINLOC and MPI_MAXLOC will go away any time
soon.
As far as I know, Nathan hasn't advanced a proposal to kill them in
MPI-4, meaning that they'll likely continue to be in MPI for at
least another 10 years.  :-)
(And even if they did get killed in MPI-4, implementations like Open
MPI would continue to keep them in our implementations for quite a
while -- i.e., years)
 > On Aug 10, 2018, at 1:13 PM, Diego Avesani
mailto:diego.aves...@gmail.com>> wrote:
 >
 > I agree about the names, it is very similar to MIN_LOC and
MAX_LOC in fortran 90.
 > However, I find difficult to define some algorithm able to do the
same things.
 >
 >
 >
 > Diego
 >
 >
 > On 10 August 2018 at 19:03, Nathan Hjelm via users
mailto:users@lists.open-mpi.org>> wrote:
 > They do not fit with the rest of the predefined operations (which
operate on a single basic type) and can easily be implemented as
user defined operations and get the same performance. Add to that
the fixed number of tuple types and the fact that some of them are
non-contiguous (MPI_SHORT_INT) plus the terrible names. If I could
kill them in MPI-4 I would.
 >
 > On Aug 10, 2018, at 9:47 AM, Diego Avesani
mailto:diego.aves...@gmail.com>> wrote:
 >
 >> Dear all,
 >> I have just implemented MAXLOC, why should they  go away?
 >> it seems working pretty well.
 >>
 >> thanks
 >>
 >> Diego
 >>
 >>
 >> On 10 August 2018 at 17:39, Nathan Hjelm via users
mailto:users@lists.open-mpi.org>> wrote:
 >> The problem is minloc and maxloc need to go away. better to use
a custom op.
 

Re: [OMPI users] know which CPU has the maximum value

2018-08-10 Thread Gus Correa

On 08/10/2018 01:27 PM, Jeff Squyres (jsquyres) via users wrote:

It is unlikely that MPI_MINLOC and MPI_MAXLOC will go away any time soon.

As far as I know, Nathan hasn't advanced a proposal to kill them in MPI-4, 
meaning that they'll likely continue to be in MPI for at least another 10 
years.  :-)

(And even if they did get killed in MPI-4, implementations like Open MPI would 
continue to keep them in our implementations for quite a while -- i.e., years)



Yeah!
Two thumbs up!




On Aug 10, 2018, at 1:13 PM, Diego Avesani  wrote:

I agree about the names, it is very similar to MIN_LOC and MAX_LOC in fortran 
90.
However, I find difficult to define some algorithm able to do the same things.



Diego


On 10 August 2018 at 19:03, Nathan Hjelm via users  
wrote:
They do not fit with the rest of the predefined operations (which operate on a 
single basic type) and can easily be implemented as user defined operations and 
get the same performance. Add to that the fixed number of tuple types and the 
fact that some of them are non-contiguous (MPI_SHORT_INT) plus the terrible 
names. If I could kill them in MPI-4 I would.

On Aug 10, 2018, at 9:47 AM, Diego Avesani  wrote:


Dear all,
I have just implemented MAXLOC, why should they  go away?
it seems working pretty well.

thanks

Diego


On 10 August 2018 at 17:39, Nathan Hjelm via users  
wrote:
The problem is minloc and maxloc need to go away. better to use a custom op.

On Aug 10, 2018, at 9:36 AM, George Bosilca  wrote:


You will need to create a special variable that holds 2 entries, one for the 
max operation (with whatever type you need) and an int for the rank of the 
process. The MAXLOC is described on the OMPI man page [1] and you can find an 
example on how to use it on the MPI Forum [2].

George.


[1] https://www.open-mpi.org/doc/v2.0/man3/MPI_Reduce.3.php
[2] https://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node79.html

On Fri, Aug 10, 2018 at 11:25 AM Diego Avesani  wrote:
  Dear all,
I have probably understood.
The trick is to use a real vector and to memorize also the rank.

Have I understood correctly?
thanks

Diego


On 10 August 2018 at 17:19, Diego Avesani  wrote:
Deal all,
I do not understand how MPI_MINLOC works. it seem locate the maximum in a 
vector and not the CPU to which the valur belongs to.

@ray: and if two has the same value?

thanks


Diego


On 10 August 2018 at 17:03, Ray Sheppard  wrote:
As a dumb scientist, I would just bcast the value I get back to the group and 
ask whoever owns it to kindly reply back with its rank.
  Ray


On 8/10/2018 10:49 AM, Reuti wrote:
Hi,

Am 10.08.2018 um 16:39 schrieb Diego Avesani :

Dear all,

I have a problem:
In my parallel program each CPU compute a value, let's say eff.

First of all, I would like to know the maximum value. This for me is quite 
simple,
I apply the following:

CALL MPI_ALLREDUCE(eff, effmaxWorld, 1, MPI_DOUBLE_PRECISION, MPI_MAX, 
MPI_MASTER_COMM, MPIworld%iErr)
Would MPI_MAXLOC be sufficient?

-- Reuti


However, I would like also to know to which CPU that value belongs. Is it 
possible?

I have set-up a strange procedure but it works only when all the CPUs has 
different values but fails when two of then has the same eff value.

Is there any intrinsic MPI procedure?
in anternative,
do you have some idea?

really, really thanks.
Diego


Diego

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users





___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


Re: [OMPI users] know which CPU has the maximum value

2018-08-10 Thread Gus Correa

Hmmm ... no, no, no!
Keep it secret why!?!?

Diego Avesani's questions and questioning
may have saved us users from getting a
useful feature deprecated in the name of code elegance.
Code elegance may be very cherished by developers,
but it is not necessarily helpful to users,
specially if it strips off useful functionality.

My cheap 2 cents from a user.
Gus Correa


On 08/10/2018 01:52 PM, Jeff Hammond wrote:
This thread is a perfect illustration of why MPI Forum participants 
should not flippantly discuss feature deprecation in discussion with 
users.  Users who are not familiar with the MPI Forum process are not 
able to evaluate whether such proposals are serious or have any hope of 
succeeding and therefore may be unnecessarily worried about their code 
breaking in the future, when that future is 5 to infinity years away.


If someone wants to deprecate MPI_{MIN,MAX}LOC, they should start that 
discussion on https://github.com/mpi-forum/mpi-issues/issues or 
https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll.


Jeff

On Fri, Aug 10, 2018 at 10:27 AM, Jeff Squyres (jsquyres) via users 
mailto:users@lists.open-mpi.org>> wrote:


It is unlikely that MPI_MINLOC and MPI_MAXLOC will go away any time
soon.

As far as I know, Nathan hasn't advanced a proposal to kill them in
MPI-4, meaning that they'll likely continue to be in MPI for at
least another 10 years.  :-)

(And even if they did get killed in MPI-4, implementations like Open
MPI would continue to keep them in our implementations for quite a
while -- i.e., years)


 > On Aug 10, 2018, at 1:13 PM, Diego Avesani
mailto:diego.aves...@gmail.com>> wrote:
 >
 > I agree about the names, it is very similar to MIN_LOC and
MAX_LOC in fortran 90.
 > However, I find difficult to define some algorithm able to do the
same things.
 >
 >
 >
 > Diego
 >
 >
 > On 10 August 2018 at 19:03, Nathan Hjelm via users
mailto:users@lists.open-mpi.org>> wrote:
 > They do not fit with the rest of the predefined operations (which
operate on a single basic type) and can easily be implemented as
user defined operations and get the same performance. Add to that
the fixed number of tuple types and the fact that some of them are
non-contiguous (MPI_SHORT_INT) plus the terrible names. If I could
kill them in MPI-4 I would.
 >
 > On Aug 10, 2018, at 9:47 AM, Diego Avesani
mailto:diego.aves...@gmail.com>> wrote:
 >
 >> Dear all,
 >> I have just implemented MAXLOC, why should they  go away?
 >> it seems working pretty well.
 >>
 >> thanks
 >>
 >> Diego
 >>
 >>
 >> On 10 August 2018 at 17:39, Nathan Hjelm via users
mailto:users@lists.open-mpi.org>> wrote:
 >> The problem is minloc and maxloc need to go away. better to use
a custom op.
 >>
 >> On Aug 10, 2018, at 9:36 AM, George Bosilca mailto:bosi...@icl.utk.edu>> wrote:
 >>
 >>> You will need to create a special variable that holds 2
entries, one for the max operation (with whatever type you need) and
an int for the rank of the process. The MAXLOC is described on the
OMPI man page [1] and you can find an example on how to use it on
the MPI Forum [2].
 >>>
 >>> George.
 >>>
 >>>
 >>> [1] https://www.open-mpi.org/doc/v2.0/man3/MPI_Reduce.3.php
<https://www.open-mpi.org/doc/v2.0/man3/MPI_Reduce.3.php>
 >>> [2]
https://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node79.html
<https://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node79.html>
 >>>
 >>> On Fri, Aug 10, 2018 at 11:25 AM Diego Avesani
mailto:diego.aves...@gmail.com>> wrote:
 >>>  Dear all,
 >>> I have probably understood.
 >>> The trick is to use a real vector and to memorize also the rank.
 >>>
 >>> Have I understood correctly?
 >>> thanks
 >>>
 >>> Diego
 >>>
 >>>
 >>> On 10 August 2018 at 17:19, Diego Avesani
mailto:diego.aves...@gmail.com>> wrote:
 >>> Deal all,
 >>> I do not understand how MPI_MINLOC works. it seem locate the
maximum in a vector and not the CPU to which the valur belongs to.
 >>>
 >>> @ray: and if two has the same value?
 >>>
 >>> thanks
 >>>
 >>>
 >>> Diego
 >>>
 >>>
 >>> On 10 August 2018 at 17:03, Ray Sheppard mailto:rshep...@iu.edu>> wrote:
 >>> As a dumb scientist, I would just bcast the valu

Re: [OMPI users] OpenMPI 3.0.1 - mpirun hangs with 2 hosts

2018-05-14 Thread Gus Correa

Hi Max

Just in case, as environment mix often happens.
Could it be that you are inadvertently picking another
installation of OpenMPI, perhaps installed from packages
in /usr , or /usr/local?
That's easy to check with 'which mpiexec' or
'which mpicc', for instance.

Have you tried to prepend (as opposed to append) OpenMPI
to your PATH? Say:

export 
PATH='/home/user/openmpi_install/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'


I hope this helps,
Gus Correa

On 05/14/2018 12:40 PM, Max Mellette wrote:

John,

Thanks for the suggestions. In this case there is no cluster manager / 
job scheduler; these are just a couple of individual hosts in a rack. 
The reason for the generic names is that I anonymized the full network 
address in the previous posts, truncating to just the host name.


My home directory is network-mounted to both hosts. In fact, I 
uninstalled OpenMPI 3.0.1 from /usr/local on both hosts, and installed 
OpenMPI 3.1.0 into my home directory at `/home/user/openmpi_install`, 
also updating .bashrc appropriately:


user@b09-30:~$ cat .bashrc
export 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/user/openmpi_install/bin

export LD_LIBRARY_PATH=/home/user/openmpi_install/lib

So the environment should be the same on both hosts.

Thanks,
Max

On Mon, May 14, 2018 at 12:29 AM, John Hearns via users 
<users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>> wrote:


One very, very stupid question here. This arose over on the Slurm
list actually.
Those hostnames look like quite generic names, ie they are part of
an HPC cluster?
Do they happen to have independednt home directories for your userid?
Could that possibly make a difference to the MPI launcher?

On 14 May 2018 at 06:44, Max Mellette <wmell...@ucsd.edu
<mailto:wmell...@ucsd.edu>> wrote:

Hi Gilles,

Thanks for the suggestions; the results are below. Any ideas
where to go from here?

- Seems that selinux is not installed:

user@b09-30:~$ sestatus
The program 'sestatus' is currently not installed. You can
install it by typing:
sudo apt install policycoreutils

- Output from orted:

user@b09-30:~$ /usr/bin/ssh -x b09-32 orted
[b09-32:197698] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in
file ess_env_module.c at line 147
[b09-32:197698] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad
parameter in file util/session_dir.c at line 106
[b09-32:197698] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad
parameter in file util/session_dir.c at line 345
[b09-32:197698] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad
parameter in file base/ess_base_std_orted.c at line 270

--
It looks like orte_init failed for some reason; your parallel
process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal
failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

   orte_session_dir failed
   --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS

--

- iptables rules:

user@b09-30:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ufw-before-logging-input  all  --  anywhere             anywhere
ufw-before-input  all  --  anywhere             anywhere
ufw-after-input  all  --  anywhere             anywhere
ufw-after-logging-input  all  --  anywhere             anywhere
ufw-reject-input  all  --  anywhere             anywhere
ufw-track-input  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ufw-before-logging-forward  all  --  anywhere             anywhere
ufw-before-forward  all  --  anywhere             anywhere
ufw-after-forward  all  --  anywhere             anywhere
ufw-after-logging-forward  all  --  anywhere             anywhere
ufw-reject-forward  all  --  anywhere             anywhere
ufw-track-forward  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ufw-before-logging-output  all  --  anywhere             anywhere
ufw-before-output  all  --  anywhere             anywhere
ufw-after-output  all  --  anywhere             anywhere
uf

Re: [OMPI users] OpenMPI with-tm is not obeying torque

2017-10-06 Thread Gus Correa

Hi Anthony, Ralph, Gilles, all

As far as I know, for core/processor assignment to user jobs to work,
Torque needs to be configured with cpuset support
(configure --enable-cpuset ...).
That is separate from what OpenMPI does in terms of process binding.
Otherwise, the user processes in the job
will be free to use any cores/processors on the nodes assigned to it.

Some additional work to setup Linux support for cpuset is also needed,
for Torque to use it at runtime (create a subdirectory in /dev/cpuset,
mount the cpuset file system there).
I do this in the pbs_mom daemon startup stcript,
but that can be done in other ways:

##
# create and mount /dev/cpuset
if [ ! -e /dev/cpuset ];then
mkdir /dev/cpuset
fi

if [ "`mount -t cpuset`x" == "x" ];then
   mount -t cpuset none /dev/cpuset
fi
##

I don't know if the epel Torque package is configured
with cpuset support, but I would guess it is not.
Look at /dev/cpuset in your compute nodes
to see if Torque created anything there.

I don't know either if OpenMPI can somehow bypass the cores/processors 
assigned by torque to a job, if any, or when Torque is configured
without cpuset support, to somehow still bind the MPI processes to 
cores/processors/sockets/etc.


I hope this helps,
Gus Correa

On 10/06/2017 02:22 AM, Anthony Thyssen wrote:
Sorry r...@open-mpi.org <mailto:r...@open-mpi.org>  as Gilles Gouaillardet 
pointed out to me the problem wasn't OpenMPI, but with the specific EPEL 
implementation (see Redhat Bugzilla 1321154)


Today, the the server was able to be taken down for maintance, and I 
wanted to try a few things.


After installing EPEL Testing Repo    torque-4.2.10-11.el7

However I found that all the nodes were 'down'  even though everything 
appears to be running, with no errors in the error logs.


After a lot of trials, errors and reseach, I eventually (on a whim) I 
decided to remove the "num_node_boards=1" entry from the 
"torque/server_priv/nodes" file and restart the server & scheduler.  
  Suddenly the nodes were "free" and my initial test job ran.


Perhaps the EPEL-Test Torque 4.2.10-11  does not contain Numa?

ALL later tests (with OpenMPI - RHEL SRPM 1.10.6-2 re-compiled 
"--with-tm")  is now responding to the Torque mode allocation correctly 
and is no longer simply running all the jobs on the first node.


That is    $PBS_NODEFILE  ,    pbsdsh hostname  and   mpirun hostname
are all in agreement.


Thank you all for your help, and putting up with with me.

   Anthony Thyssen ( System Programmer )    <a.thys...@griffith.edu.au 
<mailto:a.thys...@griffith.edu.au>>

  --
   "Around here we've got a name for people what talks to dragons."
   "Traitor?"  Wiz asked apprehensively.
   "No.  Lunch."                     -- Rick Cook, "Wizadry Consulted"
  --


On Wed, Oct 4, 2017 at 11:43 AM, r...@open-mpi.org 
<mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
wrote:


Can you try a newer version of OMPI, say the 3.0.0 release? Just
curious to know if we perhaps “fixed” something relevant.



On Oct 3, 2017, at 5:33 PM, Anthony Thyssen
<a.thys...@griffith.edu.au <mailto:a.thys...@griffith.edu.au>> wrote:

FYI...

The problem is discussed further in

Redhat Bugzilla: Bug 1321154 - numa enabled torque don't work
https://bugzilla.redhat.com/show_bug.cgi?id=1321154
<https://bugzilla.redhat.com/show_bug.cgi?id=1321154>

I'd seen this previous as it required me to add
"num_node_boards=1" to each node in the
/var/lib/torque/server_priv/nodes  to get torque to at least
work.  Specifically by munging
the $PBS_NODES" (which comes out correcT) into a host list
containing the correct
"slot=" counts.  But of course now that I have compiled OpenMPI
using "--with-tm" that
should not have been needed as in fact is now ignored by OpenMPI
in a Torque-PBS
environment.

However it seems ever since "NUMA" support was into the Torque
RPM's, has also caused
the current problems, and is still continuing.   The last action
is a new EPEL "test' version
(August 2017),  I will try shortly.

Take you for your help, though I am still open to suggestions for
a replacement.

  Anthony Thyssen ( System Programmer )   
<a.thys...@griffith.edu.au <mailto:a.thys...@griffith.edu.au>>

 --
   Encryption... is a powerful defensive weapon for free

Re: [OMPI users] Fwd: Make All error regarding either "Conflicting" or "Previous Declaration" among others

2017-09-21 Thread Gus Correa

On 09/21/2017 11:26 AM, Aragorn Inocencio wrote:
Hi, sorry about the mixup earlier.  But I have recently tried installing 
openmpi 3.0.0 using the instructions I found in the Reef3D manual 
(attached below), so


./configure CC=gcc CXX=g++ F77=gfortran FC=gfortran 
--prefix=/usr/local/openmpi


throws no errors, but again when I run the "make all" command next I am 
getting quite a few errors, to name a few:


 1. #warning "fd_set and associated macros have been defined in sys/types
 2. error: redefinition of struct hostent
 3. A bunch of other errors similar to 2
 4. A bunch of errors saying, "conflicting types for 'accept' bind,
connect etc.
 5. "Expected declaration specifiers or '...' before '("
 6. previous declaration of various functions

then all the way at the end it gives me an Error 1.

I'd like to apologize for my ignorance as I really have no background in 
this area; I don't even know how to print all the errors to a txt file.


** If using sh/bash:

./configure CC=gcc CXX=g++ F77=gfortran FC=gfortran 
--prefix=/usr/local/openmpi 2>&1 | tee my_configure.log


make 2>&1 | tee my_make.log

make install 2>&1 | tee my_make_install.log

** If using csh/tcsh:

./configure CC=gcc CXX=g++ F77=gfortran FC=gfortran 
--prefix=/usr/local/openmpi  |& tee my_configure.log


make |& tee my_make.log

make install |& tee my_make_install.log

I hope this helps,
Gus Correa


For what it's worth I have attached the config.log file.

​Inline image 1


___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] -host vs -hostfile

2017-07-31 Thread Gus Correa

Maybe something is wrong with the Torque installation?
Or perhaps with the Open MPI + Torque integration?

1) Make sure your Open MPI was configured and compiled with the
Torque "tm" library of your Torque installation.
In other words:

configure --with-tm=/path/to/your/Torque/tm_library ...

2) Check if your $TORQUE/server_priv/nodes file has all the nodes
in your cluster.  If not, edit the file and add the missing nodes.
Then restart the Torque server (service pbs_server restart).

3) Run "pbsnodes" to see if all nodes are listed.

4) Run "hostname" with mpirun in a short Torque script:

#PBS -l nodes=4:ppn=1
...
mpirun hostname

The output should show all four nodes.

Good luck!
Gus Correa

On 07/31/2017 02:41 PM, Mahmood Naderan wrote:
Well it is confusing!! As you can see, I added four nodes to the host 
file (the same nodes are used by PBS). The --map-by ppr:1:node works 
well. However, the PBS directive doesn't work


mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun 
-hostfile hosts --map-by ppr:1:node a.out


* hwloc 1.11.2 has encountered what looks like an error from the 
operating system.

*
* Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
0xff00) without inclusion!

* Error occurred in topology.c line 1048
*
* The following FAQ entry in the hwloc documentation may help:
*   What should I do when hwloc reports "operating system" warnings?
* Otherwise please report this error message to the hwloc user's mailing 
list,
* along with the output+tarball generated by the hwloc-gather-topology 
script.


Hello world from processor cluster.hpc.org <http://cluster.hpc.org>, 
rank 0 out of 4 processors

Hello world from processor compute-0-0.local, rank 1 out of 4 processors
Hello world from processor compute-0-1.local, rank 2 out of 4 processors
Hello world from processor compute-0-2.local, rank 3 out of 4 processors
mahmood@cluster:mpitest$ cat mmt.sh
#!/bin/bash
#PBS -V
#PBS -q default
#PBS -j oe
#PBS -l  nodes=4:ppn=1
#PBS -N job1
#PBS -o .
cd $PBS_O_WORKDIR
/share/apps/computer/openmpi-2.0.1/bin/mpirun a.out
mahmood@cluster:mpitest$ qsub mmt.sh
6428.cluster.hpc.org <http://6428.cluster.hpc.org>
mahmood@cluster:mpitest$ cat job1.o6428
Hello world from processor compute-0-1.local, rank 0 out of 32 processors
Hello world from processor compute-0-1.local, rank 2 out of 32 processors
Hello world from processor compute-0-1.local, rank 3 out of 32 processors
Hello world from processor compute-0-1.local, rank 4 out of 32 processors
Hello world from processor compute-0-1.local, rank 5 out of 32 processors
Hello world from processor compute-0-1.local, rank 6 out of 32 processors
Hello world from processor compute-0-1.local, rank 8 out of 32 processors
Hello world from processor compute-0-1.local, rank 9 out of 32 processors
Hello world from processor compute-0-1.local, rank 12 out of 32 processors
Hello world from processor compute-0-1.local, rank 15 out of 32 processors
Hello world from processor compute-0-1.local, rank 16 out of 32 processors
Hello world from processor compute-0-1.local, rank 18 out of 32 processors
Hello world from processor compute-0-1.local, rank 19 out of 32 processors
Hello world from processor compute-0-1.local, rank 20 out of 32 processors
Hello world from processor compute-0-1.local, rank 21 out of 32 processors
Hello world from processor compute-0-1.local, rank 22 out of 32 processors
Hello world from processor compute-0-1.local, rank 24 out of 32 processors
Hello world from processor compute-0-1.local, rank 26 out of 32 processors
Hello world from processor compute-0-1.local, rank 27 out of 32 processors
Hello world from processor compute-0-1.local, rank 28 out of 32 processors
Hello world from processor compute-0-1.local, rank 29 out of 32 processors
Hello world from processor compute-0-1.local, rank 30 out of 32 processors
Hello world from processor compute-0-1.local, rank 31 out of 32 processors
Hello world from processor compute-0-1.local, rank 7 out of 32 processors
Hello world from processor compute-0-1.local, rank 10 out of 32 processors
Hello world from processor compute-0-1.local, rank 14 out of 32 processors
Hello world from processor compute-0-1.local, rank 1 out of 32 processors
Hello world from processor compute-0-1.local, rank 11 out of 32 processors
Hello world from processor compute-0-1.local, rank 13 out of 32 processors
Hello world from processor compute-0-1.local, rank 17 out of 32 processors
Hello world from processor compute-0-1.local, rank 23 out of 32 processors
Hello world from processor compute-0-1.local, rank 25 out of 32 processors



Any idea?

Regards,
Mahmood




___
users mailing list
users@lists.open-mpi.o

Re: [OMPI users] -host vs -hostfile

2017-07-31 Thread Gus Correa

Hi

With nodes=2:ppn=2 Torque will provide two cores on two nodes each
for your job.
Open MPI will honor this, and work only on those nodes and cores.

Torque will put the list of node names (repeated twice each, since
you asked for two ppn/cores) in a "node file" that can be accessed
in your job through the environment variable $PBS_NODEFILE.
That is the default hostfile (or host list) used by Open MPI when you
start your MPI executable with mpirun.
You don't need to add any hostfile or host list to the mpirun
command line.
And in principle there is no reason why you would
want to do that either, as that would be error prone,
at least not for SPMD (single executable) programs.

You can easily print the contents of that file to your job stdout
with:

cat $PBS_NODEFILE

If you add another hostfile or host list to the mpirun command line,
and if that hostfile or host list conflicts with the
contents $PBS_NODEFILE (say, has a different set of nodes),
mpirun will fail.

In my experience, the only situation you would need
to modify this scheme under Torque, is when launching an MPMD (multiple 
executables) program, to produce an --app appfile, as a modified version 
of $PBS_NODEFILE.
However, that doesn't seem to be the case here, as the mpirun command 
line in the various emails has a single executable "a.out".


I hope this helps.
Gus Correa

On 07/31/2017 12:43 PM, Elken, Tom wrote:

“4 threads”   In MPI, we refer to this as 4 ranks or 4 processes.

So what is your question?   Are you getting errors with PBS?

-Tom

*From:*users [mailto:users-boun...@lists.open-mpi.org] *On Behalf Of 
*Mahmood Naderan

*Sent:* Monday, July 31, 2017 9:27 AM
*To:* Open MPI Users <users@lists.open-mpi.org>
*Subject:* Re: [OMPI users] -host vs -hostfile

Excuse me, my fault.. I meant

nodes=2:ppn=2

is 4 threads.


Regards,
Mahmood

On Mon, Jul 31, 2017 at 8:49 PM, r...@open-mpi.org 
<mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
wrote:


?? Doesn't that tell pbs to allocate 1 node with 2 slots on it? I
don't see where you get 4

Sent from my iPad


On Jul 31, 2017, at 10:00 AM, Mahmood Naderan <mahmood...@gmail.com
<mailto:mahmood...@gmail.com>> wrote:

OK. The next question is how touse it with torque (PBS)?
currently we write this directive

Nodes=1:ppn=2

which means 4 threads. Then we omit -np and -hostfile in the
mpirun command.

On 31 Jul 2017 20:24, "Elken, Tom" <tom.el...@intel.com
<mailto:tom.el...@intel.com>> wrote:

Hi Mahmood,

With the -hostfile case, Open MPI is trying to helpfully run
things faster by keeping both processes on one host.  Ways
to avoid this…

On the mpirun command line add:

-pernode  (runs 1 process per node), oe

-npernode 1 ,   but these two has been deprecated in favor
of the wonderful syntax:

--map-by ppr:1:node

Or you could change your hostfile to:

|cluster slots=1|

|compute-0-0 slots=1|

-Tom

*From:*users [mailto:users-boun...@lists.open-mpi.org
<mailto:users-boun...@lists.open-mpi.org>] *On Behalf Of
*Mahmood Naderan
*Sent:* Monday, July 31, 2017 6:47 AM
*To:* Open MPI Users <users@lists.open-mpi.org
<mailto:users@lists.open-mpi.org>>
*Subject:* [OMPI users] -host vs -hostfile

Hi,

I have stuck at a problem which I don't remember that on
previous versions. when I run a test program with |-host|,
it works. I mean, the process spans to the hosts I
specified. However, when I specify |-hostfile|, it doesn't
work!!

|mahmood@cluster:mpitest$
/share/apps/computer/openmpi-2.0.1/bin/mpirun -host
compute-0-0,cluster -np 2 a.out|


||

|* hwloc 1.11.2 has encountered what looks like an error from
the operating system.|

|*|

|* Package (P#1 cpuset 0x) intersects with NUMANode
(P#1 cpuset 0xff00) without inclusion!|

|* Error occurred in topology.c line 1048|

|*|

|* The following FAQ entry in the hwloc documentation may help:|

|*   What should I do when hwloc reports "operating system"
warnings?|

|* Otherwise please report this error message to the hwloc
user's mailing list,|

|* along with the output+tarball generated by the
hwloc-gather-topology script.|


||

|H

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Gus Correa

On 07/17/2017 01:06 PM, Gus Correa wrote:

Hi Boris

The nodes may have standard Gigabit Ethernet interfaces,
besides the Infiniband (RoCE).
You may want to direct OpenMPI to use the Infiniband interfaces,
not Gigabit Ethernet,
by adding something like this to "--mca btl self,vader,self":

Oops! Typo:
"--mca btl self,vader,tcp"


"--mca btl_tcp_if_include ib0,ib1"

(Where the interface names ib0,ib1 are just my guess for
what your nodes may have. Check with your "root" system administrator!)

That syntax may also use IP address, or a subnet mask,
whichever it is simpler for you.
It is better explained in this FAQ:

https://www.open-mpi.org/faq/?category=all#tcp-selection

BTW, some of your questions (and others that you may hit later)
are covered in the OpenMPI FAQ:

https://www.open-mpi.org/faq/?category=all

I hope this helps,
Gus Correa


On 07/17/2017 12:43 PM, Boris M. Vulovic wrote:

Gus, Gilles, Russell, John:

Thanks very much for the replies and the help.
I got confirmation from the "root" that it is indeed RoCE with 100G.

I'll go over the info in the link Russell provided, but have a quick 
question: if I run the "*mpiexec*" with "*-mca btl tcp,self*" do I get 
the benefit of *RoCE *(the fastest speed)?


I'll go over the details of all reply and post useful feedback.

Thanks very much all!

Best,

--Boris




On Mon, Jul 17, 2017 at 6:31 AM, Russell Dekema <deke...@umich.edu 
<mailto:deke...@umich.edu>> wrote:


It looks like you have two dual-port Mellanox VPI cards in this
machine. These cards can be set to run InfiniBand or Ethernet on a
port-by-port basis, and all four of your ports are set to Ethernet
mode. Two of your ports have active 100 gigabit Ethernet links, and
the other two have no link up at all.

With no InfiniBand links on the machine, you will, of course, not be
able to run your OpenMPI job over InfiniBand.

If your machines and network are set up for it, you might be able to
run your job over RoCE (RDMA Over Converged Ethernet) using one or
both of those 100 GbE links. I have never used RoCE myself, but one
starting point for gathering more information on it might be the
following section of the OpenMPI FAQ:

https://www.open-mpi.org/faq/?category=openfabrics#ompi-over-roce
<https://www.open-mpi.org/faq/?category=openfabrics#ompi-over-roce>

Sincerely,
Rusty Dekema
University of Michigan
Advanced Research Computing - Technology Services


On Fri, Jul 14, 2017 at 12:34 PM, Boris M. Vulovic
<boris.m.vulo...@gmail.com <mailto:boris.m.vulo...@gmail.com>> wrote:
 > Gus, Gilles and John,
 >
 > Thanks for the help. Let me first post (below) the output from
checkouts of
 > the IB network:
 > ibdiagnet
 > ibhosts
 > ibstat  (for login node, for now)
 >
 > What do you think?
 > Thanks
 > --Boris
 >
 >
 >

 


 >
 > -bash-4.1$ ibdiagnet
 > --
 > Load Plugins from:
 > /usr/share/ibdiagnet2.1.1/plugins/
 > (You can specify more paths to be looked in with
"IBDIAGNET_PLUGINS_PATH"
 > env variable)
 >
 > Plugin Name   Result Comment
 > libibdiagnet_cable_diag_plugin-2.1.1  Succeeded  Plugin
loaded
 > libibdiagnet_phy_diag_plugin-2.1.1Succeeded  Plugin
loaded
 >
 > -
 > Discovery
 > -E- Failed to initialize
 >
 > -E- Fabric Discover failed, err=IBDiag initialize wasn't done
 > -E- Fabric Discover failed, MAD err=Failed to register SMI class
 >
 > -
 > Summary
 > -I- Stage Warnings   Errors Comment
 > -I- Discovery   NA
 > -I- Lids Check  NA
 > -I- Links Check NA
 > -I- Subnet Manager  NA
 > -I- Port Counters   NA
 > -I- Nodes Information   NA
 > -I- Speed / Width checksNA
 > -I- Partition Keys  NA
 > -I- Alias GUIDs NA
 > -I- Temperature Sensing NA
 >
 > -I- You can find detailed errors/warnings in:
 > /var/tmp/ibdiagnet2/ibdiagnet2.log
 >
 > -E- A fatal error occurred, exiting...
 > -bash-4.1$
 >

%

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Gus Correa

Hi Boris

The nodes may have standard Gigabit Ethernet interfaces,
besides the Infiniband (RoCE).
You may want to direct OpenMPI to use the Infiniband interfaces,
not Gigabit Ethernet,
by adding something like this to "--mca btl self,vader,self":

"--mca btl_tcp_if_include ib0,ib1"

(Where the interface names ib0,ib1 are just my guess for
what your nodes may have. Check with your "root" system administrator!)

That syntax may also use IP address, or a subnet mask,
whichever it is simpler for you.
It is better explained in this FAQ:

https://www.open-mpi.org/faq/?category=all#tcp-selection

BTW, some of your questions (and others that you may hit later)
are covered in the OpenMPI FAQ:

https://www.open-mpi.org/faq/?category=all

I hope this helps,
Gus Correa


On 07/17/2017 12:43 PM, Boris M. Vulovic wrote:

Gus, Gilles, Russell, John:

Thanks very much for the replies and the help.
I got confirmation from the "root" that it is indeed RoCE with 100G.

I'll go over the info in the link Russell provided, but have a quick 
question: if I run the "*mpiexec*" with "*-mca btl tcp,self*" do I get 
the benefit of *RoCE *(the fastest speed)?


I'll go over the details of all reply and post useful feedback.

Thanks very much all!

Best,

--Boris




On Mon, Jul 17, 2017 at 6:31 AM, Russell Dekema <deke...@umich.edu 
<mailto:deke...@umich.edu>> wrote:


It looks like you have two dual-port Mellanox VPI cards in this
machine. These cards can be set to run InfiniBand or Ethernet on a
port-by-port basis, and all four of your ports are set to Ethernet
mode. Two of your ports have active 100 gigabit Ethernet links, and
the other two have no link up at all.

With no InfiniBand links on the machine, you will, of course, not be
able to run your OpenMPI job over InfiniBand.

If your machines and network are set up for it, you might be able to
run your job over RoCE (RDMA Over Converged Ethernet) using one or
both of those 100 GbE links. I have never used RoCE myself, but one
starting point for gathering more information on it might be the
following section of the OpenMPI FAQ:

https://www.open-mpi.org/faq/?category=openfabrics#ompi-over-roce
<https://www.open-mpi.org/faq/?category=openfabrics#ompi-over-roce>

Sincerely,
Rusty Dekema
University of Michigan
Advanced Research Computing - Technology Services


On Fri, Jul 14, 2017 at 12:34 PM, Boris M. Vulovic
<boris.m.vulo...@gmail.com <mailto:boris.m.vulo...@gmail.com>> wrote:
 > Gus, Gilles and John,
 >
 > Thanks for the help. Let me first post (below) the output from
checkouts of
 > the IB network:
 > ibdiagnet
 > ibhosts
 > ibstat  (for login node, for now)
 >
 > What do you think?
 > Thanks
 > --Boris
 >
 >
 >


 >
 > -bash-4.1$ ibdiagnet
 > --
 > Load Plugins from:
 > /usr/share/ibdiagnet2.1.1/plugins/
 > (You can specify more paths to be looked in with
"IBDIAGNET_PLUGINS_PATH"
 > env variable)
 >
 > Plugin Name   Result Comment
 > libibdiagnet_cable_diag_plugin-2.1.1  Succeeded  Plugin
loaded
 > libibdiagnet_phy_diag_plugin-2.1.1Succeeded  Plugin
loaded
 >
 > -
 > Discovery
 > -E- Failed to initialize
 >
 > -E- Fabric Discover failed, err=IBDiag initialize wasn't done
 > -E- Fabric Discover failed, MAD err=Failed to register SMI class
 >
 > -
 > Summary
 > -I- Stage Warnings   Errors Comment
 > -I- Discovery   NA
 > -I- Lids Check  NA
 > -I- Links Check NA
 > -I- Subnet Manager  NA
 > -I- Port Counters   NA
 > -I- Nodes Information   NA
 > -I- Speed / Width checksNA
 > -I- Partition Keys  NA
 > -I- Alias GUIDs NA
 > -I- Temperature Sensing NA
 >
 > -I- You can find detailed errors/warnings in:
 > /var/tmp/ibdiagnet2/ibdiagnet2.log
 >
 > -E- A fatal error occurred, exiting...
 > -bash-4.1$
 >


Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-13 Thread Gus Correa

Have you tried:

-mca btl vader,openib,self

or

-mca btl sm,openib,self

by chance?

That adds a btl for intra-node communication (vader or sm).


On 07/13/2017 05:43 PM, Boris M. Vulovic wrote:


I would like to know how to invoke InfiniBand hardware on CentOS 6x 
cluster with OpenMPI (static libs.) for running my C++ code. This is how 
I compile and run:


/usr/local/open-mpi/1.10.7/bin/mpic++ -L/usr/local/open-mpi/1.10.7/lib 
-Bstatic main.cpp -o DoWork


usr/local/open-mpi/1.10.7/bin/mpiexec -mca btl tcp,self --hostfile 
hostfile5 -host node01,node02,node03,node04,node05 -n 200 DoWork


Here, "*-mca btl tcp,self*" reveals that *TCP* is used, and the cluster 
has InfiniBand.


What should be changed in compiling and running commands for InfiniBand 
to be invoked? If I just replace "*-mca btl tcp,self*" with "*-mca btl 
openib,self*" then I get plenty of errors with relevant one saying:


/At least one pair of MPI processes are unable to reach each other for 
MPI communications. This means that no Open MPI device has indicated 
that it can be used to communicate between these processes. This is an 
error; Open MPI requires that all MPI processes be able to reach each 
other. This error can sometimes be the result of forgetting to specify 
the "self" BTL./


Thanks very much!!!


*Boris *




___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


Re: [OMPI users] Help

2017-04-27 Thread Gus Correa

On 04/27/2017 06:21 AM, Corina Jeni Tudorache wrote:

Hello,



I am trying to install Open MPI on Centos and I got stuck. I have
installed an GNU compiler and after that I run the command: _yum install
openmpi-devel.x86_64. _But when I run command mpi selector –- list I
receive this error “mpi: command not found”

I am following the instruction from here:
https://na-inet.jp/na/pccluster/centos_x86_64-en.html

Any help is much appreciated. J

Corina


You need to install openmpi.x86_64 also, not only openmpi-devel.x86_64.
That is the minimum.

I hope this helps,
Gus Correa





___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


Re: [OMPI users] MPI_ABORT was invoked on rank 0 in communicator compute with errorcode 59

2016-11-15 Thread Gus Correa

Hi Mohammadali

"Signal number 11 SEGV", is the Unix/Linux signal for a memory
violation (a.k.a. segmentation violation or segmentation fault).
This normally happens when the program tries to read
or write in a memory area that it did not allocate, already
freed, or belongs to another process.
That is most likely a programming error on the FEM code,
probably not an MPI error, probably not a PETSC error either.

The "errorcode 59" seems to be the PETSC error message
issued when it receives a signal (in this case a
segmentation fault signal, I guess) from the operational
system (Linux, probably).
Apparently it simply throws the error message and
calls MPI_Abort, and the program stops.
This is what petscerror.h include file has about error code 59:

#define PETSC_ERR_SIG  59   /* signal received */

**

One suggestion is to compile the code with debugging flags (-g),
and attach a debugger to it. Not an easy task if you have many 
processes/ranks in your program, if your debugger is the default

Linux gdb, but it is not impossible to do either.
Depending on the computer you have, you may have a parallel debugger,
such as TotalView or DDT, which are more user friendly.

You could also compile it with the flag -traceback
(or -fbacktrace, the syntax depends on the compiler, check the compiler 
man page).
This at least will tell you the location in the program where the 
segmentation fault happened (in the STDERR file of your job).


I hope this helps.
Gus Correa

PS - The zip attachment with your "myjob.sh" script
was removed from the email.
Many email server programs remove zip for safety.
Files with ".sh" suffix are also removed in general.
You could compress it with gzip or bzip2 instead.

On 11/15/2016 02:40 PM, Beheshti, Mohammadali wrote:

Hi,



I am running simulations in a software which uses ompi to solve an FEM
problem.  From time to time I receive the error “

MPI_ABORT was invoked on rank 0 in communicator compute with errorcode
59” in the output file while for the larger simulations (with larger FEM
mesh) I almost always get this error. I don’t have any idea what is the
cause of this error. The error file contains a PETSC error: ”caught
signal number 11 SEGV”. I am running my jobs on a HPC system which has
Open MPI version 2.0.0.  I am also using a bash file (myjob.sh) which is
attached. The ompi_info - - all  command and ifconfig command outputs
are also attached. I appreciate any help in this regard.



Thanks



Ali





**

Mohammadali Beheshti

Post-Doctoral Fellow

Department of Medicine (Cardiology)

Toronto General Research Institute

University Health Network

Tel: 416-340-4800  ext. 6837



**




This e-mail may contain confidential and/or privileged information for
the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it
was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and
delete all copies.
Opinions, conclusions or other information contained in this e-mail may
not be that of the organization.



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


Re: [OMPI users] how to build with memchecker using valgrind, preferable linux distro install of valgrind?

2016-07-14 Thread Gus Correa

Maybe just --with-valgrind or --with-valgrind=/usr would work?

On 07/14/2016 11:32 AM, David A. Schneider wrote:

I thought it would be a good idea to build a debugging version of
openmpi 1.10.3. Following the instructions in the FAQ:
https://www.open-mpi.org/faq/?category=debugging#memchecker_how

I went with

.configure —with-verbs —enable-debug —enable-memchecker
—with-valgrind=/usr/bin/valgrind

Since I discovered valgrind was already installed on our network (this
is rhel7). However configure fails, and part of the error messages read

WARNING Expected file /usr/bin/include/valgrind/valgrind.h not found

Is there a way to build openmpi to use the installation of valgrind on
my system? I’m not sure if/where the valgrind.h file is, I don’t see it
in /usr/include. If not, I will build against my own build of valgrind,
but I run into similiar error messages - the auto tools script is not
finding the headers my valgrind install - I've tried giving a full path
to valgrind, the directory I used for PREFIX when I installed valgrind,
there must be some assumptions about environment variables I need to
have set to get the openmpi install script to work?

Reading through a page on valgrind:
http://valgrind.org/docs/manual/mc-manual.html#mc-manual.mpiwrap



I found instructions to the effect of how to wrap valgrind around an
existing open-mpi install. Does any one know which is better? That is,
what I'm trying, to build open-mpi based on a valgrind install, or to
install valgrind based on a open-mpi install? Doing both looks like a
circular dependency.

Best,

David
___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2016/07/29668.php





Re: [OMPI users] Restart after code hangs

2016-06-16 Thread Gus Correa

Hi Alex

You know all this, but just in case ...

Restartable code goes like this:

*
.start

read the initial/previous configuration from a file
...
final_step = first_step + nsteps
time_step = first_step
while ( time_step .le. final_step )
  ... march in time ...
  time_step = time_step +1
end

save the final_step configuration (or phase space) to a file
[depending on the algorithm you may need to save the
previous config also, or perhaps a few more]

.end


Then restart the job time and again, until the desired
number of time steps is completed.

Job queue systems/resource managers allow a job to resubmit itself,
so unless a job fails it feels like a single time integration.

If a job fails in the middle, you don't lose all work,
just restart from the previous saved configuration.
That is the only situation where you need to "monitor" the code.
Resource managers/ queue systems can also email you in
case the job fails, warning you to do manual intervention.

The time granularity per job (nsteps) is up to you.
Normally it is adjusted to the max walltime of job queues
(in a shared computer/cluster),
but in your case it can be adjusted to how often the program fails.

All atmosphere/ocean/climate/weather_forecast models work
this way (that's what we mostly run here).
I guess most CFD, computational Chemistry, etc, programs also do.

I hope this helps,
Gus Correa


On 06/16/2016 05:25 PM, Alex Kaiser wrote:

Hello,

I have an MPI code which sometimes hangs, simply stops running. It is
not clear why and it uses many large third party libraries so I do not
want to try to fix it. The code is easy to restart, but then it needs to
be monitored closely by me, and I'd prefer to do it automatically.

Is there a way, within an MPI process, to detect the hang and abort if so?

In psuedocode, I would like to do something like

for (all time steps)
 step
 if (nothing has happened for x minutes)

 call mpi abort to return control to the shell

 endif

endfor

This structure does not work however, as it can no longer do anything,
including check itself, when it is stuck.


Thank you,
Alex



___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/06/29471.php





Re: [OMPI users] "failed to create queue pair" problem, but settings appear OK

2016-06-15 Thread Gus Correa

On 06/15/2016 02:35 PM, Sasso, John (GE Power, Non-GE) wrote:

Chuck,

The per-process limits appear fine, including those for the resource mgr 
daemons:

Limit Soft Limit   Hard Limit   Units
Max address space unlimitedunlimitedbytes
Max core file size00bytes
Max cpu time  unlimitedunlimitedseconds
Max data size unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max file size unlimitedunlimitedbytes
Max locked memory unlimitedunlimitedbytes
Max msgqueue size 819200   819200   bytes
Max nice priority 00
Max open files1638416384files
Max pending signals   515625   515625   signals
Max processes 515625   515625   processes
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus
Max resident set  unlimitedunlimitedbytes
Max stack size30720unlimitedbytes



As for the FAQ re registered memory, checking our OpenMPI settings with 
ompi_info, we have:



Hi John

The FAQ I referred to (#18 in tuning runtime MPI to OpenFabrics)
regards the OFED kernel module parameters
log_num_mtt and log_mtts_per_seg, not to the openib btl mca parameters.
They may default to a less-than-optimal value.

https://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem

Gus Correa (not Chuck!)


mpool_rdma_rcache_size_limit = 0  ==> Open MPI will register as much user 
memory as necessary
btl_openib_free_list_max =  -1==> Open MPI will try to allocate as many 
registered buffers as it needs
btl_openib_eager_rdma_num = 16
btl_openib_max_eager_rdma = 16
btl_openib_eager_limit = 12288


Other suggestions welcome.   Hitting a brick wall here.  Thanks!

--john



-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
Sent: Wednesday, June 15, 2016 1:39 PM
To: Open MPI Users
Subject: EXT: Re: [OMPI users] "failed to create queue pair" problem, but 
settings appear OK

Hi John

1) For diagnostic, you could check the actual "per process" limits on the nodes 
while that big job is running:

cat /proc/$PID/limits

2) If you're using a resource manager to launch the job, the resource manager 
daemon/deamons (local to the nodes) may have to to set the memlock and other 
limits, so that the Open MPI processes inherit them.
I use Torque, so I put these lines in the pbs_mom (Torque local daemon) 
initialization script:

# pbs_mom system limits
# max file descriptors
ulimit -n 32768
# locked memory
ulimit -l unlimited
# stacksize
ulimit -s unlimited

3) See also this FAQ related to registered memory.
I set these parameters in /etc/modprobe.d/mlx4_core.conf, but where they're set 
may depend on the Linux distro/release and the OFED you're using.

https://urldefense.proofpoint.com/v2/url?u=https-3A__www.open-2Dmpi.org_faq_-3Fcategory-3Dopenfabrics-23ib-2Dlow-2Dreg-2Dmem=CwIF-g=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI=tqKZ2vRCLufSSXPvzNxBrKr01YPimBPnb-JT-Js0Fmk=fkBwjwn1Rvenp2NGwrQM3JtenpfbO_fxYUSK4lrHnzE=UFQ0uSWQoNPCfwg9q02YzMJczt7g4jEcaCvYOd46RRA=

I hope this helps,
Gus Correa

On 06/15/2016 11:05 AM, Sasso, John (GE Power, Non-GE) wrote:


In doing testing with IMB, I find that running a 4200+ core case with
the IMB test Alltoall, and message lengths of 16..1024 bytes (as per
-msglog 4:10 IMB option), it fails with:

--


A process failed to create a queue pair. This usually means either

the device has run out of queue pairs (too many connections) or

there are insufficient resources available to allocate a queue pair

(out of memory). The latter can happen if either 1) insufficient

memory is available, or 2) no more physical memory can be registered

with the device.

For more information on memory registration see the Open MPI FAQs at:

https://urldefense.proofpoint.com/v2/url?u=http-3A__www.open-2Dmpi.org
_faq_-3Fcategory-3Dopenfabrics-23ib-2Dlocked-2Dpages=CwIF-g=IV_clA
zoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI=tqKZ2vRCLufSSXPvzNxBrKr01YPimB
Pnb-JT-Js0Fmk=fkBwjwn1Rvenp2NGwrQM3JtenpfbO_fxYUSK4lrHnzE=dKT5yJta
2xW_ZUh06x95KTWjE1LgO8NU3OsjbwQsYLc=

Local host: node7106

Local device:   mlx4_0

Queue pair type:Reliable connected (RC)

--


[node7106][[51922,1],0][connect/btl_openib_connect_oob.c:867:rml_recv_
cb]
error in endpoint reply start connect

[node7106:06503] [[51922,0],0]-[[51922,1],0] mca_oob_

Re: [OMPI users] "failed to create queue pair" problem, but settings appear OK

2016-06-15 Thread Gus Correa

Hi John

1) For diagnostic, you could check the actual "per process" limits on 
the nodes while that big job is running:


cat /proc/$PID/limits

2) If you're using a resource manager to launch the job,
the resource manager daemon/deamons (local to the nodes) may have to
to set the memlock and other limits, so that the Open MPI processes
inherit them.
I use Torque, so I put these lines in the pbs_mom (Torque local daemon) 
initialization script:


# pbs_mom system limits
# max file descriptors
ulimit -n 32768
# locked memory
ulimit -l unlimited
# stacksize
ulimit -s unlimited

3) See also this FAQ related to registered memory.
I set these parameters in /etc/modprobe.d/mlx4_core.conf,
but where they're set may depend on the Linux distro/release and the 
OFED you're using.


https://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem

I hope this helps,
Gus Correa

On 06/15/2016 11:05 AM, Sasso, John (GE Power, Non-GE) wrote:


In doing testing with IMB, I find that running a 4200+ core case with 
the IMB test Alltoall, and message lengths of 16..1024 bytes (as per 
-msglog 4:10 IMB option), it fails with:


--

A process failed to create a queue pair. This usually means either

the device has run out of queue pairs (too many connections) or

there are insufficient resources available to allocate a queue pair

(out of memory). The latter can happen if either 1) insufficient

memory is available, or 2) no more physical memory can be registered

with the device.

For more information on memory registration see the Open MPI FAQs at:

http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

Local host: node7106

Local device:   mlx4_0

Queue pair type:Reliable connected (RC)

--

[node7106][[51922,1],0][connect/btl_openib_connect_oob.c:867:rml_recv_cb] 
error in endpoint reply start connect


[node7106:06503] [[51922,0],0]-[[51922,1],0] mca_oob_tcp_msg_recv: 
readv failed: Connection reset by peer (104)


--

mpirun has exited due to process rank 0 with PID 6504 on

node node7106 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in

the job did. This can cause a job to hang indefinitely while it waits

for all processes to call "init". By rule, if one process calls "init",

then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".

By rule, all processes that call "init" MUST call "finalize" prior to

exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be

terminated by signals sent by mpirun (as reported here).

--

Yes, these are ALL of the error messages. I did not get a message 
about not being able to register enough memory.   I verified that 
log_num_mtt = 24  and log_mtts_per_seg = 0 (via catting of their files 
in /sys/module/mlx4_core/parameters and what is set in 
/etc/modprobe.d/mlx4_core.conf).  While such a large-scale job runs, I 
run ‘vmstat 10’ to examine memory usage, but there appears to be a 
good amount of memory still available and swap is never used.   In 
terms of settings in /etc/security/limits.conf:


* soft memlock  unlimited

* hard memlock  unlimited

* soft stack 30

* hard stack unlimited

I don’t know if btl_openib_connect_oob.c or mca_oob_tcp_msg_recv are 
clues, but I am now at a loss as to where the problem lies.


This is for an application using OpenMPI 1.6.5, and the systems have 
Mellanox OFED 3.1.1 installed.


*--john*



___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/06/29455.php




Re: [OMPI users] [slightly off topic] hardware solutions with monetary cost in mind

2016-05-20 Thread Gus Correa

On 05/20/2016 02:40 PM, MM wrote:

Hello,

Say I don't have access to a actual cluster, yet I'm considering cloud
compute solutions for my MPI program ultimately, but such a cost may be
highly prohibitive at the moment.
In terms of middle ground, if I am interesting in compute only, no
storage, what are possible hardware solutions out there to deploy my MPI
program?
By no storage, I mean that my control linux box running the frontend of
the program, but is also part of the mpi communicator always gathers all
results and stores them locally.
At the moment, I have a second box over ethernet.

I am looking at something like Intel Compute Stick (is it possible at
all to buy a few, is linux running on them, the arch seems to be the
same x86-64, is there a possible setup with tcp for those and have
openmpi over tcp)?

Is it more cost-effective to look at extra regular linux commodity boxes?
If a no hard drive box is possible, can the executables of my MPI
program sendable over the wire before running them?

If we exclude GPU or other nonMPI solutions, and cost being a primary
factor, what is progression path from 2boxes to a cloud based solution
(amazon and the like...)

Regards,
MM


___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29257.php



1. You can run MPI programs in a single computer (multi-core 
multi-processor). So, in principle, you don't need a cluster, not even 
two machines.  If you want a proof of concept across Ethernet, two old 
desktops/laptops connected back to back (or through a cheap SOHO switch)

will do.

2. Not trying to dismiss your question, although its scope goes beyond 
MPI (and OpenMPI), and is more about HPC and clusters.

However, if you ask this question in the Beowulf mailing list,
you will get lots, tons, of advice, as the focus there is precisely
on HPC and clusters (of all sizes and for all budgets).

http://www.beowulf.org/mailman/listinfo/beowulf

I hope this helps,
Gus Correa



Re: [OMPI users] No core dump in some cases

2016-05-10 Thread Gus Correa

On 05/09/2016 04:59 PM, dpchoudh . wrote:

Hi Gus

Thanks for your suggestion. But I am not using any resource manager
(i.e. I am launching mpirun from the bash shell.). In fact, both of the
two clusters I talked about run CentOS 7 and I launch the job the same
way on both of these, yet one of them creates standard core files and
the other creates the 'btr; files. Strange thing is, I could not find
anything on the .btr (= Backtrace?) files on Google, which is any I
asked on this forum.

Best regards
Durga

The surgeon general advises you to eat right, exercise regularly and
quit ageing.


Hi Durga

My search showed something, but quite weirdly related to databases.
Maybe the same file extension used for two different things?
Does "file *.btr" tell anything?

Databases:

http://cs.pervasive.com/forums/p/14533/50237.aspx

... more databases ...

http://www.openthefile.net/extension/btr

... binary tree indexes ...

http://www.velocityreviews.com/threads/index-btr-file-in-windows-xp-help-please.307459/

... and a catalog of buterflies!  :)

http://filext.com/file-extension/BTR
http://review-tech.appspot.com/btr-file.html

Oh well ...

... and finally a previous incarnation of an OpenMPI 1.6.5 question 
similar to yours (where .btr stands for backtrace):


http://stackoverflow.com/questions/25275450/cause-all-processes-running-under-openmpi-to-dump-core

Could this be due to a (unlikely) mix of OpenMPI 1.10 with 1.6.5?

Gus Correa



On Mon, May 9, 2016 at 12:04 PM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:

Hi Durga

Just in case ...
If you're using a resource manager to start the jobs (Torque, etc),
you need to have them set the limits (for coredump size, stacksize,
locked memory size, etc).
This way the jobs will inherit the limits from the
resource manager daemon.
On Torque (which I use) I do this on the pbs_mom daemon
init script (I am still before the systemd era, that lovely POS).
And set the hard/soft limits on /etc/security/limits.conf as well.

I hope this helps,
Gus Correa

On 05/07/2016 12:27 PM, Jeff Squyres (jsquyres) wrote:

I'm afraid I don't know what a .btr file is -- that is not
something that is controlled by Open MPI.

You might want to look into your OS settings to see if it has
some kind of alternate corefile mechanism...?


On May 6, 2016, at 8:58 PM, dpchoudh . <dpcho...@gmail.com
<mailto:dpcho...@gmail.com>> wrote:

Hello all

I run MPI jobs (for test purpose only) on two different
'clusters'. Both 'clusters' have two nodes only, connected
back-to-back. The two are very similar, but not identical,
both software and hardware wise.

Both have ulimit -c set to unlimited. However, only one of
the two creates core files when an MPI job crashes. The
other creates a text file named something like

.80s-,.btr

I'd much prefer a core file because that allows me to debug
with a lot more options than a static text file with
addresses. How do I get a core file in all situations? I am
using MPI source from the master branch.

Thanks in advance
Durga

The surgeon general advises you to eat right, exercise
regularly and quit ageing.
___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:
https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2016/05/29124.php




___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2016/05/29141.php




___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29145.php





Re: [OMPI users] No core dump in some cases

2016-05-09 Thread Gus Correa

Hi Durga

Just in case ...
If you're using a resource manager to start the jobs (Torque, etc),
you need to have them set the limits (for coredump size, stacksize, 
locked memory size, etc).

This way the jobs will inherit the limits from the
resource manager daemon.
On Torque (which I use) I do this on the pbs_mom daemon
init script (I am still before the systemd era, that lovely POS).
And set the hard/soft limits on /etc/security/limits.conf as well.

I hope this helps,
Gus Correa

On 05/07/2016 12:27 PM, Jeff Squyres (jsquyres) wrote:

I'm afraid I don't know what a .btr file is -- that is not something that is 
controlled by Open MPI.

You might want to look into your OS settings to see if it has some kind of 
alternate corefile mechanism...?



On May 6, 2016, at 8:58 PM, dpchoudh . <dpcho...@gmail.com> wrote:

Hello all

I run MPI jobs (for test purpose only) on two different 'clusters'. Both 
'clusters' have two nodes only, connected back-to-back. The two are very 
similar, but not identical, both software and hardware wise.

Both have ulimit -c set to unlimited. However, only one of the two creates core 
files when an MPI job crashes. The other creates a text file named something 
like
.80s-,.btr

I'd much prefer a core file because that allows me to debug with a lot more 
options than a static text file with addresses. How do I get a core file in all 
situations? I am using MPI source from the master branch.

Thanks in advance
Durga

The surgeon general advises you to eat right, exercise regularly and quit 
ageing.
___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/05/29124.php







Re: [OMPI users] Segmentation Fault (Core Dumped) on mpif90 -v

2016-05-05 Thread Gus Correa

Hi Giacomo

Some programs fail with segmentation fault
because the stack size is too small.
[But others because of bugs in memory allocation/management, etc.]

Have you tried

ulimit -s unlimited

before you run the program?

Are you using a single machine or a cluster?
If you're using infiniband you may need also to make the locked memory 
unlimited:


ulimit -l unlimited

I hope this helps,
Gus Correa

On 05/05/2016 05:15 AM, Giacomo Rossi wrote:

  gdb /opt/openmpi/1.10.2/intel/16.0.3/bin/mpif90
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/openmpi/1.10.2/intel/16.0.3/bin/mpif90...(no
debugging symbols found)...done.
(gdb) r -v
Starting program: /opt/openmpi/1.10.2/intel/16.0.3/bin/mpif90 -v

Program received signal SIGSEGV, Segmentation fault.
0x76858f38 in ?? ()
(gdb) bt
#0  0x76858f38 in ?? ()
#1  0x77de5828 in _dl_relocate_object () from
/lib64/ld-linux-x86-64.so.2
#2  0x77ddcfa3 in dl_main () from /lib64/ld-linux-x86-64.so.2
#3  0x77df029c in _dl_sysdep_start () from
/lib64/ld-linux-x86-64.so.2
#4  0x774a in _dl_start () from /lib64/ld-linux-x86-64.so.2
#5  0x77dd9d98 in _start () from /lib64/ld-linux-x86-64.so.2
#6  0x0002 in ?? ()
#7  0x7fffaa8a in ?? ()
#8  0x7fffaab6 in ?? ()
#9  0x in ?? ()

Giacomo Rossi Ph.D., Space Engineer

Research Fellow at Dept. of Mechanical and Aerospace Engineering,
"Sapienza" University of Rome
*p: *(+39) 0692927207 | *m**: *(+39) 3408816643 | *e:
*giacom...@gmail.com <mailto:giacom...@gmail.com>
<mailto:giacomo.ro...@uniroma1.it>
Member of Fortran-FOSS-programmers
<https://github.com/Fortran-FOSS-Programmers>


2016-05-05 10:44 GMT+02:00 Giacomo Rossi <giacom...@gmail.com
<mailto:giacom...@gmail.com>>:

Here the result of ldd command:
'ldd /opt/openmpi/1.10.2/intel/16.0.3/bin/mpif90
linux-vdso.so.1 (0x7ffcacbbe000)
libopen-pal.so.13 =>
/opt/openmpi/1.10.2/intel/16.0.3/lib/libopen-pal.so.13
(0x7fa9597a9000)
libm.so.6 => /usr/lib/libm.so.6 (0x7fa9594a4000)
libpciaccess.so.0 => /usr/lib/libpciaccess.so.0 (0x7fa95929a000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x7fa959096000)
librt.so.1 => /usr/lib/librt.so.1 (0x7fa958e8e000)
libutil.so.1 => /usr/lib/libutil.so.1 (0x7fa958c8b000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fa958a75000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x7fa958858000)
libc.so.6 => /usr/lib/libc.so.6 (0x7fa9584b7000)
libimf.so =>

/home/giacomo/intel/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64/libimf.so
(0x7fa957fb9000)
libsvml.so =>

/home/giacomo/intel/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64/libsvml.so
(0x7fa9570ad000)
libirng.so =>

/home/giacomo/intel/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64/libirng.so
(0x7fa956d3b000)
libintlc.so.5 =>

/home/giacomo/intel/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64/libintlc.so.5
(0x7fa956acf000)
/lib64/ld-linux-x86-64.so.2 (0x7fa959ab9000)'

I can't provide a core file, because I can't compile or launch any
program with mpifort... I've always the error 'core dumped' also
when I try to compile a program with mpifort, and of course there
isn't any core file.


Giacomo Rossi Ph.D., Space Engineer

Research Fellow at Dept. of Mechanical and Aerospace
Engineering, "Sapienza" University of Rome
*p: *(+39) 0692927207 <tel:%28%2B39%29%200692927207> | *m**:
*(+39) 3408816643 <tel:%28%2B39%29%203408816643> | *e:
*giacom...@gmail.com <mailto:giacom...@gmail.com>
<mailto:giacomo.ro...@uniroma1.it>
Member of Fortran-FOSS-programmers
<https://github.com/Fortran-FOSS-Programmers>


2016-05-05 8:50 GMT+02:00 Giacomo Rossi <giacom...@gmail.com
<mailto:giacom...@gmail.com>>:

I’ve installed the latest version of Intel Parallel Studio
(16.0.3), t

Re: [OMPI users] MPIRUN SEGMENTATION FAULT

2016-04-25 Thread Gus Correa

Hi Elio

As Gilles said, if you change the integer size to -i8 in the 
application, and MPI was built with the default-sized integers

(4 bytes), things will get really ugly and mismatched.
Better avoid flags such as -i8, -r8, etc, when compiling MPI programs.

Have you tried to compile the code with the -traceback flag?
This at least should tell you where the code is failing (source file
and line).
As Ralph said, most likely the program is trying to
go beyond array boundaries,
or accessing non-allocated memory.
That can happen even with innocent strings
(say a file name or an informative message that are too big).

A better approach, as suggested by Ralph,
is to open the core file with gdb.
It should be named something like "core.98765", where the "98765" is the 
process number.
However, many Linux distributions set the core file size to zero by 
default, which prevents the core file to be created when the program 
crashes, but on the upside also prevents disk to fill up with big core 
files that are forgotten and hang around forever.

[ulimit -a will tell.]

I hope this helps,
Gus Correa


On 04/23/2016 07:06 PM, Gilles Gouaillardet wrote:

If you build your application with intel compilers and -i8, then openmpi
must also be built with intel compilers and -i8.

Cheers,

Gilles

On Sunday, April 24, 2016, Elio Physics <elio-phys...@live.com
<mailto:elio-phys...@live.com>> wrote:

Well, I changed the compiler from mpif90 to mpiifort with
corresponding flags -i8 -g and recompiled. i am not getting the
segmentation fault problem anymore and the program runs but later
stops with no errors (except that the Fermi energy was not found!)
and with some strange empty files that are produced something like:
   fortDgcQe3  fortechvF2  fortMaN6a1  fortnxoYy1  fortvR5F8q.  i
still feel something is wrong.. Does anybody know what are these files?


Regards




*From:* users <users-boun...@open-mpi.org
<javascript:_e(%7B%7D,'cvml','users-boun...@open-mpi.org');>> on
behalf of Ralph Castain <r...@open-mpi.org
<javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');>>
*Sent:* Saturday, April 23, 2016 1:38 PM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] MPIRUN SEGMENTATION FAULT
I don’t see any way this could be compilation related - I suspect
there is simply some error in the program (e.g., forgetting to
initialize some memory region).



On Apr 23, 2016, at 8:03 AM, Elio Physics <elio-phys...@live.com
<javascript:_e(%7B%7D,'cvml','elio-phys...@live.com');>> wrote:

Hello Andy,

the program is not mine. I have got it from a group upon request.
It might be program related because I run other codes such as
quantum espresso and work perfectly fine although it is the
cluster people who compiled it. Since I have compiled the program
I am having problems with, I am thinking that it might be
"compilation" related. This is why i wanted some experts' opinion
on this




*From:*users <users-boun...@open-mpi.org
<javascript:_e(%7B%7D,'cvml','users-boun...@open-mpi.org');>> on
behalf of Andy Riebs <andy.ri...@hpe.com
<javascript:_e(%7B%7D,'cvml','andy.ri...@hpe.com');>>
*Sent:*Saturday, April 23, 2016 12:49 PM
*To:*us...@open-mpi.org
<javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
*Subject:*Re: [OMPI users] MPIRUN SEGMENTATION FAULT
The challenge for the MPI experts here (of which I am NOT one!) is
that the problem appears to be in your program; MPI is simply
reporting that your program failed. If you got the program from
someone else, you will need to solicit their help. If you wrote
it, well, it is never a bad time to learn to use gdb!

Best regards
Andy

On 04/23/2016 10:41 AM, Elio Physics wrote:

I am not really an expert with gdb. What is the core file? and
how to use gdb? I have got three files as an output when the
executable is used. One is the actual output which stops and the
other two are error files (from which I knew about the
segmentation fault).


thanks



*From:*users<users-boun...@open-mpi.org>
<javascript:_e(%7B%7D,'cvml','users-boun...@open-mpi.org');>on
behalf of Ralph Castain<r...@open-mpi.org>
<javascript:_e(%7B%7D,'cvml','r...@open-mpi.org');>
*Sent:*Saturday, April 23, 2016 11:39 AM
*To:*Open MPI Users
*Subject:*Re: [OMPI users] MPIRUN SEGMENTATION FAULT
valgrind isn’t going to help here - there are multiple reasons
why your application could be segfaulting. Take a look at the
core file with gdb and find out whe

Re: [OMPI users] Problems in compiling a code with dynamic linking

2016-03-24 Thread Gus Correa

Hi Elie

Besides Gilles' and Thomas' suggestions:

1) Do you have any file system in your cluster head node that is an
NFS export, and presumably mounted on the compute nodes?

If you do, that would be the best place to install the Intel compiler.
This would make it available on the compute nodes, and the 
compilervars.sh script would be OK everywhere.

Say, something like /my/nfs/shared/software/intel/version

You will need to append the corresponding bin
subdirectories to your PATH, and the corresponding
lib subdirectories to your LD_LIBRARY_PATH.

Actually, in a small cluster, that is the best/easy location to install
any software applications that need to be shared by all nodes,
and this includes compilers, MPI (Open MPI), etc.

Installing applications in /opt or /usr/local, which as Gilles' said
are local to each node, is sure to put you in a dead end.
Everything will be only available on the head node, nothing on the
compute nodes.

Often times the person that installed the cluster first time
didn't realize this, and only made /home an NFS share,
and now (s)he doesn't have any free disk or disk partition to make
an additional NFS share for software.
In this case the remedy is to install such software applications
in, say, /home/software.

**

2) As Gilles' said, /opt is a local file system, you have one on the
head node, where the compiler was installed, and a (different) /opt
on each compute node (where there is no compiler installed).

Hence, even Thomas' suggestion is unlikely to work, because there is no
intel compiler on compute_node:/opt.

A brute force solution would be to install the Intel compiler on all 
nodes, on /opt, but this is not very nice (and a maintenance/consistency 
nightmare).


You could also install only the Intel runtime libraries on the nodes'
/opt, which *probably* will work:

https://software.intel.com/en-us/articles/intelr-composer-redistributable-libraries-by-version

**

I hope this helps,
Gus Correa

On 03/24/2016 12:01 AM, Gilles Gouaillardet wrote:

Elio,

usually, /opt is a local filesystem, so it is possible /opt/intel is
only available on your login nodes.

your best option is to ask your sysadmin where the mkl libs are on the
compute nodes, and/or how to use mkl in your jobs.

feel free to submit a dumb pbs script
ls -l /opt
ls -l /opt/intel
ls -l /opt/intel/mkl
so you can hopefully find that by yourself.

an other option is to use the static mkl libs if they are available
for example, your LIB line could be

LIB = -static -L/opt/intel/composer_xe_2013_sp1/mkl/lib/intel64
-lmkl_blas95_lp64 -lmkl_lapack95_lp64  -lmkl_intel_lp64 -lmkl_core
-lmkl_sequential -dynamic

Cheers,

Gilles

On 3/24/2016 12:43 PM, Elio Physics wrote:


Dear Gilles,


thanks for your reply and your options. I have tried the first option,
hich for me basically is the easiest. I have compiled using "make.inc"
but now setting  LIB = -L/opt/intel/mkl/lib/intel64 -lmkl_blas95_lp64
-lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential


Every went well. Then I tried the PBS script wjere I have added  these
two lines:


source /opt/intel/mkl/bin/mklvars.sh
export LD_LIBRARY_PATH=/opt/intel/mkl/bin/mklvars.sh


But i still get the same error:


/opt/intel/mkl/bin/mklvars.sh: No such file or directory
/home/emoujaes/Elie/SPRKKR/bin/kkrscf6.3MPI: error while loading
shared libraries: libmkl_intel_lp64.so: cannot open shared object
file: No such file or directory



I just cannot understand why is it giving the same error and why it
could not find the file : /opt/intel/mkl/bin/mklvars.sh although the
link is true!


Any advice please?


Thanks





*From:* users <users-boun...@open-mpi.org> on behalf of Gilles
Gouaillardet <gil...@rist.or.jp>
*Sent:* Thursday, March 24, 2016 12:22 AM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] Problems in compiling a code with dynamic
linking
Elio,

it seems /opt/intel/composer_xe_2013_sp1/bin/compilervars.sh is only
available on your login/frontend nodes,
but not on your compute nodes.
you might be luckier with
/opt/intel/mkl/bin/mklvars.sh

an other option is to
ldd /home/emoujaes/Elie/SPRKKR/bin/kkrscf6.3MPI
on your login node, and explicitly set the LD_LIBRARY_PATH in your PBS
script

if /opt/intel/composer_xe_2013_sp1/mkl/lib/intel64 is available on
your compute nodes, you might want to append
-Wl,-rpath,/opt/intel/composer_xe_2013_sp1/mkl/lib/intel64
to LIB
/* if you do that, keep in mind you might not automatically use the
most up to date mkl lib when they get upgraded by your sysadmin */

Cheers,

Gilles

On 3/24/2016 11:03 AM, Elio Physics wrote:


Dear all,


I have been trying ,for the last week, compiling a code (SPRKKR). the
compilation went through ok. however, there are problems with the
executable (kkrscf6.3MPI) not finding the MKL library links. i could
not fix the problem..I have tried several things but in vain..I will
post both the &q

Re: [OMPI users] cleaning up old ROMIO (MPI-IO) drivers

2016-01-05 Thread Gus Correa

Hi Rob

Your email says you'll keep PVFS2.
However, on your blog PVFS2 is not mentioned (on the "Keep" list).
I suppose it will be kept, right?

Thank you,
Gus Correa

On 01/05/2016 12:31 PM, Rob Latham wrote:

I'm itching to discard some of the little-used file system drivers in
ROMIO, an MPI-IO implementation used by, well, everyone.

I've got more details in this ROMIO blog post:
http://press3.mcs.anl.gov/romio/2016/01/05/cleaning-out-old-romio-file-system-drivers/



Right now the plan is to keep these drivers:

- Lustre
- GPFS
- UFS
- PVFS2
- PANFS
- NFS
- TESTFS
- XFS

Do you use any of the other ROMIO file system drivers?  If you don't
know if you do, or don't know what a ROMIO file system driver is, then
it's unlikely you are using one.

What if you use a driver and it's not on the list?  First off, let me
know and I will probably want to visit your site and take a picture of
your system.  Then, let me know how much longer you foresee using the
driver and we'll create a "deprecated" list for N more years.

Thanks
==rob






Re: [OMPI users] pbs vs openmpi node allocation

2015-08-03 Thread Gus Correa

Hi Abhisek

On 08/03/2015 12:59 PM, abhisek Mondal wrote:

Hi,

   I'm using openmpi-1.6.4 to distribute a jobs in 2 different nodes
using this command:
/"mpirun --hostfile myhostfile -np 10 nwchem my_code.nw"/
Here, "myhostfile" contains:
/cx0937 slots=5 /
/cx0934 slots=5/


I am assuming by pbs you mean Torque.
If your Open MPI was built with Torque support (--with-tm),
then you don't even need the --hostfile option
(and probably shouldn't use it).
Unless newchem behaves in a very non-standard way,
which I don't really know.

Open MPI will use the nodes provided by Torque.

To check this do;

ompi_info |grep tm

In the Open MPI parlance Torque is "tm".



But as I have to submit the jobs using .pbs script, I'm wondering in
this case, how "mpirun" going to choose the node (free node allocation
is done by pbs) from "myhostfile".
I mean, does it happen that until the specific-nodes (as mentioned in
myhostfile) become free "mpirun" is going to wait and then start ?
How can I forward the allocated node name(by pbs) to /mpirun/ command ?

A little light on this matter would be really great.



Your script only starts after Torque allocates the nodes and starts the 
script on the first node.

Mpirun doesn't choose the nodes, it uses it.
If you are using Torque it may be worth looking into some of its 
environment variables.


"man qsub" will tell you a lot about them, and probably will clarify
many things more.

Some very useful ones are:

   PBS_O_WORKDIR
  the absolute path of the current working directory of the 
qsub command.


   PBS_JOBID
  the job identifier assigned to the job by the batch system.

   PBS_JOBNAME
  the job name supplied by the user.

   PBS_NODEFILE
  the name of the file contain the list of nodes assigned 
to the job (for parallel and cluster systems).



***

In your script you can

cd $PBS_O_WORKDIR

as by default Torque puts you in your home directory in the compute 
node, which may not be where you want to be.


Another way to document the nodes that you're using is to put this
line in your script:

cat $PBS_NODEFILE

which will list the nodes (repeated by as many cores/CPUs as you have 
requested from each node).  Actually, if you ever want to use the

mpirun --hostfile option, the actual node file would be $PBS_NODEFILE.
[You don't need to do it if Open MPI was built with Torque support.]



I hope this helps.
Gus Correa



Thank you.

--
Abhisek Mondal
/Research Fellow
/
/Structural Biology and Bioinformatics
/
/Indian Institute of Chemical Biology/
/Kolkata 700032
/
/INDIA
/


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/08/27383.php





Re: [OMPI users] shared memory performance

2015-07-22 Thread Gus Correa

Hi Christian, list

I haven't been following the shared memory details of OMPI lately,
but my recollection from some time ago is that in the 1.8 series the
default (and recommended) shared memory transport btl switched from
"sm" to "vader", which is the latest greatest.

In this case, I guess the mpirun options would be:

mpirun --machinefile machine_mpi_bug.txt --mca btl self,vader,tcp


I am not even sure if with "vader" the "self" btl is needed,
as it was the case with "sm".

An OMPI developer could jump into this conversation and clarify.
Thank you.

I hope this helps,
Gus Correa


On 07/22/2015 04:42 AM, Crisitan RUIZ wrote:

Thank you for your answer Harald

Actually I was already using TAU before but it seems that it is not
maintained any more and there are problems when instrumenting
applications with the version 1.8.5 of OpenMPI.

I was using the OpenMPI 1.6.5 before to test the execution of HPC
application on Linux containers. I tested the performance of NAS
benchmarks in three different configurations:

- 8 Linux containers on the same machine configured with 2 cores
- 8 physical machines
- 1 physical machine

So, as I already described it, each machine counts with 2 processors (8
cores each). I instrumented and run all NAS benchmark in these three
configurations and I got the results that I attached in this email.
In the table "native" corresponds to using 8 physical machines and "SM"
corresponds to 1 physical machine using the sm module, time is given in
miliseconds.

What surprise me in the results is that using containers in the worse
case have equal communication time than just using plain mpi processes.
Even though the containers use virtual network interfaces to communicate
between them. Probably this behaviour is due to process binding and
locality, I wanted to redo the test using OpenMPI version 1.8.5 but
unfourtunately I couldn't sucessfully instrument the applications. I was
looking for another MPI profiler but I couldn't find any. HPCToolkit
looks like it is not maintain anymore, Vampir does not maintain any more
the tool that instrument the application.  I will probably give a try to
Paraver.



Best regards,

Cristian Ruiz



On 07/22/2015 09:44 AM, Harald Servat wrote:


Cristian,

  you might observe super-speedup heres because in 8 nodes you have 8
times the cache you have in only 1 node. You can also validate that by
checking for cache miss activity using the tools that I mentioned in
my other email.

Best regards.

On 22/07/15 09:42, Crisitan RUIZ wrote:

Sorry, I've just discovered that I was using the wrong command to run on
8 machines. I have to get rid of the "-np 8"

So, I corrected the command and I used:

mpirun --machinefile machine_mpi_bug.txt --mca btl self,sm,tcp
--allow-run-as-root mg.C.8

And got these results

8 cores:

Mop/s total = 19368.43


8 machines

Mop/s total = 96094.35


Why is the performance of mult-node almost 4 times better than
multi-core? Is this normal behavior?

On 07/22/2015 09:19 AM, Crisitan RUIZ wrote:


 Hello,

I'm running OpenMPI 1.8.5 on a cluster with the following
characteristics:

Each node is equipped with two Intel Xeon E5-2630v3 processors (with 8
cores each), 128 GB of RAM and a 10 Gigabit Ethernet adapter.

When I run the NAS benchmarks using 8 cores in the same machine, I'm
getting almost the same performance as using 8 machines running a mpi
process per machine.

I used the following commands:

for running multi-node:

mpirun -np 8 --machinefile machine_file.txt --mca btl self,sm,tcp
--allow-run-as-root mg.C.8

for running in with 8 cores:

mpirun -np 8  --mca btl self,sm,tcp --allow-run-as-root mg.C.8


I got the following results:

8 cores:

 Mop/s total = 19368.43

8 machines:

Mop/s total = 19326.60


The results are similar for other benchmarks. Is this behavior normal?
I was expecting to see a better behavior using 8 cores given that they
use directly the memory to communicate.

Thank you,

Cristian
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/07/27295.php


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/07/27297.php



WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosin

Re: [OMPI users] mpirun fails across cluster

2015-02-27 Thread Gus Correa

Hi Syed Ahsan Ali

On 02/27/2015 12:46 PM, Syed Ahsan Ali wrote:

Oh sorry. That is related to application. I need to recompile
application too I guess.


You surely do.

Also, make sure the environment, in particular PATH and LD_LIBRARY_PATH
is propagated to the compute nodes.
Not doing that is a common cause of trouble.
OpenMPI needs PATH and LD_LIBRARY_PATH at runtime also.

I hope this helps,
Gus Correa



On Fri, Feb 27, 2015 at 10:44 PM, Syed Ahsan Ali <ahsansha...@gmail.com> wrote:

Dear Gus

Thanks once again for suggestion. Yes I did that before installation
to new path. I am getting error now about some library
tstint2lm: error while loading shared libraries:
libmpi_usempif08.so.0: cannot open shared object file: No such file or
directory

While library is present
[pmdtest@hpc bin]$ locate libmpi_usempif08.so.0
/state/partition1/apps/openmpi-1.8.4_gcc-4.9.2/lib/libmpi_usempif08.so.0
/state/partition1/apps/openmpi-1.8.4_gcc-4.9.2/lib/libmpi_usempif08.so.0.6.0
in path as well

echo $LD_LIBRARY_PATH
/share/apps/openmpi-1.8.4_gcc-4.9.2/lib:/share/apps/libpng-1.6.16/lib:/share/apps/netcdf-fortran-4.4.1_gcc-4.9.2_wo_hdf5/lib:/share/apps/netcdf-4.3.2_gcc_wo_hdf5/lib:/share/apps/grib_api-1.11.0/lib:/share/apps/jasper-1.900.1/lib:/share/apps/zlib-1.2.8_gcc-4.9.2/lib:/share/apps/gcc-4.9.2/lib64:/share/apps/gcc-4.9.2/lib:/usr/lib64:/usr/share/Modules/lib:/opt/python/lib
[pmdtest@hpc bin]$

Ahsan

On Fri, Feb 27, 2015 at 10:17 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:

Hi Syed Ahsan Ali

To avoid any leftovers and further confusion,
I suggest that you delete completely the old installation directory.
Then start fresh from the configure step with the prefix pointing to
--prefix=/share/apps/openmpi-1.8.4_gcc-4.9.2

I hope this helps,
Gus Correa

On 02/27/2015 12:11 PM, Syed Ahsan Ali wrote:


Hi Gus

Thanks for prompt response. Well judged, I compiled with /export/apps
prefix so that is most probably the reason. I'll check and update you.

Best wishes
Ahsan

On Fri, Feb 27, 2015 at 10:07 PM, Gus Correa <g...@ldeo.columbia.edu>
wrote:


Hi Syed

This really sounds as a problem specific to Rocks Clusters,
not an issue with Open MPI.
A confusion related to mount points, and soft links used by Rocks.

I haven't used Rocks Clusters in a while,
and I don't remember the details anymore, so please take my
suggestions with a grain of salt, and check them out
before committing to them

Which --prefix did you use when you configured Open MPI?
My suggestion is that you don't use "/export/apps" as a prefix
(and this goes to any application that you install).
but instead use a /share/apps subdirectory, something like:

--prefix=/share/apps/openmpi-1.8.4_gcc-4.9.2

This is because /export/apps is just a mount point on the
frontend/head node, whereas /share/apps is a mount point
across all nodes in the cluster (and, IIRR, a soft link on the
head node).

My recollection is that the Rocks documentation was obscure
about this, not making clear the difference between
/export/apps and /share/apps.

Issuing the Rocks commands:
"tentakel 'ls -d /export/apps'"
"tentakel 'ls -d /share/apps'"
may show something useful.

I hope this helps,
Gus Correa


On 02/27/2015 11:47 AM, Syed Ahsan Ali wrote:



I am trying to run openmpi application on my cluster.  But the mpirun
fails, simple hostname command gives this error

[pmdtest@hpc bin]$ mpirun --host compute-0-0 hostname

--
Sorry!  You were supposed to get help about:
   opal_init:startup:internal-failure
But I couldn't open the help file:


/export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:
No such file or directory.  Sorry!

--

--
Sorry!  You were supposed to get help about:
   orte_init:startup:internal-failure
But I couldn't open the help file:

/export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-orte-runtime:
No such file or directory.  Sorry!

--
[compute-0-0.local:03410] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in
file orted/orted_main.c at line 369

--
ORTE was unable to reliably start one or more daemons.

I am using Environment modules to load OpenMPI 1.8.4 and PATH and
LD_LIBRARY_PATH points to same openmpi on nodes

[pmdtest@hpc bin]$ which mpirun
/share/apps/openmpi-1.8.4_gcc-4.9.2/bin/mpirun
[pmdtest@hpc bin]$ ssh compute-0-0
Last login: Sat Feb 28 02:15:50 2015 from hpc.local
Rocks Compute Node
Rocks 6.1.1 (Sand Boa)
Profile built 01:53 28-Feb-2015
Kickstarted 01:59 28-Feb-2015
[pmdtest@compute-0-0 ~]$ which mpirun
/share/apps/openmpi-1.8.4_gcc-4.9.2/bin/mpirun

The only this I notice important is that in the error it is ref

Re: [OMPI users] mpirun fails across cluster

2015-02-27 Thread Gus Correa

Hi Syed Ahsan Ali

To avoid any leftovers and further confusion,
I suggest that you delete completely the old installation directory.
Then start fresh from the configure step with the prefix pointing to
--prefix=/share/apps/openmpi-1.8.4_gcc-4.9.2

I hope this helps,
Gus Correa

On 02/27/2015 12:11 PM, Syed Ahsan Ali wrote:

Hi Gus

Thanks for prompt response. Well judged, I compiled with /export/apps
prefix so that is most probably the reason. I'll check and update you.

Best wishes
Ahsan

On Fri, Feb 27, 2015 at 10:07 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:

Hi Syed

This really sounds as a problem specific to Rocks Clusters,
not an issue with Open MPI.
A confusion related to mount points, and soft links used by Rocks.

I haven't used Rocks Clusters in a while,
and I don't remember the details anymore, so please take my
suggestions with a grain of salt, and check them out
before committing to them

Which --prefix did you use when you configured Open MPI?
My suggestion is that you don't use "/export/apps" as a prefix
(and this goes to any application that you install).
but instead use a /share/apps subdirectory, something like:

--prefix=/share/apps/openmpi-1.8.4_gcc-4.9.2

This is because /export/apps is just a mount point on the
frontend/head node, whereas /share/apps is a mount point
across all nodes in the cluster (and, IIRR, a soft link on the
head node).

My recollection is that the Rocks documentation was obscure
about this, not making clear the difference between
/export/apps and /share/apps.

Issuing the Rocks commands:
"tentakel 'ls -d /export/apps'"
"tentakel 'ls -d /share/apps'"
may show something useful.

I hope this helps,
Gus Correa


On 02/27/2015 11:47 AM, Syed Ahsan Ali wrote:


I am trying to run openmpi application on my cluster.  But the mpirun
fails, simple hostname command gives this error

[pmdtest@hpc bin]$ mpirun --host compute-0-0 hostname
--
Sorry!  You were supposed to get help about:
  opal_init:startup:internal-failure
But I couldn't open the help file:

/export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:
No such file or directory.  Sorry!
--
--
Sorry!  You were supposed to get help about:
  orte_init:startup:internal-failure
But I couldn't open the help file:
  /export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-orte-runtime:
No such file or directory.  Sorry!
--
[compute-0-0.local:03410] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in
file orted/orted_main.c at line 369
--
ORTE was unable to reliably start one or more daemons.

I am using Environment modules to load OpenMPI 1.8.4 and PATH and
LD_LIBRARY_PATH points to same openmpi on nodes

[pmdtest@hpc bin]$ which mpirun
/share/apps/openmpi-1.8.4_gcc-4.9.2/bin/mpirun
[pmdtest@hpc bin]$ ssh compute-0-0
Last login: Sat Feb 28 02:15:50 2015 from hpc.local
Rocks Compute Node
Rocks 6.1.1 (Sand Boa)
Profile built 01:53 28-Feb-2015
Kickstarted 01:59 28-Feb-2015
[pmdtest@compute-0-0 ~]$ which mpirun
/share/apps/openmpi-1.8.4_gcc-4.9.2/bin/mpirun

The only this I notice important is that in the error it is referring to

/export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:

While it should have shown
/share/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:
which is the path compute nodes see.

Please help!
Ahsan
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/02/26411.php



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/02/26412.php








Re: [OMPI users] mpirun fails across cluster

2015-02-27 Thread Gus Correa

Hi Syed

This really sounds as a problem specific to Rocks Clusters,
not an issue with Open MPI.
A confusion related to mount points, and soft links used by Rocks.

I haven't used Rocks Clusters in a while,
and I don't remember the details anymore, so please take my
suggestions with a grain of salt, and check them out
before committing to them

Which --prefix did you use when you configured Open MPI?
My suggestion is that you don't use "/export/apps" as a prefix
(and this goes to any application that you install).
but instead use a /share/apps subdirectory, something like:

--prefix=/share/apps/openmpi-1.8.4_gcc-4.9.2

This is because /export/apps is just a mount point on the
frontend/head node, whereas /share/apps is a mount point
across all nodes in the cluster (and, IIRR, a soft link on the
head node).

My recollection is that the Rocks documentation was obscure
about this, not making clear the difference between
/export/apps and /share/apps.

Issuing the Rocks commands:
"tentakel 'ls -d /export/apps'"
"tentakel 'ls -d /share/apps'"
may show something useful.

I hope this helps,
Gus Correa

On 02/27/2015 11:47 AM, Syed Ahsan Ali wrote:

I am trying to run openmpi application on my cluster.  But the mpirun
fails, simple hostname command gives this error

[pmdtest@hpc bin]$ mpirun --host compute-0-0 hostname
--
Sorry!  You were supposed to get help about:
 opal_init:startup:internal-failure
But I couldn't open the help file:
 /export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:
No such file or directory.  Sorry!
--
--
Sorry!  You were supposed to get help about:
 orte_init:startup:internal-failure
But I couldn't open the help file:
 /export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-orte-runtime:
No such file or directory.  Sorry!
--
[compute-0-0.local:03410] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in
file orted/orted_main.c at line 369
--
ORTE was unable to reliably start one or more daemons.

I am using Environment modules to load OpenMPI 1.8.4 and PATH and
LD_LIBRARY_PATH points to same openmpi on nodes

[pmdtest@hpc bin]$ which mpirun
/share/apps/openmpi-1.8.4_gcc-4.9.2/bin/mpirun
[pmdtest@hpc bin]$ ssh compute-0-0
Last login: Sat Feb 28 02:15:50 2015 from hpc.local
Rocks Compute Node
Rocks 6.1.1 (Sand Boa)
Profile built 01:53 28-Feb-2015
Kickstarted 01:59 28-Feb-2015
[pmdtest@compute-0-0 ~]$ which mpirun
/share/apps/openmpi-1.8.4_gcc-4.9.2/bin/mpirun

The only this I notice important is that in the error it is referring to
 /export/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:

While it should have shown
/share/apps/openmpi-1.8.4_gcc-4.9.2/share/openmpi/help-opal-runtime.txt:
which is the path compute nodes see.

Please help!
Ahsan
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/02/26411.php





Re: [OMPI users] How to handle strides in MPI_Create_type_subarray - Re: MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

2015-01-16 Thread Gus Correa

Hi George

Many thanks for your answer and interest in my questions.
... so ... more questions inline ...

On 01/16/2015 03:41 PM, George Bosilca wrote:

Gus,

Please see my answers inline.


On Jan 16, 2015, at 14:24 , Gus Correa <g...@ldeo.columbia.edu> wrote:

Hi George

It is still not clear to me how to deal with strides in 
MPI_Create_type_subarray.
The function/subroutine interface doesn’t mention strides at all.


That’s indeed a little tricky.
However, the trick here is that when you try to understand the subarray
type you should think recursively about the datatypes involved in the operation.


It is a pity that there is little literature (books) about MPI,
and the existing books are lagging behind the new MPI developments and 
standards (MPI-2, MPI-3).
My most reliable sources so far were the "MPI - The complete reference" books, 
vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2).
However, they do not even mention MPI_Create_type_subarray,
which is part of the MPI-2 standard.


Let me do a wild guess: you two guys must be the firsts to ask questions about 
it …



Did anybody but the MPI developers actually *used* MPI_Create_type_subarray?
Could this explain the scarcity of questions about it?  :)


I found it in the MPI-2 standard on the web, but it is not clear to me
how to achieve the same effect of strides that are available in MPI_Type_vector.
MPI_Create_type_subarray is in section 4.1.3.
The OMPI MPI_Create_type_subarray man page says there is an example in section 
9.9.2 of the MPI-2 standard.
However, there is no section 9.9.2.
Chapter 9 is about something else ("The info object"), not derived types.
No good example of MPI_Create_type_subarray in section 4.1.3 either,
which is written in the typical terse and hermetic style in which
standards are.


No comments on subjective topics … ;)


It flows almost as smoothly as Backus-Naur prose.
Makes a great reading with some Mozart in the background.


You just blew my day away, I was totally under the impression that the
MPI standard reads like a children’s bedtime story book !!!




Did you write it?
Do you read it for your kids at bed time?
Do they fall asleep right away?

Oh, if AEsop, Grimm Brothers, Charles Perrault, Andersen could only have 
helped as copy-desks ...




So, how can one handle strides in MPI_Create_type_subarray?
Would one have to first create several MPI_Type_vector for the various dimensions, then 
use them as "oldtype" in  MPI_Create_type_subarray?
That sounds awkward, because there is only one “oldtype" in 
MPI_Create_type_subarray, not one for each dimension.


Exactly. Take a look at how we handle the subarray in Open MPI,
more precisely at the file ompi/datatype/ompi_datatype_create_subarray.c.
My comment from few days ago referred exactly to this code, where the
subarray is basically described in terms of vector
(well maybe vector as I was lazy to check the LB/UB).



When documentation equates to reading actual function code ...
... that is when users drop trying to use new developments ...

BTW, ominously a bug report on LB/UB misuse in MPI_Type_struct
*and* in  MPI_Type_create_subarray ... gosh ...

http://lists.mpich.org/pipermail/discuss/2015-January/003608.html

But hopefully that doesn't affect Open MPI, right?


As I said above think recursively.
You start with the old type,
then build another try on a dimension,
then you use this to expose the second dimensions and so on.
For each dimension your basic type is not the user provided old type,
but the type you built so far.

- size_array[i] is the number of elements in big data in the dimension i
- subsize_array[i] is the of element you will include in your datatype in the 
dimension i
- start_array[i] is how many elements you will skip in the dimension i before 
you start including them in your datatype. start[i] + subside[i] must be 
smaller or equal to size[i]



OK, that starts to make more sense than the yawning bedtime story
in the MPI-2 standard.

I should peel off (that would be recursive)
or build up (that would be non-recursive, right?) each dimension,
one by one,
like an onion,
creating new subarrays of increasing dimensions,
one by one,
based on the subarray previously created.
Did I get it right?
So, should I peel off or build up the dimensions?

In which regard is this any better than using MPI_Type_Vector,
which can be setup in a single non-recursive call,
as long as the sizes, strides, etc, are properly calculated?


Is there any simple example of how to achieve  stride effect with
MPI_Create_type_subarray in a multi-dimensional array?


Not as far as I know.
But now that people expressed interest in this topic,
maybe someone will write a blog or something about.



An example, just a simple example ...
... to help those that have to write all steps from 1 to N,
when it comes to thinking recursively ...
When it comes to recursion, I stopped at the Fibonacci numbers.

Well, even if it is on a

[OMPI users] How to handle strides in MPI_Create_type_subarray - Re: MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

2015-01-16 Thread Gus Correa

Hi George

It is still not clear to me how to deal with strides in 
MPI_Create_type_subarray.

The function/subroutine interface doesn't mention strides at all.

It is a pity that there is little literature (books) about MPI,
and the existing books are lagging behind the new MPI developments and 
standards (MPI-2, MPI-3).
My most reliable sources so far were the "MPI - The complete reference" 
books, vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2).

However, they do not even mention MPI_Create_type_subarray,
which is part of the MPI-2 standard.

I found it in the MPI-2 standard on the web, but it is not clear to me
how to achieve the same effect of strides that are available in 
MPI_Type_vector.

MPI_Create_type_subarray is in section 4.1.3.
The OMPI MPI_Create_type_subarray man page says there is an example in 
section 9.9.2 of the MPI-2 standard.

However, there is no section 9.9.2.
Chapter 9 is about something else ("The info object"), not derived types.
No good example of MPI_Create_type_subarray in section 4.1.3 either,
which is written in the typical terse and hermetic style in which
standards are.

So, how can one handle strides in MPI_Create_type_subarray?
Would one have to first create several MPI_Type_vector for the various 
dimensions, then use them as "oldtype" in  MPI_Create_type_subarray?
That sounds awkward, because there is only one "oldtype" in 
MPI_Create_type_subarray, not one for each dimension.


Is there any simple example of how to achieve  stride effect with
MPI_Create_type_subarray in a multi-dimensional array?

BTW, when are you gentlemen going to write an updated version of the
"MPI - The Complete Reference"?  :)

Thank you,
Gus Correa
(Hijacking Diego Avesani's thread, apologies to Diego.)
(Also, I know this question is not about Open MPI, but about MPI in 
general.  But the lack of examples may warrant asking the question here.)



On 01/16/2015 01:39 AM, George Bosilca wrote:

  The subarray creation is an multi-dimension extension of the vector type.
You can see it as a vector of vector of vector and so on, one vector per 
dimension.
The stride array is used to declare on each dimension what is the 
relative displacement

(in number of elements) from the beginning of the dimension array.


It is important to use regular type creation when you can take advantage of such

regularity instead of resorting to use of struct or h*. This insure better
packing/unpacking performance, as well as possible future support for 
one-sided

communications.


George.




On Jan 15, 2015, at 19:31, Gus Correa <g...@ldeo.columbia.edu> wrote:

I never used MPI_Type_create_subarray, only MPI_Type_Vector.
What I like about MPI_Type_Vector is that you can define a stride,
hence you can address any regular pattern in memory.
However, it envisages the array layout in memory as a big 1-D array,
with a linear index progressing in either Fortran or C order.

Somebody correct me please if I am wrong, but at first sight MPI_Type_Vector 
sounds more flexible to me than MPI_Type_create_subarray, exactly because the 
latter doesn't have strides.

The downside is that you need to do some index arithmetic to figure
the right strides, etc, to match the corresponding
Fortran90 array sections.

There are good examples in the "MPI - The complete reference" books I suggested 
to you before (actually in vol 1).

Online I could find the two man pages (good information, but no example):

http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php

There is a very simple 2D example of MPI_Type_vector using strides here:

https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types

and a similar one here:

http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html

Gus Correa


On 01/15/2015 06:53 PM, Diego Avesani wrote:
dear George, dear Gus, dear all,
Could you please tell me where I can find a good example?
I am sorry but I can not understand the 3D array.


Really Thanks

Diego


On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu
<mailto:bosi...@icl.utk.edu>> wrote:



On Jan 15, 2015, at 06:02 , Diego Avesani <diego.aves...@gmail.com
<mailto:diego.aves...@gmail.com>> wrote:

Dear Gus, Dear all,
Thanks a lot.
MPI_Type_Struct works well for the first part of my problem, so I
am very happy to be able to use it.

Regarding MPI_TYPE_VECTOR.

I have studied it and for simple case it is clear to me what id
does (at least I believe). Foe example if I have a matrix define as:
REAL, ALLOCATABLE (AA(:,:))
ALLOCATE AA(100,5)

I could send part of it defining

CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)

after that I can send part of it with

CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
MPI_COMM_WORLD );

Have I understood correctly?

   

Re: [OMPI users] MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

2015-01-15 Thread Gus Correa

I never used MPI_Type_create_subarray, only MPI_Type_Vector.
What I like about MPI_Type_Vector is that you can define a stride,
hence you can address any regular pattern in memory.
However, it envisages the array layout in memory as a big 1-D array,
with a linear index progressing in either Fortran or C order.

Somebody correct me please if I am wrong, but at first sight 
MPI_Type_Vector sounds more flexible to me than 
MPI_Type_create_subarray, exactly because the latter doesn't have strides.


The downside is that you need to do some index arithmetic to figure
the right strides, etc, to match the corresponding
Fortran90 array sections.

There are good examples in the "MPI - The complete reference" books I 
suggested to you before (actually in vol 1).


Online I could find the two man pages (good information, but no example):

http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php

There is a very simple 2D example of MPI_Type_vector using strides here:

https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types

and a similar one here:

http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html

Gus Correa

On 01/15/2015 06:53 PM, Diego Avesani wrote:

dear George, dear Gus, dear all,
Could you please tell me where I can find a good example?
I am sorry but I can not understand the 3D array.


Really Thanks

Diego


On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu
<mailto:bosi...@icl.utk.edu>> wrote:



On Jan 15, 2015, at 06:02 , Diego Avesani <diego.aves...@gmail.com
<mailto:diego.aves...@gmail.com>> wrote:

Dear Gus, Dear all,
Thanks a lot.
MPI_Type_Struct works well for the first part of my problem, so I
am very happy to be able to use it.

Regarding MPI_TYPE_VECTOR.

I have studied it and for simple case it is clear to me what id
does (at least I believe). Foe example if I have a matrix define as:
REAL, ALLOCATABLE (AA(:,:))
ALLOCATE AA(100,5)

I could send part of it defining

CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)

after that I can send part of it with

CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
MPI_COMM_WORLD );

Have I understood correctly?

What I can do in case of three dimensional array? for example
AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY.
Is that the correct way?

Thanks again


Indeed, using the subarray is the right approach independent on the
number of dimensions of the data (you can use it instead of
MPI_TYPE_VECTOR as well).

   George.







Diego


    On 13 January 2015 at 19:04, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:

Hi Diego
I guess MPI_Type_Vector is the natural way to send and receive
Fortran90 array sections (e.g. your QQMLS(:,50:100,:)).
I used that before and it works just fine.
I think that is pretty standard MPI programming style.
I guess MPI_Type_Struct tries to emulate Fortran90 and C
structures
(as you did in your previous code, with all the surprises
regarding alignment, etc), not array sections.
Also, MPI type vector should be more easy going (and probably
more efficient) than MPI type struct, with less memory
alignment problems.
I hope this helps,
Gus Correa

PS - These books have a quite complete description and several
examples
of all MPI objects and functions, including MPI types (native
and user defined):
http://mitpress.mit.edu/books/__mpi-complete-reference-0
<http://mitpress.mit.edu/books/mpi-complete-reference-0>
http://mitpress.mit.edu/books/__mpi-complete-reference-1
<http://mitpress.mit.edu/books/mpi-complete-reference-1>

[They cover MPI 1 and 2. I guess there is a new/upcoming book
with MPI 3, but for what you're doing 1 and 2 are more than
enough.]


On 01/13/2015 09:22 AM, Diego Avesani wrote:

Dear all,

I had some wonderful talking about MPI_type_create_struct adn
isend\irecv with
Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is
more clear and my
program works.

Now I have another question. In may program I have matrix:

/QQMLS(:,:,:) /that is allocate as

/ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of
particles

QQMLS is double precision.

I would like to sent form a CPU to another part of it, for
example,
sending QQMLS(:,50:100,:). I mean sending the QQMLS of the
particles
between 50 to 100.
I suppose that i could use MPI_Type_vector but I am not
sure. The
 

Re: [OMPI users] MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

2015-01-13 Thread Gus Correa

Hi Diego
I guess MPI_Type_Vector is the natural way to send and receive Fortran90 
array sections (e.g. your QQMLS(:,50:100,:)).

I used that before and it works just fine.
I think that is pretty standard MPI programming style.
I guess MPI_Type_Struct tries to emulate Fortran90 and C structures
(as you did in your previous code, with all the surprises regarding 
alignment, etc), not array sections.
Also, MPI type vector should be more easy going (and probably more 
efficient) than MPI type struct, with less memory alignment problems.

I hope this helps,
Gus Correa

PS - These books have a quite complete description and several examples
of all MPI objects and functions, including MPI types (native and user 
defined):

http://mitpress.mit.edu/books/mpi-complete-reference-0
http://mitpress.mit.edu/books/mpi-complete-reference-1

[They cover MPI 1 and 2. I guess there is a new/upcoming book
with MPI 3, but for what you're doing 1 and 2 are more than enough.]


On 01/13/2015 09:22 AM, Diego Avesani wrote:

Dear all,

I had some wonderful talking about MPI_type_create_struct adn
isend\irecv with
Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is more clear and my
program works.

Now I have another question. In may program I have matrix:

/QQMLS(:,:,:) /that is allocate as

/ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of particles

QQMLS is double precision.

I would like to sent form a CPU to another part of it, for example,
sending QQMLS(:,50:100,:). I mean sending the QQMLS of the particles
between 50 to 100.
I suppose that i could use MPI_Type_vector but I am not sure. The
particle that I want to sent could be from 25 to 50 ecc.. ecc..so
  blocklength changes everytime.

Do I have to use MPI_type_create_struct?
Do I have correctly understood MPI_Type_vector?

Thanks a lot


Diego



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26171.php





Re: [OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

2015-01-08 Thread Gus Correa

Hi Diego

*EITHER*
declare your QQ and PR (?) structure components as DOUBLE PRECISION
*OR*
keep them REAL(dp) but *fix* your "dp" definition, as George Bosilca 
suggested.


Gus Correa

On 01/08/2015 06:36 PM, Diego Avesani wrote:

Dear Gus, Dear All,
so are you suggesting to use DOUBLE PRECISION and not REAL(dp)?
Thanks again

Diego


On 9 January 2015 at 00:02, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:

On 01/08/2015 05:50 PM, Diego Avesani wrote:

Dear George, Dear all,
what are the other issues?

Why did you put in selected_real_kind(15, 307) the number 307


Hi Diego

That is the Fortran 90 (and later) syntax for selected_real_kind.
The first number is the number of digits in the mantissa,
the second is the exponent range.
For (simpler) alternatives, see here:

http://fortranwiki.org/__fortran/show/Real+precision
<http://fortranwiki.org/fortran/show/Real+precision>

A lot of grief (and probably the segfault)
could have been saved if you just used
"DOUBLE PRECISION", instead of REAL in your
structure components declaration, as I suggested a while back.

I hope this helps,
Gus Correa


Thanks again

Diego


On 8 January 2015 at 23:24, George Bosilca <bosi...@icl.utk.edu
<mailto:bosi...@icl.utk.edu>
<mailto:bosi...@icl.utk.edu <mailto:bosi...@icl.utk.edu>>> wrote:

 Diego,

 Please find below the corrected example. There were several
issues
 but the most important one, which is certainly the cause of the
 segfault, is that "real(dp)" (with dp =
selected_real_kind(p=16)) is
 NOT equal to MPI_DOUBLE_RECISION. For double precision you
should
 use 15 (and not 16).

George.


 On Thu, Jan 8, 2015 at 6:08 AM, Jeff Squyres (jsquyres)
 <jsquy...@cisco.com <mailto:jsquy...@cisco.com>
<mailto:jsquy...@cisco.com <mailto:jsquy...@cisco.com>>> wrote:

 There were still some minor errors left over in the
attached
 program.

 I strongly encourage you to use "use mpi" instead of
"include
 'mpif.h'" because you will get compile time errors when
you pass
 incorrect / forget to pass parameters to MPI
subroutines.  When
 I switched your program to "use mpi", here's what the
compiler
 showed me:

 1. the name "MPI" is reserved
 2. need to pass displacements(1:nstruct+1) to
mpi_type_create_struct
 3. need to pass request(1) to mpi_isend
 4. need to pass request(1) to mpi_wait
 5. need to pass ierr argument to mpi_wait

 1-4 are technically not correct, but the program would
likely
 (usually) compile/run "correctly" anyway.  5 is
probably what
 caused your segv.

 Attached is my copy of your program with fixes for the
 above-mentioned issues.

 BTW, I missed the beginning of this thread -- I assume
that this
 is an artificial use of mpi_type_create_resized for the
purposes
 of a small example.  The specific use of it in this program
 appears to be superfluous.





 On Jan 8, 2015, at 4:26 AM, Gilles Gouaillardet
 <gilles.gouaillar...@iferc.org
<mailto:gilles.gouaillar...@iferc.org>
 <mailto:gilles.gouaillardet@__iferc.org
<mailto:gilles.gouaillar...@iferc.org>>> wrote:

 > Diego,
 >
 > yes, it works for me (at least with the v1.8 head and
gnu compilers)
 >
 > Cheers,
 >
 > Gilles
 >
 > On 2015/01/08 17:51, Diego Avesani wrote:
 >> Dear Gilles,
 >> thanks again, however it does not work.
 >>
 >> the program says:  "SIGSEGV, segmentation fault
occurred"
 >>
 >> Does the program run in your case?
 >>
 >> Thanks again
 >>
 >>
 >>
 >> Diego
 >>
 >>
 >> On 8 January 2015 at 03:02, Gilles Gouaillardet <
 >>
 >>gilles.gouaillardet@ifer

Re: [OMPI users] MPI_Type_Create_Struct + MPI_TYPE_CREATE_RESIZED

2015-01-08 Thread Gus Correa

On 01/08/2015 05:50 PM, Diego Avesani wrote:

Dear George, Dear all,
what are the other issues?

Why did you put in selected_real_kind(15, 307) the number 307



Hi Diego

That is the Fortran 90 (and later) syntax for selected_real_kind.
The first number is the number of digits in the mantissa,
the second is the exponent range.
For (simpler) alternatives, see here:

http://fortranwiki.org/fortran/show/Real+precision

A lot of grief (and probably the segfault)
could have been saved if you just used
"DOUBLE PRECISION", instead of REAL in your
structure components declaration, as I suggested a while back.

I hope this helps,
Gus Correa



Thanks again

Diego


On 8 January 2015 at 23:24, George Bosilca <bosi...@icl.utk.edu
<mailto:bosi...@icl.utk.edu>> wrote:

Diego,

Please find below the corrected example. There were several issues
but the most important one, which is certainly the cause of the
segfault, is that "real(dp)" (with dp = selected_real_kind(p=16)) is
NOT equal to MPI_DOUBLE_RECISION. For double precision you should
use 15 (and not 16).

   George.


On Thu, Jan 8, 2015 at 6:08 AM, Jeff Squyres (jsquyres)
<jsquy...@cisco.com <mailto:jsquy...@cisco.com>> wrote:

There were still some minor errors left over in the attached
program.

I strongly encourage you to use "use mpi" instead of "include
'mpif.h'" because you will get compile time errors when you pass
incorrect / forget to pass parameters to MPI subroutines.  When
I switched your program to "use mpi", here's what the compiler
showed me:

1. the name "MPI" is reserved
2. need to pass displacements(1:nstruct+1) to mpi_type_create_struct
3. need to pass request(1) to mpi_isend
4. need to pass request(1) to mpi_wait
5. need to pass ierr argument to mpi_wait

1-4 are technically not correct, but the program would likely
(usually) compile/run "correctly" anyway.  5 is probably what
caused your segv.

Attached is my copy of your program with fixes for the
above-mentioned issues.

BTW, I missed the beginning of this thread -- I assume that this
is an artificial use of mpi_type_create_resized for the purposes
of a small example.  The specific use of it in this program
appears to be superfluous.





On Jan 8, 2015, at 4:26 AM, Gilles Gouaillardet
<gilles.gouaillar...@iferc.org
<mailto:gilles.gouaillar...@iferc.org>> wrote:

> Diego,
>
> yes, it works for me (at least with the v1.8 head and gnu compilers)
>
> Cheers,
>
> Gilles
>
> On 2015/01/08 17:51, Diego Avesani wrote:
>> Dear Gilles,
>> thanks again, however it does not work.
>>
>> the program says:  "SIGSEGV, segmentation fault occurred"
>>
>> Does the program run in your case?
>>
>> Thanks again
>>
>>
>>
>> Diego
>>
>>
>> On 8 January 2015 at 03:02, Gilles Gouaillardet <
>>
>>gilles.gouaillar...@iferc.org <mailto:gilles.gouaillar...@iferc.org>
>> > wrote:
>>
>>
>>>  Diego,
>>>
>>> my bad, i should have passed displacements(1) to 
MPI_Type_create_struct
>>>
>>> here is an updated version
>>>
>>> (note you have to use a REQUEST integer for MPI_Isend and MPI_Irecv,
>>> and you also have to call MPI_Wait to ensure the requests complete)
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On 2015/01/08 8:23, Diego Avesani wrote:
>>>
>>> Dear Gilles, Dear all,
>>>
>>> I'm sorry to bother you again, but I have some problem with send and
>>> receive the struct_data.
>>>
>>> I tried to send a MPI_Type_Create_Struct but I get a segmentation 
fault
>>> occurred and I do not know why. The program is very simple, it is 
the old
>>> one with the isend and irecv subroutines
>>>
>>> (you can find it in the attachment)
>>>
>>> Thanks again
>>>
>>>
>>> Diego
>>>
>>>
>>> On 5 January 2015 at 15:54, Di

Re: [OMPI users] libpsm_infinipath issues?

2015-01-08 Thread Gus Correa

Hi Michael, Andrew, list

knem is doesn't work in OMPI 1.8.3.
See this thread:
http://www.open-mpi.org/community/lists/users/2014/10/25511.php

A fix was promised on OMPI 1.8.4:
http://www.open-mpi.org/software/ompi/v1.8/

Have you tried it?

I hope this helps,
Gus Correa

On 01/08/2015 04:36 PM, Friedley, Andrew wrote:

Hi Mike,

Have you contacted your admins, or the vendor that provided your True
Scale and/or PSM installation? E.g. Redhat, or Intel via
ibsupp...@intel.com <mailto:ibsupp...@intel.com>?  They are normally the
recommended path for True Scale support.

That said, here’s some things to look into:

I think it’s strange that you’re hitting PSM linker problems while
compiling an app with Open MPI.  Normally only the PSM MTL is linked
directly against PSM.  Try verifying that nothing else is linking in
PSM. Also, it’s possible your OMPI build is different from normal too.

PSM has optional KNEM support that can be compiled in, but it’s off by
default and I’ve never heard of it being used.  It’s possible that your
PSM was changed or recompiled and part of your cluster upgrade, and the
installation isn’t quite right.  It looks like your PSM was compiled
with KNEM support, but the KNEM library isn’t being linked in or is not
being found.

Andrew

*From:* users [mailto:users-boun...@open-mpi.org] *On Behalf Of
*VanEss.Michael
*Sent:* Thursday, January 8, 2015 1:07 PM
*To:* us...@open-mpi.org
*Subject:* [OMPI users] libpsm_infinipath issues?

Hello all,

Our clusters were just upgraded to both a new version of PGI (14.9) as
well as openmpi (1.8.3).  Previous versions were 12.1 and 1.6
respectively, and those compiled and linked just fine.  The newest
versions are not linking my mpi applications at all.  Here’s the problem:

/opt/scyld/openmpi/1.8.3/pgi/bin/mpif90 -C chemcode_mpi.o mod_mpichem.o
plume_mpi.o amb2D_mpi.o fex.o jex.o use_tuv.o run_lsode.o mod_amb.o
utmgeo.o lsode.o newamb.o plume.o amb2D.o solve.o mod_cdf.o calc_rates.o
mod_tuv.o flux.o amb1D.o amb_com.o newamb2D.o vs_ccode.o ck_errors.o
newamb1D.o doChem.o mod_lsode.o stode.o plume_com.o typeSizes.o netcdf.o
mod_parameters.o mod_chem.o runfile.o com_codes.o mod_SLAM.o mod_CPUFF.o
calc_za.o mod_releases.o mod_site.o mod_distance.o nuclear_dates.o
mod_luparms.o deposition.o diffusion.o getTJ.o mod_met.o met_data.o
mod_h5tuv.o tuv10.o mod_h5camx.o cmxamb.o \

 -L/ensco/apps/cm/CMSOL/linux/zlib-1.2.1/lib
-L/ensco/apps/cm/CMSOL/linux/szip-2.1/lib -o \

 chemcode_mpi  -L/opt/pgi/linux86-64/14.9/lib -lpgf90 -lpgf90rtl \

 -L/ensco/apps/cm/CMSOL/linux/hdf5-1.8.3/lib -lhdf5_fortran -l
hdf5 -lz -lm

/usr/lib64/libpsm_infinipath.so.1: undefined reference to `knem_put'

/usr/lib64/libpsm_infinipath.so.1: undefined reference to `knem_open_device'

/usr/lib64/libpsm_infinipath.so.1: undefined reference to `knem_get'

/usr/lib64/libpsm_infinipath.so.1: undefined reference to
`knem_register_region'

I’ve searched the net for any information on this and nothing has seemed
to help.  I’m fairly confident that all my variables and paths to the
new software is correct.

Any ideas?

Regards,

Mike



The information contained in this email message is intended only for the
use of the individual(s) to whom it is addressed and may contain
information that is privileged and sensitive. If you are not the
intended recipient, or otherwise have received this communication in
error, please notify the sender immediately by email at the above
referenced address and note that any further dissemination, distribution
or copying of this communication is strictly prohibited.

The U.S. Export Control Laws regulate the export and re-export of
technology originating in the United States. This includes the
electronic transmission of information and software to foreign countries
and to certain foreign nationals. Recipient agrees to abide by these
laws and their regulations -- including the U.S. Department of Commerce
Export Administration Regulations and the U.S. Department of State
International Traffic in Arms Regulations -- and not to transfer, by
electronic transmission or otherwise, any content derived from this
email to either a foreign national or a foreign destination in violation
of such laws.



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26135.php





Re: [OMPI users] Icreasing OFED registerable memory

2015-01-06 Thread Gus Correa

Hi Waleed

As Devendar said (and I tried to explain before),
you need to allow the locked memory limit to be unlimited for
user processes (in /etc/security/limits.conf),
*AND* somehow the daemon/job_script/whatever that launches the mpiexec
command must request "ulimit -l unlimited" (directly or indirectly).
The latter part depends on how your system's details.
I am not familiar to SGE (I use Torque), but presumably you can
add "ulimit -l unlimited" when you launch
the SGE daemons on the nodes.
Presumably this will make the processes launched by that daemon
(i.e. your mpiexec) inherit those limits,
and that is how I do it on Torque.
A more brute force way is just to include "ulimit -l unlimited"
in you job script before mpiexec.
Inserting a "ulimit -a" in your jobscript may help diagnose what you
actually have.
Please, see the OMPI FAQ that I sent you before for more details.

I hope this helps,
Gus Correa

On 01/06/2015 01:37 PM, Deva wrote:

Hi Waleed,

--
   Memlock limit: 65536
--

such a low limit should be due to per-user lock memory limit . Can you
make sure it is  set to "unlimited" on all nodes ( "ulimit -l unlimited")?

-Devendar

On Tue, Jan 6, 2015 at 3:42 AM, Waleed Lotfy <waleed.lo...@bibalex.org
<mailto:waleed.lo...@bibalex.org>> wrote:

Hi guys,

Sorry for getting back so late, but we ran into some problems during
the installation process and as soon as the system came up I tested
the new versions for the problem but it showed another memory
related warning.

--
The OpenFabrics (openib) BTL failed to initialize while trying to
allocate some locked memory.  This typically can indicate that the
memlock limits are set too low.  For most HPC installations, the
memlock limits should be set to "unlimited".  The failure occured
here:

   Local host:comp003.local
   OMPI source:   btl_openib_component.c:1200
   Function:  ompi_free_list_init_ex_new()
   Device:mlx4_0
   Memlock limit: 65536

You may need to consult with your system administrator to get this
problem fixed.  This FAQ entry on the Open MPI web site may also be
helpful:

http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
--
--
WARNING: There was an error initializing an OpenFabrics device.

   Local host:   comp003.local
   Local device: mlx4_0
--

<<>>

My current running versions:

OpenMPI: 1.6.4
OFED-internal-2.3-2

I checked /etc/security/limits.d/, the scheduler's configurations
(grid engine) and tried adding the following line to
/etc/modprobe.d/mlx4_core: 'options mlx4_core log_num_mtt=22
log_mtts_per_seg=1' as suggested by Gus.

I am running out of ideas here, so please any help is appreciated.

P.S. I am not sure if I should open a new thread with this issue or
continue with the current one, so please advice.

Waleed Lotfy
Bibliotheca Alexandrina
___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/01/26107.php




--


-Devendar


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26109.php





Re: [OMPI users] Icreasing OFED registerable memory

2014-12-30 Thread Gus Correa

Hi Waleed

Even before any OFED upgrades, you could try the items
in the list below.
I have OMPI 1.6.5 and 1.8.3 working with an older OFED version,
with those settings.
That is not really OMPI fault, but Infinband/OFED's.

1) Make sure your locked memory is set to unlimited in
/etc/security/limits.conf

For instance:

*   softmemlock unlimited
*   hardmemlock unlimited


2) If you are using a queue system, make sure it sets the
locked memory to unlimited, so that all child processes
(including your mpiexec and mpi executable) will get it.

For instance, in Torque /etc/init.d/pbs_mom
or in /etc/sysconfig/pbs_mom:

# locked memory
ulimit -l unlimited

3) Add the parameters below to
/etc/modprobe.d/mlx4_core.conf

options mlx4_core log_num_mtt=22 log_mtts_per_seg=1

Do this with care, as the settings vary according to the physical RAM.
In addition,  the parameters seem to have been deprecated in 3.X 
kernels, which makes this tricky.


See these FAQs:

http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-user
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-more
http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem

***
Having said that, a question remains unanswered:
Why is Infiniband such a nightmare?
***

I hope this helps,
Gus Correa

On 12/30/2014 09:16 AM, Waleed Lotfy wrote:

Thank Devendar for your response.

I'll test it on a new installation with OFED 2.3.2 and OMPI v1.6.5. If it 
didn't work I'll give 1.8.4 a try.

Thank you for your help and I'll get back to you with hopefully good results.

Waleed Lotfy
Bibliotheca Alexandrina

From: users [users-boun...@open-mpi.org] on behalf of Deva 
[devendar.bure...@gmail.com]
Sent: Monday, December 29, 2014 8:29 PM
To: Open MPI Users
Subject: Re: [OMPI users] Icreasing OFED registerable memory

Hi Waleed,

It is highly recommended to upgrade to latest OFED.  Meanwhile, Can you try 
latest OMPI release (v1.8.4), where this warning is ignored on older OFEDs

-Devendar

On Sun, Dec 28, 2014 at 6:03 AM, Waleed Lotfy 
<waleed.lo...@bibalex.org<mailto:waleed.lo...@bibalex.org>> wrote:
I have a bunch of 8 GB memory nodes in a cluster who were lately
upgraded to 16 GB. When I run any jobs I get the following warning:
--
WARNING: It appears that your OpenFabrics subsystem is configured to
only
allow registering part of your physical memory.  This can cause MPI jobs
to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel
module
parameters:

 http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

   Local host:  comp022.local
   Registerable memory: 8192 MiB
   Total memory:16036 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--

Searching for a fix to this issue, I found that I have to set
log_num_mtt within the kernel module, so I added this line to
modprobe.conf:

options mlx4_core log_num_mtt=21

But then ib0 interface fails to start showing this error:
ib_ipoib device ib0 does not seem to be present, delaying
initialization.

Reducing the value of log_num_mtt to 20, allows ib0 to start but shows
the registerable memory of 8 GB warning.

I am using OFED 1.3.1, I know it is pretty old and we are planning to
upgrade soon.

Output on all nodes for 'ompi_info  -v ompi full --parsable':

ompi:version:full:1.2.7
ompi:version:svn:r19401
orte:version:full:1.2.7
orte:version:svn:r19401
opal:version:full:1.2.7
opal:version:svn:r19401

Any help would be appreciated.

Waleed Lotfy
Bibliotheca Alexandrina
___
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/12/26076.php



--


-Devendar
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/12/26088.php





Re: [OMPI users] Open mpi based program runs as root and gives SIGSEGV under unprivileged user

2014-12-10 Thread Gus Correa

Hi Luca

Another possibility that comes to mind,
besides mixed versions mentioned by Gilles,
is the OS limits.
Limits may vary according to the user and user privileges.

Large programs tend to require big stacksize (even unlimited),
and typically segfault when the stack is not large enough.
Max number of open files is yet another hurdle.
And if you're using Infinband, the max locked memory size should be 
unlimited.

Check /etc/security/limits.conf and "ulimit -a".

I hope this helps,
Gus Correa

On 12/10/2014 08:28 AM, Gilles Gouaillardet wrote:

Luca,

your email mentions openmpi 1.6.5
but gdb output points to openmpi 1.8.1.

could the root cause be a mix of versions that does not occur with root
account ?

which openmpi version are you expecting ?

you can run
pmap 
when your binary is running and/or under gdb to confirm the openmpi
library that is really used

Cheers,

Gilles

On Wed, Dec 10, 2014 at 7:21 PM, Luca Fini <lf...@arcetri.astro.it
<mailto:lf...@arcetri.astro.it>> wrote:

I've a problem running a well tested MPI based application.

The program has been used for years with no problems. Suddenly the
executable which was run many times with no problems crashed with
SIGSEGV. The very same executable if run with root privileges works
OK. The same happens with other executables and across various
recompilation attempts.

We could not find any relevant difference in the O.S. since a few days
ago when the program worked also under unprivileged user ID. Actually
about in the same span of time we changed the GID of the user
experiencing the fault, but we think this is not relevant because the
same SIGSEGV happens to another user which was not modified. Moreover
we cannot see how that change can affect the running executabe (we
checked all file permissions in the directory tree where the program
is used).

Running the program under GDB we get the trace reported below. The
segfault happens at the very beginning during MPI initialization.

We can use the program with sudo, but I'd like to find out what
happened to go back to "normal" usage.

I'd appreciate any hint on the issue.

Many thanks,

Luca Fini

==
Here follows a few environment details:

Program started with: mpirun -debug -debugger gdb  -np 1

/home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD

OPEN-MPI 1.6.5

Linux 2.6.32-431.29.2.2.6.32-431.29.2.el6.x86_64

Intel fortran Compiler: 2011.7.256

=
Here follows the stack trace:

Starting program:

/home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD

/home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/M51b2_OT_2POINT_RH_v1_mod/PREP_PGD
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0x2af652c7 in mca_base_component_find (directory=0x0,
type=0x3b914a7fb5 "rte", static_components=0x3b916cb040,
requested_component_names=0x0, include_mode=128, found_components=0x1,
open_dso_components=16)
 at mca_base_component_find.c:162
162OBJ_CONSTRUCT(found_components, opal_list_t);
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.149.el6.x86_64 libgcc-4.4.7-11.el6.x86_64
libgfortran-4.4.7-11.el6.x86_64 libtool-ltdl-2.2.6-15.5.el6.x86_64
openmpi-1.8.1-1.el6.x86_64
(gdb) where
#0  0x2af652c7 in mca_base_component_find (directory=0x0,
type=0x3b914a7fb5 "rte", static_components=0x3b916cb040,
requested_component_names=0x0, include_mode=128, found_components=0x1,
open_dso_components=16)
 at mca_base_component_find.c:162
#1  0x003b90c4870a in mca_base_framework_components_register ()
from /usr/lib64/openmpi/lib/libopen-pal.so.6
#2  0x003b90c48c06 in mca_base_framework_register () from
/usr/lib64/openmpi/lib/libopen-pal.so.6
#3  0x003b90c48def in mca_base_framework_open () from
/usr/lib64/openmpi/lib/libopen-pal.so.6
#4  0x003b914407e7 in ompi_mpi_init () from
/usr/lib64/openmpi/lib/libmpi.so.1
#5  0x003b91463200 in PMPI_Init () from
/usr/lib64/openmpi/lib/libmpi.so.1
#6  0x2acd9295 in mpi_init_f (ierr=0x7fffd268) at
pinit_f.c:75
#7  0x005bb159 in MODE_MNH_WORLD::init_nmnh_comm_world
(kinfo_ll=Cannot access memory at address 0x0
) at

/home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/MASTER/spll_mode_mnh_world.f90:45
#8  0x005939d3 in MODE_IO_LL::initio_ll () at

/home/lascaux/MNH-V5-1-2/src/dir_obj-LXifortI4-MNH-V5-1-2-OMPI12X-O2/MASTER/spll_mode_io_ll.f90:107
#9  0x0049d02f in prep_pgd () at

/home/lascaux/MNH-V5-1-

Re: [OMPI users] How OMPI picks ethernet interfaces

2014-11-13 Thread Gus Correa

On 11/13/2014 11:14 AM, Ralph Castain wrote:

Hmmm…I’m beginning to grok the issue. It is a tad unusual for people to
assign different hostnames to their interfaces - I’ve seen it in the
Hadoop world, but not in HPC. Still, no law against it.


No, not so unusual.
I have clusters from respectable vendors that come with
/etc/hosts for name resolution of the various interfaces.
If I remember right, Rocks clusters also does that (or actually
allow the sys admin to setup additional networks and at that point
will append /etc/hosts with the additional names, or perhaps put those
names in DHCP).
I am not so familiar to xcat, but I think it has similar DHCP 
functionality, or maybe DNS on the head node.


Having said that, I don't think this is an obstacle to setting up the 
right "if_include/if_exlculde" choices (along with the btl, oob, etc),

for each particular cluster in the mca parameter configuration file.
That is what my parallel conversation with Reuti was about.

I believe the current approach w.r.t. interfaces:
"use everythint, let the sysadmin/user restrict as
(s)he sees fit" is both a wise and flexible way to do it.
Guessing the "right interface to use" sounds risky to me (wrong choices 
may happen), and a bit of a cast.




This will take a little thought to figure out a solution. One problem
that immediately occurs is if someone includes a hostfile that has lines
which refer to the same physical server, but using different interface
names. We’ll think those are completely distinct servers, and so the
process placement will be totally messed up.



Sure, and besides this, there will be machines with
inconsistent/wrong/conflicting name resolution schemes
that the current OMPI approach simply (and wisely) ignores.



We’ll also encounter issues with the daemon when it reports back, as the
hostname it gets will almost certainly differ from the hostname we were
expecting. Not as critical, but need to check to see where that will
impact the code base



I'm sure that will happen.
Torque uses hostname by default for several things, and it can be a 
configuration nightmare to workaround that when what hostname reports is 
not what you want.


IMHO, you may face a daunting guesswork task to get this right,
to pick the
interfaces that are best for a particular computer or cluster.
It is so much easier to let the sysadmin/user, who presumably knows 
his/her machine, to write an MCA parameter config file,

as it is now in OMPI.


We can look at the hostfile changes at that time - no real objection to
them, but would need to figure out how to pass that info to the
appropriate subsystems. I assume you want this to apply to both the oob
and tcp/btl?

Obviously, this won’t make it for 1.8 as it is going to be fairly
intrusive, but we can probably do something for 1.9



The status quo is good.
Long life to the OMPI status quo.
(You don't know how reluctant I am to support the status quo, any status 
quo.  :) )
My vote (... well, I don't have voting rights on that, but I'll vote 
anyway ...) is to keeep the current approach.
It is wise and flexible, and easy to adjust and configure to specific 
machines with their own oddities, via MCA parameters, as I tried to 
explain in previous postings.


My two cents,
Gus Correa




On Nov 13, 2014, at 4:23 AM, Reuti <re...@staff.uni-marburg.de
<mailto:re...@staff.uni-marburg.de>> wrote:

Am 13.11.2014 um 00:34 schrieb Ralph Castain:


On Nov 12, 2014, at 2:45 PM, Reuti <re...@staff.uni-marburg.de
<mailto:re...@staff.uni-marburg.de>> wrote:

Am 12.11.2014 um 17:27 schrieb Reuti:


Am 11.11.2014 um 02:25 schrieb Ralph Castain:


Another thing you can do is (a) ensure you built with
—enable-debug, and then (b) run it with -mca oob_base_verbose 100
 (without the tcp_if_include option) so we can watch the
connection handshake and see what it is doing. The —hetero-nodes
will have not affect here and can be ignored.


Done. It really tries to connect to the outside interface of the
headnode. But being there a firewall or not: the nodes have no clue
how to reach 137.248.0.0 - they have no gateway to this network at all.


I have to revert this. They think that there is a gateway although
it isn't. When I remove the entry by hand for the gateway in the
routing table it starts up instantly too.

While I can do this on my own cluster I still have the 30 seconds
delay on a cluster where I'm not root, while this can be because of
the firewall there. The gateway on this cluster is indeed going to
the outside world.

Personally I find this behavior a little bit too aggressive to use
all interfaces. If you don't check this carefully beforehand and
start a long running application one might even not notice the delay
during the startup.


Agreed - do you have any suggestions on how we should choose the
order in which to try them? I haven’t been able to come up with
anything yet. Jeff has some fancy algo in his usnic BTL that we a

Re: [OMPI users] How OMPI picks ethernet interfaces

2014-11-12 Thread Gus Correa

On 11/12/2014 05:45 PM, Reuti wrote:

Am 12.11.2014 um 17:27 schrieb Reuti:


Am 11.11.2014 um 02:25 schrieb Ralph Castain:


Another thing you can do is (a) ensure you built with —enable-debug,

and then (b) run it with -mca oob_base_verbose 100
(without the tcp_if_include option) so we can watch
the connection handshake and see what it is doing.
The —hetero-nodes will have not affect here and can be ignored.


Done. It really tries to connect to the outside

interface of the headnode. But being there a firewall or not:
the nodes have no clue how to reach 137.248.0.0 -
they have no gateway to this network at all.

I have to revert this.
They think that there is a gateway although it isn't.
When I remove the entry by hand for the gateway in the
routing table it starts up instantly too.

While I can do this on my own cluster I still have the
30 seconds delay on a cluster where I'm not root,
while this can be because of the firewall there.
The gateway on this cluster is indeed going
to the outside world.

Personally I find this behavior a little bit too aggressive
to use all interfaces. If you don't check this carefully
beforehand and start a long running application one might
even not notice the delay during the startup.

-- Reuti



Hi Reuti

You could use the mca parameter file
(say, $prefix/etc/openmpi-mca-params.conf) to configure cluster-wide
the oob (and btl) interfaces to be used.
The users can still override your choices if they want.

Just put a line like this in openmpi-mca-params.conf :
oob_tcp_if_include=192.168.154.0/26

(and similar for btl_tcp_if_include, btl_openib_if_include).

Get a full list from "ompi_info --all --all |grep if_include".

See these FAQ:

http://www.open-mpi.org/faq/?category=tcp#tcp-selection
http://www.open-mpi.org/faq/?category=tuning#setting-mca-params

Compute nodes tend to be multi-homed, so what criterion would OMPI use
to select one interface among many,
not knowing beforehand what exists in a particular computer?
There would be a risk to make a bad choice.
The current approach gives you everything, and you
pick/select/restrict what you want to fit your needs,
with mca parameters (which can be set in several
ways and with various scopes).

I don't think this bad.
However, I am biased about this.
I like and use the openmpi-mca-params.conf file
to setup sensible defaults.
At least I think they are sensible. :)

Cheers,
Gus Correa




It tries so independent from the internal or external name of the headnode

given in the machinefile - I hit ^C then.
I attached the output of Open MPI 1.8.1 for this setup too.


-- Reuti

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/11/25777.php


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/11/25781.php





Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-30 Thread Gus Correa

On 10/30/2014 07:32 PM, Ralph Castain wrote:

Just for FYI: I believe Nathan misspoke.
The new capability is in 1.8.4, which I hope
to release next Friday (Nov 7th)



Hi Ralph

That is even better!
Look forward to OMPI 1.8.4.

I still would love to hear from Nathan / OMPI team
about my remaining questions below
(specially the 12 vader parameters).

Many thanks,
Gus Correa


On Oct 30, 2014, at 4:24 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:

Hi Nathan

Thank you very much for addressing this problem.

I read your notes on Jeff's blog about vader,
and that clarified many things that were obscure to me
when I first started this thread
whining that knem was not working in OMPI 1.8.3.
Thank you also for writing that blog post,
and for sending the link to it.
That was very helpful indeed.

As your closing comments on the blog post point out,
and your IMB benchmark graphs of pingpong/latency &
sendrecv/bandwidth show,
vader+xpmem outperforms the other combinations
of btl+memory_copy_mechanism of intra-node communication.

For the benefit of pedestrian OpenMPI users like me:

1) What is the status of xpmem in the Linux world at this point?
[Proprietary (SGI?) / open source, part of the Linux kernel (which),
part of standard distributions (which) ?]

2) Any recommendation for the values of the
various vader btl parameters?
[There are 12 of them in OMPI 1.8.3!
That is real challenge to get right.]

Which values did you use in your benchmarks?
Defaults?
Other?

In particular, is there an optimal value for the eager/rendevous threshold 
value? (btl_vader_eager_limit, default=4kB)
[The INRIA web site suggests 32kB for the sm+knem counterpart 
(btl_sm_eager_limit, default=4kB).]

3) Did I understand it right, that the upcoming OpenMPI 1.8.5
can be configured with more than one memory copy mechanism altogether
(e.g. --with-knem and --with-cma and --with-xpmem),
then select one of them at runtime with the btl_vader_single_copy_mechanism 
parameter?
Or must OMPI be configured with only one memory copy mechanism?

Many thanks,
Gus Correa


On 10/30/2014 05:44 PM, Nathan Hjelm wrote:

I want to close the loop on this issue. 1.8.5 will address it in several
ways:

  - knem support in btl/sm has been fixed. A sanity check was disabling
knem during component registration. I wrote the sanity check before
the 1.7 release and didn't intend this side-effect.

  - vader now supports xpmem, cma, and knem. The best available
single-copy mechanism will be used. If multiple single-copy
mechanisms are available you can select which one you want to use are
runtime.

More about the vader btl can be found here:
http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/

-Nathan Hjelm
HPC-5, LANL

On Fri, Oct 17, 2014 at 01:02:23PM -0700, Ralph Castain wrote:

  On Oct 17, 2014, at 12:06 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:
  Hi Jeff

  Many thanks for looking into this and filing a bug report at 11:16PM!

  Thanks to Aurelien, Ralph and Nathan for their help and clarifications
  also.

  **

  Related suggestion:

  Add a note to the FAQ explaining that in OMPI 1.8
  the new (default) btl is vader (and what it is).

  It was a real surprise to me.
  If Aurelien Bouteiller didn't tell me about vader,
  I might have never realized it even existed.

  That could be part of one of the already existent FAQs
  explaining how to select the btl.

  **

  Doubts (btl in OMPI 1.8):

  I still don't understand clearly the meaning and scope of vader
  being a "default btl".

We mean that it has a higher priority than the other shared memory
implementation, and so it will be used for intra-node messaging by
default.

  Which is the scope of this default: intra-node btl only perhaps?

Yes - strictly intra-node

  Was there a default btl before vader, and which?

The "sm" btl was the default shared memory transport before vader

  Is vader the intra-node default only (i.e. replaces sm  by default),

Yes

  or does it somehow extend beyond node boundaries, and replaces (or
  brings in) network btls (openib,tcp,etc) ?

Nope - just intra-node

  If I am running on several nodes, and want to use openib, not tcp,
  and, say, use vader, what is the right syntax?

  * nothing (OMPI will figure it out ... but what if you have
  IB,Ethernet,Myrinet,OpenGM, altogether?)

If you have higher-speed connections, we will pick the fastest for
inter-node messaging as the "default" since we expect you would want the
fastest possible transport.

  * -mca btl openib (and vader will come along automatically)

Among the ones you show, this would indeed be the likely choices (openib
and vader)

  * -mca btl openib,self (and vader will come along automatically)

The

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-30 Thread Gus Correa

Hi Nathan

Thank you very much for addressing this problem.

I read your notes on Jeff's blog about vader,
and that clarified many things that were obscure to me
when I first started this thread
whining that knem was not working in OMPI 1.8.3.
Thank you also for writing that blog post,
and for sending the link to it.
That was very helpful indeed.

As your closing comments on the blog post point out,
and your IMB benchmark graphs of pingpong/latency &
sendrecv/bandwidth show,
vader+xpmem outperforms the other combinations
of btl+memory_copy_mechanism of intra-node communication.

For the benefit of pedestrian OpenMPI users like me:

1) What is the status of xpmem in the Linux world at this point?
[Proprietary (SGI?) / open source, part of the Linux kernel (which),
part of standard distributions (which) ?]

2) Any recommendation for the values of the
various vader btl parameters?
[There are 12 of them in OMPI 1.8.3!
That is real challenge to get right.]

Which values did you use in your benchmarks?
Defaults?
Other?

In particular, is there an optimal value for the eager/rendevous 
threshold value? (btl_vader_eager_limit, default=4kB)
[The INRIA web site suggests 32kB for the sm+knem counterpart 
(btl_sm_eager_limit, default=4kB).]


3) Did I understand it right, that the upcoming OpenMPI 1.8.5
can be configured with more than one memory copy mechanism altogether
(e.g. --with-knem and --with-cma and --with-xpmem),
then select one of them at runtime with the 
btl_vader_single_copy_mechanism parameter?

Or must OMPI be configured with only one memory copy mechanism?

Many thanks,
Gus Correa


On 10/30/2014 05:44 PM, Nathan Hjelm wrote:

I want to close the loop on this issue. 1.8.5 will address it in several
ways:

  - knem support in btl/sm has been fixed. A sanity check was disabling
knem during component registration. I wrote the sanity check before
the 1.7 release and didn't intend this side-effect.

  - vader now supports xpmem, cma, and knem. The best available
single-copy mechanism will be used. If multiple single-copy
mechanisms are available you can select which one you want to use are
runtime.

More about the vader btl can be found here:
http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/

-Nathan Hjelm
HPC-5, LANL

On Fri, Oct 17, 2014 at 01:02:23PM -0700, Ralph Castain wrote:

  On Oct 17, 2014, at 12:06 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:
  Hi Jeff

  Many thanks for looking into this and filing a bug report at 11:16PM!

  Thanks to Aurelien, Ralph and Nathan for their help and clarifications
  also.

  **

  Related suggestion:

  Add a note to the FAQ explaining that in OMPI 1.8
  the new (default) btl is vader (and what it is).

  It was a real surprise to me.
  If Aurelien Bouteiller didn't tell me about vader,
  I might have never realized it even existed.

  That could be part of one of the already existent FAQs
  explaining how to select the btl.

  **

  Doubts (btl in OMPI 1.8):

  I still don't understand clearly the meaning and scope of vader
  being a "default btl".

We mean that it has a higher priority than the other shared memory
implementation, and so it will be used for intra-node messaging by
default.

  Which is the scope of this default: intra-node btl only perhaps?

Yes - strictly intra-node

  Was there a default btl before vader, and which?

The "sm" btl was the default shared memory transport before vader

  Is vader the intra-node default only (i.e. replaces sm  by default),

Yes

  or does it somehow extend beyond node boundaries, and replaces (or
  brings in) network btls (openib,tcp,etc) ?

Nope - just intra-node

  If I am running on several nodes, and want to use openib, not tcp,
  and, say, use vader, what is the right syntax?

  * nothing (OMPI will figure it out ... but what if you have
  IB,Ethernet,Myrinet,OpenGM, altogether?)

If you have higher-speed connections, we will pick the fastest for
inter-node messaging as the "default" since we expect you would want the
fastest possible transport.

  * -mca btl openib (and vader will come along automatically)

Among the ones you show, this would indeed be the likely choices (openib
and vader)

  * -mca btl openib,self (and vader will come along automatically)

The "self" btl is *always* active as the loopback transport

  * -mca btl openib,self,vader (because vader is default only for 1-node
  jobs)
  * something else (or several alternatives)

  Whatever happened to the "self" btl in this new context?
  Gone? Still there?

  Many thanks,
  Gus Correa

  On 10/16/2014 11:16 PM, Jeff Squyres (jsquyres) wrote:

On Oct 16, 2014, at 1:35 PM, Gus Correa <g..

Re: [OMPI users] New ib locked pages behavior?

2014-10-21 Thread Gus Correa

Hi Bill

I have 2.6.X CentOS stock kernel.
I set both parameters.
It works.

Maybe the parameter names may changed in 3.X kernels?
(Which is really bad ...)
You could check if there is more information in:
/sys/module/mlx4_core/parameters/

There seems to be a thread on the list about this (but apparently
no solution):
http://www.open-mpi.org/community/lists/users/2013/02/21430.php

Maybe Mellanox has more information about this?

Gus Correa

On 10/21/2014 08:15 PM, Bill Broadley wrote:

On 10/21/2014 04:18 PM, Gus Correa wrote:

Hi Bill

Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ?

http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem


Ah, that helped.  Although:
/lib/modules/3.13.0-36-generic/kernel/drivers/net/ethernet/mellanox/mlx4$
modinfo mlx4_core | grep "^parm"

Lists some promising looking parameters:
parm:   log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) 
(int)

The FAQ recommends log_num_mtt or num_mtt and NOT log_mtts_per_seg, sadly:
$ modinfo mlx4_core | grep "^parm" | grep mtt
parm:   log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) 
(int)
$

Looks like the best I can do is bump log_mtts_per_seg.

I tried:
$ cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core log_num_mtt=24
$

But:
[6.691959] mlx4_core: unknown parameter 'log_num_mtt' ignored

I ended up with:
options mlx4_core log_mtts_per_seg=2

I'm hoping that doubles the registerable memory, although I did see a
recommendation to raise it to double the system ram (in this case 64GB ram/128GB
locakable.

Maybe an update to the FAQ is needed?





Re: [OMPI users] New ib locked pages behavior?

2014-10-21 Thread Gus Correa

Hi Bill

Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ?

http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem

I hope this helps,
Gus Correa

On 10/21/2014 06:36 PM, Bill Broadley wrote:


I've setup several clusters over the years with OpenMPI.  I often get the below
error:

WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.
...
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:  c2-31
  Registerable memory: 32768 MiB
  Total memory:64398 MiB

I'm well aware of the normal fixes, and have implemented them in puppet to
ensure compute nodes get the changes.  To be paranoid I've implemented all the
changes, and they all worked under ubuntu 13.10.

However with ubuntu 14.04 it seems like it's not working, thus the above 
message.

As recommended by the faq's I've implemented:
1) ulimit -l unlimited in /etc/profile.d/slurm.sh
2) PropagateResourceLimitsExcept=MEMLOCK in slurm.conf
3) UsePAM=1 in slurm.conf
4) in /etc/security/limits.conf
* hard memlock unlimited
* soft memlock unlimited
* hard stack unlimited
* soft stack unlimited

My changes seem to be working, of I submit this to slurm:
#!/bin/bash -l
ulimit -l
hostname
mpirun bash -c ulimit -l
mpirun ./relay 1 131072

I get:
unlimited
c2-31
unlimited
unlimited
unlimited
unlimited



Is there some new kernel parameter, ofed parameter, or similar that controls
locked pages now?  The kernel is 3.13.0-36 and the libopenmpi-dev package is 
1.6.5.

Since the ulimit -l is getting to both the slurm launched script and also to the
mpirun launched binaries I'm pretty puzzled.

Any suggestions?
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/10/25544.php





Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-17 Thread Gus Correa

Hi Jeff

Many thanks for looking into this and filing a bug report at 11:16PM!

Thanks to Aurelien, Ralph and Nathan for their help and clarifications
also.

**

Related suggestion:

Add a note to the FAQ explaining that in OMPI 1.8
the new (default) btl is vader (and what it is).

It was a real surprise to me.
If Aurelien Bouteiller didn't tell me about vader,
I might have never realized it even existed.

That could be part of one of the already existent FAQs
explaining how to select the btl.

**

Doubts (btl in OMPI 1.8):

I still don't understand clearly the meaning and scope of vader
being a "default btl".
Which is the scope of this default: intra-node btl only perhaps?
Was there a default btl before vader, and which?
Is vader the intra-node default only (i.e. replaces sm  by default),
or does it somehow extend beyond node boundaries, and replaces (or 
brings in) network btls (openib,tcp,etc) ?


If I am running on several nodes, and want to use openib, not tcp,
and, say, use vader, what is the right syntax?

* nothing (OMPI will figure it out ... but what if you have 
IB,Ethernet,Myrinet,OpenGM, altogether?)

* -mca btl openib (and vader will come along automatically)
* -mca btl openib,self (and vader will come along automatically)
* -mca btl openib,self,vader (because vader is default only for 1-node jobs)
* something else (or several alternatives)

Whatever happened to the "self" btl in this new context?
Gone? Still there?

Many thanks,
Gus Correa

On 10/16/2014 11:16 PM, Jeff Squyres (jsquyres) wrote:

On Oct 16, 2014, at 1:35 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:


and on the MCA parameter file:

btl_sm_use_knem = 1


I think the logic enforcing this MCA param got broken when we revamped the MCA 
param system.  :-(


I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.


It looks like this MCA param was also dropped when we revamped the MCA system.  
Doh!  :-(

There's some deep mojo going on that is somehow causing knem to not be used; 
I'm too tired to understand the logic right now.  I just opened 
https://github.com/open-mpi/ompi/issues/239 to track this issue -- feel free to 
subscribe to the issue to get updates.





Re: [OMPI users] Open MPI 1.8.3 openmpi-mca-params.conf: old and new parameters

2014-10-17 Thread Gus Correa

Hi Ralph

Thank you.
Your fixes covered much more than I could find.
The section about the three levels of process placement
(" Mapping, Ranking, and Binding: Oh My!") really helps.
I would just add at the very beginning
short sentences quickly characterizing each of the three levels.
Kind of an "abstract".
Then explain each level in more detail.

**

Also, I found Jeff's 2013 presentation about the new style
of process placement.

http://www.slideshare.net/jsquyres/open-mpi-explorations-in-process-affinity-eurompi13-presentation

The title calls it "LAMA".
(That is mud in Portuguese! But the presentation is clear.)
OK, the acronym means "Locality Aware Mapping Algorithm".
In any case, it sounds very similar to the current process placement
features of OMPI 1.8, although only Jeff and you can really tell if it
is exactly the same.

If it is the same, it may help to link it to the OMPI FAQ,
or somehow make it more visible, printable, etc.
If there are differences between OMPI 1.8 and the presentation,
it may be worth adjusting the presentation to the
current OMPI 1.8, and posting it as well.
That would be a good way to convey the OMPI 1.8
process placement conceptual model, along with its syntax
and examples.

Thank you,
Gus Correa


On 10/17/2014 12:10 AM, Ralph Castain wrote:

I know this commit could be a little hard to parse, but I have updated
the mpirun man page on the trunk and will port the change over to the
1.8 series tomorrow. FWIW, I’ve provided the link to the commit below so
you can “preview” it.

https://github.com/open-mpi/ompi/commit/f9d620e3a772cdeddd40b4f0789cf59c75b44868

HTH
Ralph



On Oct 16, 2014, at 9:43 AM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:

Hi Ralph

Yes, I know the process placement features are powerful.
They were already very good in 1.6, even in 1.4,
and I just tried the new 1.8
"-map-by l2cache" (works nicely on Opteron 6300).

Unfortunately I couldn't keep track, test, and use the 1.7 series.
I did that in the previous "odd/new feature" series (1.3, 1.5).
However, my normal workload require that
I focus my attention on the "even/stable" series
(less fun, more production).
Hence I hopped directly from 1.6 to 1.8,
although I read a number of mailing list postings about the new
style of process placement.

Pestering you again about documentation (last time for now):
The mpiexec man page also seems to need an update.
That is probably the first place people look for information
about runtime features.
For instance, the process placement examples still
use deprecated parameters and mpiexec options:
-bind-to-core, rmaps_base_schedule_policy, orte_process_binding, etc.

Thank you,
Gus Correa

On 10/15/2014 11:10 PM, Ralph Castain wrote:


On Oct 15, 2014, at 11:46 AM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>
<mailto:g...@ldeo.columbia.edu>> wrote:


Thank you Ralph and Jeff for the help!

Glad to hear the segmentation fault is reproducible and will be fixed.

In any case, one can just avoid the old parameter name
(rmaps_base_schedule_policy),
and use instead the new parameter name
(rmaps_base_mapping_policy)
without any problem in OMPI 1.8.3.



Fix is in the nightly 1.8 tarball - I'll release a 1.8.4 soon to cover
the problem.


**

Thanks Ralph for sending the new (OMPI 1.8)
parameter names for process binding.

My recollection is that sometime ago somebody (Jeff perhaps?)
posted here a link to a presentation (PDF or PPT) explaining the
new style of process binding, but I couldn't find it in the
list archives.
Maybe the link could be part of the FAQ (if not already there)?


I don't think it is, but I'll try to add it over the next day or so.



**

The Open MPI runtime environment is really great.
However, to take advantage of it one often has to do parameter guessing,
and to do time consuming tests by trial and error,
because the main sources of documentation are
the terse output of ompi_info, and several sparse
references in the FAQ.
(Some of them outdated?)

In addition, the runtime environment has evolved over time,
which is certainly a good thing.
However, along with this evolution, several runtime parameters
changed both name and functionality, new ones were introduced,
old ones were deprecated, which can be somewhat confusing,
and can lead to an ineffective use of the runtime environment.
(In 1.8.3 I was using several deprecated parameters from 1.6.5
that seem to be silently ignored at runtime.
I only noticed the problem because that segmentation fault happened.)

I know asking for thorough documentation is foolish,


Not really - it is something we need to get better about :-(


but I guess a simple table of runtime parameter names and valid values
in the FAQ, maybe sorted by their purpose/function, along with a few
examples of use, could help a lot.
Some of this material is now spread across several FA

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

On 10/16/2014 07:32 PM, Jeff Squyres (jsquyres) wrote:

Gus --

Can you send the output of configure and your config.log?



Hi Jeff.

Sure.

This is for the OMPI 1.8.3 build with Intel compilers that I've been
using to compile and run IMB.
The config.log is attached.
The configure command and environment are
(is that what you meant by "the output of configure"?):

export CC=icc
export CXX=icpc
export FC=ifort
export CFLAGS='-msse2 -fp-model precise -Wall'
export CXXFLAGS=${CFLAGS}
export FCFLAGS='-msse2 -fp-model precise -warn all'

../configure \
--prefix=${MYINSTALLDIR} \
--with-tm=/opt/torque/4.2.5/gnu-4.4.7 \
--with-verbs=/usr \
--with-knem=/opt/knem-1.1.1 \
2>&1 | tee configure_${build_id}.log

Many thanks,
Gus



On Oct 16, 2014, at 4:24 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:


On 10/16/2014 05:38 PM, Nathan Hjelm wrote:

On Thu, Oct 16, 2014 at 05:27:54PM -0400, Gus Correa wrote:

Thank you, Aurelien!

Aha, "vader btl", that is new to me!
I tought Vader was that man dressed in black in Star Wars,
Obi-Wan Kenobi's nemesis.
That was a while ago, my kids were children,
and Alec Guiness younger than Harrison Ford is today.
Oh, how nostalgic code developers can get when it comes
to naming things ...

If I am using "vader", it is totally inadvertent.
There was no such a thing in Open MPI 1.6 and earlier.

Now that you mentioned, I can see lots of it in the 1.8.3
ompi_info output.
In addition, my stderr files show messages like this:

imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to
[[59987,1],26] on node node13 because vader btl has higher exclusivity
(65536 > 65535)

So, you are right, "vader" is taking over and knocking off "sm" (and openib
and everybody else).
Darn Vader!
Probably knem is going down the tubes along with sm, right?


Depends. If there is a reason to continue supporting knem then vader
will be updated to support it. I don't currently see a reason to at this
time though (since sm continues to live for now).



Right now knem is not working in OMPI 1.8.3, even if I turn off vader,
and leave only sm,self,openib.
I just sent another email documenting that.


I was used to sm, openib, self and tcp BTLs.
I normally just do "btl = ^tcp" in the MCA parameters file,
to stick to sm, openib, and self.

That worked fine in 1.6.5 (and earlier), and knem worked
flawlessly there.
The same settings in 1.8.3 don't bring up the knem functionality.
So, this seems to be yet another change in 1.8.3 that I need to learn.

Can you or some other list subscriber elaborate a bit about
this 'vader' btl?
The Open MPI FAQ doesn't have anthing about it.
What is it after all?
Does it play the same role as "sm", i.e., an intra-node btl?
Considering the name, is "vader" good or bad?
Or better: In which circumstances is "vader" good and when is it bad?


Vader is a btl I originally wrote to support Cray's XPMEM shared memory
interface. It was designed to be cleaner than btl/sm have better small
message latency, bandwidth, and message rates. Because its latency is so
much better than sm I removed the XPMEM requirement and added CMA
support.



I presume this requires kernel 3.X, as Aurelien pointed out.
As a matter of policy, and to keep your user base broad,
I would suggest to keep a generous
range of backwards compatible support built into OMPI.
This would be sm, knem, etc, which I suppose can coexist with vader, or not?
I can't speak for others but we run production codes in
standard Linux distributions (Centos 6.X, 5.X) whith 2.6.Y kernels.
I suppose other people have similar situations.


Should I give in to the dark side of the force and keep "vader"
turned on, or should I just do something like
"btl = ^tcp,^vader" ?


You can turn off vader if you want to use knem. I would run some tests
to see if there is much of a difference between sm/knem and vader
though. I don't have any systems that have knem installed so I haven't
been able to run these tests myself. I would primarily focus on the
memory usage and the bandwidth.

-Nathan


Please, see my last email.
Turning off vader and sm on, still doesn't make knem work,
unless I made some big mistake along the way.
I would love to use 1.8.3 in production,
as long as sm+knem support works, hence it it would be
great if somebody points out any mistake that I may have made.

Also, for large messages, IMB with 1.6.5+sm+knem gives
me ~30% speedups w.r.t. 1.8.3+sm+(broken)-knem or w.r.t. 1.8.3+vader,
although admittedly due to our 2.6 kernel, no CMA, etc,
the environment is not favorable to vader to begin with.
[And yet another good reason to fix/keep sm+knem in OMPI 1.8.]

Thank you,
Gus Correa





___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/10/25

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

Hi Ralph

I have clusters with CentOS 6.4, 6.5, and 5.5.

OK, completing my table (ran on CentOS 6.4):

#bytes #repetitions  t[usec]   Mbytes/sec
262144  16048.04  5203.93 :OMPI 1.6.5+knem
262144  16063.72  3923.30 :OMPI 1.8.3+vader
262144  16067.00  3731.34 :OMPI 1.8.3+sm+knem_broken

Indeed, as you said, in 1.8.3 vader is faster than sm (lines 2 an 3).
However, 1.6.5+knem still beats the crap on both (line 1).
Anyway, as others have said, vader needs the CMA thing,
kernel 3.X, etc, which I don't have, so comparing knem with
vader-less-the-support-it-requires is bogus.

Nevertheless, I would guess I am not the only one to live
in a kernel 2.6.Z world, hence still need knem support.

It would be great if the knem functionality could be restored to
OMPI 1.8.

Many thanks,
Gus

On 10/16/2014 07:13 PM, Ralph Castain wrote:

You probably have this somewhere below, but what OS are you running?

I have CentOS6, and vader works fine for me and is much faster than the
sm btl.


I can certainly ask to see if someone has time to fix the knem support -

if they do, we would definitely include the fix in the 1.8 series.



On Oct 16, 2014, at 4:06 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:


Hi All

Back to the original issue of knem in Open MPI 1.8.3.
It really seems to be broken.

I launched the Intel MPI benchmarks (IMB) job both with
'-mca btl ^vader,tcp', and with '-mca btl sm,self,openib'.
Both syntaxes seem to have turned off vader (along with tcp),
as shown in stderr by messages like this
(I also used -mca btl_base_verbose 30):

[1,11]:[node26:13439] mca: bml: Using sm btl to [[39251,1],0] on node 
node26

*However*, in both cases /dev/knem continues to *show only zeros*.

My conclusion is that the knem seems not to be working
at all in OMPI 1.8.3.

That is a real pity, because without knem performance really suffers.
I took a quick look at the Intel MPI benchmarks output
using OMPI 1.6.5 with knem, and OMPI 1.8.5 where knem doesn't work (despite my 
attempts to make it work).
The older OMPI with knem shows very good speedups.
For instance, ping-pong on two processors, message size 256kB,
OMPI 1.6.5+knem has a ~32% speeedup w.r.t. OMPI 1.8.3.

#bytes #repetitions  t[usec]   Mbytes/sec
262144  16048.04  5203.93 (OMPI 1.6.5 + knem)
262144  16063.72  3923.30 (OMPI 1.8.3, broken knem)

Numbers like these don't give me any incentive to upgrade
our production codes to OMPI 1.8.
Will this be fixed in the next Open MPI 1.8 release?

Thank you,
Gus Correa

PS - Many thanks to Aurelien Boutelier for pointing out the existence
of the vader btl.  Without his tip I would still be in the dark side.

On 10/16/2014 05:46 PM, Gus Correa wrote:


On 10/16/2014 05:28 PM, Nathan Hjelm wrote:

And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan



Thanks, Nathan

But for the benefit of mere mortals like me
who don't share the dark or the bright side of the force,
and just need to keep their MPI applications running in production mode,
hopefully with Open MPI 1.8,
can somebody explain more clearly what "vader" is about?

Thank you,
Gus Correa



On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:

FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller
<boute...@icl.utk.edu> wrote:


Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some
knem initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has
similar features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa <g...@ldeo.columbia.edu> a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up bot

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

On 10/16/2014 05:38 PM, Nathan Hjelm wrote:

On Thu, Oct 16, 2014 at 05:27:54PM -0400, Gus Correa wrote:

Thank you, Aurelien!

Aha, "vader btl", that is new to me!
I tought Vader was that man dressed in black in Star Wars,
Obi-Wan Kenobi's nemesis.
That was a while ago, my kids were children,
and Alec Guiness younger than Harrison Ford is today.
Oh, how nostalgic code developers can get when it comes
to naming things ...

If I am using "vader", it is totally inadvertent.
There was no such a thing in Open MPI 1.6 and earlier.

Now that you mentioned, I can see lots of it in the 1.8.3
ompi_info output.
In addition, my stderr files show messages like this:

imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to
[[59987,1],26] on node node13 because vader btl has higher exclusivity
(65536 > 65535)

So, you are right, "vader" is taking over and knocking off "sm" (and openib
and everybody else).
Darn Vader!
Probably knem is going down the tubes along with sm, right?


Depends. If there is a reason to continue supporting knem then vader
will be updated to support it. I don't currently see a reason to at this
time though (since sm continues to live for now).



Right now knem is not working in OMPI 1.8.3, even if I turn off vader,
and leave only sm,self,openib.
I just sent another email documenting that.


I was used to sm, openib, self and tcp BTLs.
I normally just do "btl = ^tcp" in the MCA parameters file,
to stick to sm, openib, and self.

That worked fine in 1.6.5 (and earlier), and knem worked
flawlessly there.
The same settings in 1.8.3 don't bring up the knem functionality.
So, this seems to be yet another change in 1.8.3 that I need to learn.

Can you or some other list subscriber elaborate a bit about
this 'vader' btl?
The Open MPI FAQ doesn't have anthing about it.
What is it after all?
Does it play the same role as "sm", i.e., an intra-node btl?
Considering the name, is "vader" good or bad?
Or better: In which circumstances is "vader" good and when is it bad?


Vader is a btl I originally wrote to support Cray's XPMEM shared memory
interface. It was designed to be cleaner than btl/sm have better small
message latency, bandwidth, and message rates. Because its latency is so
much better than sm I removed the XPMEM requirement and added CMA
support.



I presume this requires kernel 3.X, as Aurelien pointed out.
As a matter of policy, and to keep your user base broad,
I would suggest to keep a generous
range of backwards compatible support built into OMPI.
This would be sm, knem, etc, which I suppose can coexist with vader, or not?
I can't speak for others but we run production codes in
standard Linux distributions (Centos 6.X, 5.X) whith 2.6.Y kernels.
I suppose other people have similar situations.


Should I give in to the dark side of the force and keep "vader"
turned on, or should I just do something like
"btl = ^tcp,^vader" ?


You can turn off vader if you want to use knem. I would run some tests
to see if there is much of a difference between sm/knem and vader
though. I don't have any systems that have knem installed so I haven't
been able to run these tests myself. I would primarily focus on the
memory usage and the bandwidth.

>
> -Nathan

Please, see my last email.
Turning off vader and sm on, still doesn't make knem work,
unless I made some big mistake along the way.
I would love to use 1.8.3 in production,
as long as sm+knem support works, hence it it would be
great if somebody points out any mistake that I may have made.

Also, for large messages, IMB with 1.6.5+sm+knem gives
me ~30% speedups w.r.t. 1.8.3+sm+(broken)-knem or w.r.t. 1.8.3+vader,
although admittedly due to our 2.6 kernel, no CMA, etc,
the environment is not favorable to vader to begin with.
[And yet another good reason to fix/keep sm+knem in OMPI 1.8.]

Thank you,
Gus Correa





___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/10/25516.php





Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

Hi All

Back to the original issue of knem in Open MPI 1.8.3.
It really seems to be broken.

I launched the Intel MPI benchmarks (IMB) job both with
'-mca btl ^vader,tcp', and with '-mca btl sm,self,openib'.
Both syntaxes seem to have turned off vader (along with tcp),
as shown in stderr by messages like this
(I also used -mca btl_base_verbose 30):

[1,11]:[node26:13439] mca: bml: Using sm btl to [[39251,1],0] 
on node node26


*However*, in both cases /dev/knem continues to *show only zeros*.

My conclusion is that the knem seems not to be working
at all in OMPI 1.8.3.

That is a real pity, because without knem performance really suffers.
I took a quick look at the Intel MPI benchmarks output
using OMPI 1.6.5 with knem, and OMPI 1.8.5 where knem doesn't work 
(despite my attempts to make it work).

The older OMPI with knem shows very good speedups.
For instance, ping-pong on two processors, message size 256kB,
OMPI 1.6.5+knem has a ~32% speeedup w.r.t. OMPI 1.8.3.

#bytes #repetitions  t[usec]   Mbytes/sec
262144  16048.04  5203.93 (OMPI 1.6.5 + knem)
262144  16063.72  3923.30 (OMPI 1.8.3, broken knem)

Numbers like these don't give me any incentive to upgrade
our production codes to OMPI 1.8.
Will this be fixed in the next Open MPI 1.8 release?

Thank you,
Gus Correa

PS - Many thanks to Aurelien Boutelier for pointing out the existence
of the vader btl.  Without his tip I would still be in the dark side.

On 10/16/2014 05:46 PM, Gus Correa wrote:


On 10/16/2014 05:28 PM, Nathan Hjelm wrote:

And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan



Thanks, Nathan

But for the benefit of mere mortals like me
who don't share the dark or the bright side of the force,
and just need to keep their MPI applications running in production mode,
hopefully with Open MPI 1.8,
can somebody explain more clearly what "vader" is about?

Thank you,
Gus Correa



On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:

FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller
<boute...@icl.utk.edu> wrote:


Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some
knem initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has
similar features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa <g...@ldeo.columbia.edu> a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data
source: default value)  Whether this component supports the knem
Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular
item ("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min",
"btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vani

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa


On 10/16/2014 05:28 PM, Nathan Hjelm wrote:

And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan



Thanks, Nathan

But for the benefit of mere mortals like me
who don't share the dark or the bright side of the force,
and just need to keep their MPI applications running in production mode,
hopefully with Open MPI 1.8,
can somebody explain more clearly what "vader" is about?

Thank you,
Gus Correa



On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:

FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller <boute...@icl.utk.edu> wrote:


Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has similar 
features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa <g...@ldeo.columbia.edu> a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
default value)  Whether this component supports the knem Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", "btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environment?

- Was knem somehow phased out in 1.8.3?

- Could there be a bad interaction with other runtime parameters that
somehow is knocking out knem in 1.8.3?
(FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
reporting the bindings, which is exactly what I do on 1.6.5,
although the runtime parameter syntax has changed.)

- Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
(i.e. a bug)

- Is there a way to increase verbosity to detect if knem is being
used by OMPI?
That would certainly help to check what is going on.
I tried '-mca btl_base_verbose 30' but there was no trace of knem
in sderr/stdout of either 1.6.5 or 1.8.3.
So, the evidence I have that knem is
active in 1.6.5 but not in 1.8.3 comes only from the statistics in
/dev/knem.

***


Thank you,
Gus Correa

***

PS - As an aside, I also have some questions on the knem setup,
which I mostly copied from the knem web site
(hopefully Brice Goglin is listening ...):

- Is 32768 in 'btl_sm_eager_limit 32768' a good number,
or should it be larger/smaller/something else?
[OK, I know I should benchmark it, but exploring the whole parameter
space takes long, so why not asking? ]

- Is it worth using 'btl_sm_knem_dma_min 1048576'?
[I think I read somewhere that this dma engine offload
is an Intel thing, not AMD.]

- How about btl_sm_knem_max_simultaneous?
That one is not mentioned in t

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

On 10/16/2014 04:49 PM, Ralph Castain wrote:
> FWIW: vader is the default in 1.8

Yes, Ralph, thank you, I just noticed it in my job's stderr,
after Aurelien pointed out that new "vader" thing existed.
What a quick promotion: from inexistent to default btl!

But what is "vader" after all?
Any pointers, links, ... eh ... documentation (oops, I said I would not 
ask for it ...)


There is nothing about "vader" in the FAQ.
How does it affect the other btl?
Is it some kind of "wrapper btl" that decides which of the lower
level btls to use (sm, openib, etc)?
How does it affect knem?
What are vader's pros/cons w.r.t. using the other btls?
In which conditions is it good or bad to use it vs. the other btls?
What do I gain/lose if I do "btl = sm,self,openib"
(which presumably will knock off tcp and "vader'),
or maybe "btl=^tcp,^vader" ?

Many thanks,
Gus Correa

> On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller 
<boute...@icl.utk.edu> wrote:



Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has similar 
features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa <g...@ldeo.columbia.edu> a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
default value)  Whether this component supports the knem Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", "btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environment?

- Was knem somehow phased out in 1.8.3?

- Could there be a bad interaction with other runtime parameters that
somehow is knocking out knem in 1.8.3?
(FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
reporting the bindings, which is exactly what I do on 1.6.5,
although the runtime parameter syntax has changed.)

- Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
(i.e. a bug)

- Is there a way to increase verbosity to detect if knem is being
used by OMPI?
That would certainly help to check what is going on.
I tried '-mca btl_base_verbose 30' but there was no trace of knem
in sderr/stdout of either 1.6.5 or 1.8.3.
So, the evidence I have that knem is
active in 1.6.5 but not in 1.8.3 comes only from the statistics in
/dev/knem.

***


Thank you,
Gus Correa

***

PS - As an aside, I also have some questions on the knem setup,
which I mostly copied from the knem web site
(hopefully Brice Goglin is listening ...):

- Is 32768 in 'btl_sm_eager_limit 327

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

Thank you, Aurelien!

Aha, "vader btl", that is new to me!
I tought Vader was that man dressed in black in Star Wars,
Obi-Wan Kenobi's nemesis.
That was a while ago, my kids were children,
and Alec Guiness younger than Harrison Ford is today.
Oh, how nostalgic code developers can get when it comes
to naming things ...

If I am using "vader", it is totally inadvertent.
There was no such a thing in Open MPI 1.6 and earlier.

Now that you mentioned, I can see lots of it in the 1.8.3
ompi_info output.
In addition, my stderr files show messages like this:

imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to 
[[59987,1],26] on node node13 because vader btl has higher exclusivity 
(65536 > 65535)


So, you are right, "vader" is taking over and knocking off "sm" (and 
openib and everybody else).

Darn Vader!
Probably knem is going down the tubes along with sm, right?

I was used to sm, openib, self and tcp BTLs.
I normally just do "btl = ^tcp" in the MCA parameters file,
to stick to sm, openib, and self.

That worked fine in 1.6.5 (and earlier), and knem worked
flawlessly there.
The same settings in 1.8.3 don't bring up the knem functionality.
So, this seems to be yet another change in 1.8.3 that I need to learn.

Can you or some other list subscriber elaborate a bit about
this 'vader' btl?
The Open MPI FAQ doesn't have anthing about it.
What is it after all?
Does it play the same role as "sm", i.e., an intra-node btl?
Considering the name, is "vader" good or bad?
Or better: In which circumstances is "vader" good and when is it bad?
Should I give in to the dark side of the force and keep "vader"
turned on, or should I just do something like
"btl = ^tcp,^vader" ?

I am in CentOS 6.5, stock kernel 2.6.32, no 3.1,no  CMA linux,
so I believe I need knem for now.

I tried '-mca btl_base_verbose 30' but no knem information came out.

Many thanks,
Gus Correa

On 10/16/2014 04:40 PM, Aurélien Bouteiller wrote:

Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit

out some knem initialization info.


The CMA linux system (that ships with most 3.1x linux kernels)

has similar features, and is also supported in sm.


Aurelien
--
   ~~~ Aurélien Bouteiller, Ph.D. ~~~
  ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa <g...@ldeo.columbia.edu> a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
default value)  Whether this component supports the knem Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", "btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environ

[OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa

Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data 
source: default value)  Whether this component supports the knem Linux 
kernel module or not'


By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),

although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", 
"btl_sm_knem_max_simultaneous".


I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environment?

- Was knem somehow phased out in 1.8.3?

- Could there be a bad interaction with other runtime parameters that
somehow is knocking out knem in 1.8.3?
(FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
reporting the bindings, which is exactly what I do on 1.6.5,

although the runtime parameter syntax has changed.)

- Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
(i.e. a bug)

- Is there a way to increase verbosity to detect if knem is being
used by OMPI?
That would certainly help to check what is going on.
I tried '-mca btl_base_verbose 30' but there was no trace of knem
in sderr/stdout of either 1.6.5 or 1.8.3.
So, the evidence I have that knem is
active in 1.6.5 but not in 1.8.3 comes only from the statistics in
/dev/knem.

***


Thank you,
Gus Correa

***

PS - As an aside, I also have some questions on the knem setup,
which I mostly copied from the knem web site
(hopefully Brice Goglin is listening ...):

- Is 32768 in 'btl_sm_eager_limit 32768' a good number,
or should it be larger/smaller/something else?
[OK, I know I should benchmark it, but exploring the whole parameter
space takes long, so why not asking? ]

- Is it worth using 'btl_sm_knem_dma_min 1048576'?
[I think I read somewhere that this dma engine offload
is an Intel thing, not AMD.]

- How about btl_sm_knem_max_simultaneous?
That one is not mentioned in the knem web site.
Should I leave it default to zero or set it to 1? 2? 4? Something else?


Thanks again,
Gus Correa


Re: [OMPI users] Open MPI 1.8.3 openmpi-mca-params.conf: old and new parameters

2014-10-16 Thread Gus Correa

Hi Ralph

Yes, I know the process placement features are powerful.
They were already very good in 1.6, even in 1.4,
and I just tried the new 1.8
"-map-by l2cache" (works nicely on Opteron 6300).

Unfortunately I couldn't keep track, test, and use the 1.7 series.
I did that in the previous "odd/new feature" series (1.3, 1.5).
However, my normal workload require that
I focus my attention on the "even/stable" series
(less fun, more production).
Hence I hopped directly from 1.6 to 1.8,
although I read a number of mailing list postings about the new
style of process placement.

Pestering you again about documentation (last time for now):
The mpiexec man page also seems to need an update.
That is probably the first place people look for information
about runtime features.
For instance, the process placement examples still
use deprecated parameters and mpiexec options:
-bind-to-core, rmaps_base_schedule_policy, orte_process_binding, etc.

Thank you,
Gus Correa

On 10/15/2014 11:10 PM, Ralph Castain wrote:


On Oct 15, 2014, at 11:46 AM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:


Thank you Ralph and Jeff for the help!

Glad to hear the segmentation fault is reproducible and will be fixed.

In any case, one can just avoid the old parameter name
(rmaps_base_schedule_policy),
and use instead the new parameter name
(rmaps_base_mapping_policy)
without any problem in OMPI 1.8.3.



Fix is in the nightly 1.8 tarball - I'll release a 1.8.4 soon to cover
the problem.


**

Thanks Ralph for sending the new (OMPI 1.8)
parameter names for process binding.

My recollection is that sometime ago somebody (Jeff perhaps?)
posted here a link to a presentation (PDF or PPT) explaining the
new style of process binding, but I couldn't find it in the
list archives.
Maybe the link could be part of the FAQ (if not already there)?


I don't think it is, but I'll try to add it over the next day or so.



**

The Open MPI runtime environment is really great.
However, to take advantage of it one often has to do parameter guessing,
and to do time consuming tests by trial and error,
because the main sources of documentation are
the terse output of ompi_info, and several sparse
references in the FAQ.
(Some of them outdated?)

In addition, the runtime environment has evolved over time,
which is certainly a good thing.
However, along with this evolution, several runtime parameters
changed both name and functionality, new ones were introduced,
old ones were deprecated, which can be somewhat confusing,
and can lead to an ineffective use of the runtime environment.
(In 1.8.3 I was using several deprecated parameters from 1.6.5
that seem to be silently ignored at runtime.
I only noticed the problem because that segmentation fault happened.)

I know asking for thorough documentation is foolish,


Not really - it is something we need to get better about :-(


but I guess a simple table of runtime parameter names and valid values
in the FAQ, maybe sorted by their purpose/function, along with a few
examples of use, could help a lot.
Some of this material is now spread across several FAQ, but not so
easy to find/compare.
That doesn't need to be a comprehensive table, but commonly used
items like selecting the btl, selecting interfaces,
dealing with process binding,
modifying/enriching the stdout/sterr output
(tagging output, increasing verbosity, etc),
probably have their place there.


Yeah, we fell down on this one. The changes were announced with each
step in the 1.7 series, but if you step from 1.6 directly to 1.8, you'll
get caught flat-footed. We honestly didn't think of that case, and so we
mentally assumed that "of course people have been following the series -
they know what happened".

You know what they say about those who "assume" :-/

I'll try to get something into the FAQ about the entire new mapping,
ranking, and binding system. It is actually VERY powerful, allowing you
to specify pretty much any placement pattern you can imagine and bind it
to whatever level you desire. It was developed in response to requests
from researchers who wanted to explore application performance versus
placement strategies - but we provided some simplified options to
support more common usage patterns.





Many thanks,
Gus Correa


On 10/15/2014 11:12 AM, Jeff Squyres (jsquyres) wrote:

We talked off-list -- fixed this on master and just filed
https://github.com/open-mpi/ompi-release/pull/33 to get this into the
v1.8 branch.


On Oct 14, 2014, at 7:39 PM, Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>> wrote:



On Oct 14, 2014, at 5:32 PM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:


Dear Open MPI fans and experts

This is just a note in case other people run into the same problem.

I just built Open MPI 1.8.3.
As usual I put my old settings on openmpi-mca-params.conf,
with no further t

Re: [OMPI users] Hybrid OpenMPI/OpenMP leading to deadlocks?

2014-10-16 Thread Gus Correa

Hi Kevin

Wouldn't it be possible to make your code restartable, by saving
the appropriate fluid configuration/phase space variables,
and splitting your long run into smaller pieces?
That is a very common strategy for large PDE integrations.
Time invested in programming the restart features may pay off
in more steady jobs, and in integrations that actually complete.

If a long run fails, because of a network glitch, exhaustion
of resources, slow NFS server, or any other reason,
you loose a long computing time.
If a short run fails, you can always restart from the previous leg/job,
and your loss is small.

Nearly all parallel code that we run here is restartable
(GFD is a form of CFD :) ),
both to prevent long run failures like the one you mentioned,
as well as to provide a better throughput and user time share
on the cluster as a whole.
This is nothing new, and
restartable codes + short job queue time policy
is very common out there.

Here most problems with long runs
(we have some non-restartable serial code die-hards),
happen due to NFS issues (busy, slow response, etc),
and code with poorly designed IO.

My two cents,
Gus Correa

On 10/16/2014 10:16 AM, McGrattan, Kevin B. Dr. wrote:

The individual MPI processes appear to be using a few percent of the
system memory.

I have created a loop containing repeated calls to MPI_TESTALL. When the
process is in this loop for more than 10 s, it calls MPI_ABORT. So the
only error message I see is related to all the processes being stopped
suddenly.

Is it reasonable to ABORT after 10 s? If I just use MPI_WAITALL, the job
hangs indefinitely. I at least know for which MPI exchange and which MPI
process the code is hanging.

Is there a way to change the number of retries for a given MPI
send/receive, or a more graceful timeout function than what I have coded?

*From:*users [mailto:users-boun...@open-mpi.org] *On Behalf Of *Ralph
Castain
*Sent:* Wednesday, October 15, 2014 11:05 PM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] Hybrid OpenMPI/OpenMP leading to deadlocks?

If you only have one thread doing MPI calls, then single and funneled
are indeed the same. If this is only happening after long run times, I'd
suspect resource exhaustion. You might check your memory footprint to
see if you are running into leak issues (could be in our library as well
as your app). When you eventually deadlock, do you get any error output?
If you are using IB and run out of QP, you should at least get something
saying that.

On Oct 15, 2014, at 8:22 AM, McGrattan, Kevin B. Dr.
<kevin.mcgrat...@nist.gov <mailto:kevin.mcgrat...@nist.gov>> wrote:



I am using OpenMPI 1.8.3 on a linux cluster to run fairly long CFD
(computational fluid dynamics) simulations using 16 MPI processes. The
calculations last several days and typically involve millions of MPI
exchanges. I use the Intel Fortran compiler, and when I compile with the
–openmp option and run with only one OpenMP thread per MPI process, I
tend to get deadlocks after several days of computing. These deadlocks
only occur in about 1 out of 10 calculations, and they only occur after
running for several days. I have eliminated things like network
glitches, power spikes, etc, as possibilities. The only thing left is
the inclusion of the OpenMP option – even though I am running with just
one OpenMP thread per MPI process. I have read about the issues with
MPI_THREAD_INIT, and I have reduced the REQUIRED level of support to
MPI_THREAD_FUNNELED, down from MPI_THREAD_SERIALIZED. The latter was not
necessary for my application, and I think the reduction in level of
support has helped, but not completely removed, the deadlocking. Of
course, there is always the possibility that I have coded my MPI calls
improperly, even though the code runs for days on end. Maybe there’s
that one in a million possibility that rank x gets to a point in the
code that is so far ahead of all the other ranks that a deadlock occurs.
Placing MPI_BARRIERs has not helped me find any such situation.

So I have two questions. First, has anyone experienced something similar
to this where inclusion of OpenMP in an MPI code has caused deadlocks?
Second, is it possible that reducing the REQUIRED level of support to
MPI_THREAD_SINGLE will cause the code to behave differently than
FUNNELED? I have read in another post that SINGLE and FUNNELED are
essentially the same thing. I have even noted that I can spawn OpenMP
threads even when I use SINGLE.

Thanks

Kevin McGrattan

National Institute of Standards and Technology

100 Bureau Drive, Mail Stop 8664

Gaithersburg, Maryland 20899

301 975 2712

___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2014/10/25500.php



___
users mailing list
us...@open-mpi.org
Subscriptio

Re: [OMPI users] Open MPI 1.8.3 openmpi-mca-params.conf: old and new parameters

2014-10-15 Thread Gus Correa

Thank you Ralph and Jeff for the help!

Glad to hear the segmentation fault is reproducible and will be fixed.

In any case, one can just avoid the old parameter name
(rmaps_base_schedule_policy),
and use instead the new parameter name
(rmaps_base_mapping_policy)
without any problem in OMPI 1.8.3.

**

Thanks Ralph for sending the new (OMPI 1.8)
parameter names for process binding.

My recollection is that sometime ago somebody (Jeff perhaps?)
posted here a link to a presentation (PDF or PPT) explaining the
new style of process binding, but I couldn't find it in the
list archives.
Maybe the link could be part of the FAQ (if not already there)?

**

The Open MPI runtime environment is really great.
However, to take advantage of it one often has to do parameter guessing,
and to do time consuming tests by trial and error,
because the main sources of documentation are
the terse output of ompi_info, and several sparse
references in the FAQ.
(Some of them outdated?)

In addition, the runtime environment has evolved over time,
which is certainly a good thing.
However, along with this evolution, several runtime parameters
changed both name and functionality, new ones were introduced,
old ones were deprecated, which can be somewhat confusing,
and can lead to an ineffective use of the runtime environment.
(In 1.8.3 I was using several deprecated parameters from 1.6.5
that seem to be silently ignored at runtime.
I only noticed the problem because that segmentation fault happened.)

I know asking for thorough documentation is foolish,
but I guess a simple table of runtime parameter names and valid values
in the FAQ, maybe sorted by their purpose/function, along with a few 
examples of use, could help a lot.

Some of this material is now spread across several FAQ, but not so
easy to find/compare.
That doesn't need to be a comprehensive table, but commonly used
items like selecting the btl, selecting interfaces,
dealing with process binding,
modifying/enriching the stdout/sterr output
(tagging output, increasing verbosity, etc),
probably have their place there.


Many thanks,
Gus Correa


On 10/15/2014 11:12 AM, Jeff Squyres (jsquyres) wrote:

We talked off-list -- fixed this on master and just filed 
https://github.com/open-mpi/ompi-release/pull/33 to get this into the v1.8 
branch.


On Oct 14, 2014, at 7:39 PM, Ralph Castain <r...@open-mpi.org> wrote:



On Oct 14, 2014, at 5:32 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:


Dear Open MPI fans and experts

This is just a note in case other people run into the same problem.

I just built Open MPI 1.8.3.
As usual I put my old settings on openmpi-mca-params.conf,
with no further thinking.
Then I compiled the connectivity_c.c program and tried
to run it with mpiexec.
That is a routine that never failed before.

Bummer!
I've got a segmentation fault right away.


Strange  - it works fine from the cmd line:

07:27:04  (v1.8) /home/common/openmpi/ompi-release$ mpirun -n 1 -mca 
rmaps_base_schedule_policy core hostname
--
A deprecated MCA variable value was specified in the environment or
on the command line.  Deprecated MCA variables should be avoided;
they may disappear in future releases.

  Deprecated variable: rmaps_base_schedule_policy
  New variable:rmaps_base_mapping_policy
--
bend001

HOWEVER, I can replicate that behavior when it is in the default params file! I 
don't see the immediate cause of the difference, but will investigate.



After some head scratching, checking my environment, etc,
I thought I might have configured OMPI incorrectly.
Hence, I tried to get information from ompi_info.
Oh well, ompi_info also segfaulted!

It took me a while to realize that the runtime parameter
configuration file was the culprit.

When I inserted the runtime parameter settings one by one,
the segfault came with this one:

rmaps_base_schedule_policy = core

Ompi_info (when I got it to work) told me that the parameter above
is now a deprecated synonym of:

rmaps_base_mapping_policy = core

In any case, the old synonym doesn't work and makes ompi_info and
mpiexec segfault (and I'd guess anything else that requires the OMPI runtime 
components).
Only the new parameter name works.


That's because the segfault is happening in the printing of the deprecation 
warning.



***

I am also missing in the ompi_info output the following
(OMPI 1.6.5) parameters (not reported by ompi_info --all --all):



1) orte_process_binding  ===> hwloc_base_binding_policy

2) orte_report_bindings   ===> hwloc_base_report_bindings

3) opal_paffinity_alone  ===> gone, use hwloc_base_binding_policy=none if you 
don't want any binding



Are they gone forever?

Are there replacements for them, with approximately the same functionality?

Is there a list comparing the new (1.8) vs. old (1.6)
OMPI runtime parameters, and/or any add

Re: [OMPI users] General question about running single-node jobs.

2014-10-02 Thread Gus Correa


Hi Lee-Ping

Computational Chemistry is Greek to me.

However, on pp. 12 of the Q-Chem manual 3.2

(PDF online 
http://www.q-chem.com/qchem-website/doc_for_web/qchem_manual_3.2.pdf)


there are explanations of the meaning of QCSCRATCH and
QLOCALSRC, etc, which as Ralph pointed out, seem to be a sticking point,
and showed up in the warning messages, which I enclose below.

QLOCALSRC specifies a local disk for IO.
I wonder if the node(s) is (are) diskless, and this might cause the problem.
Another possibility is that mpiexec may not be passing these
environment variables.
(Do you pass them in the mpiexec/mpirun command line?)


QCSCRATCH defines a directory for temporary files.
If this is a network shared directory, could it be that some nodes
are not mounting it correctly?
Likewise, if your home directory or your job run directory are not
mounted that could be a problem.
Or maybe you don't have write permission (sometimes this
happens in /tmp, specially if it is a ramdir/tmpdir, which may also have 
a small size).


Your BlueWaters system administrator may be able to shed some light on 
these things.


Also the Q-Chem manual says it is a pre-compiled executable,
which as far as I know would require a matching version of OpenMPI.
(Ralph, please correct me if I am wrong.).

However, you seem to have the source code, at least you sent a
snippet of it. [With all those sockets being opened besides MPI ...]

Did you recompile with OpenMPI?
Did you add the $OMPI/bin to PATH and $OMPI/lib to LD_LIBRARY_PATH
and are these environment variables propagated to the job execution 
nodes (specially those that are failing)?



Anyway, just a bunch of guesses ...
Gus Correa

*
QCSCRATCH Defines the directory in which
Q-Chem
will store temporary files.
Q-Chem
will usually remove these files on successful completion of t
he job, but they
can be saved, if so wished. Therefore,
QCSCRATCH
should not reside in
a directory that will be automatically removed at the end of a
job, if the
files are to be kept for further calculations.
Note that many of these files can be very large, and it should be
ensured that
the volume that contains this directory has sufficient disk sp
ace available.
The
QCSCRATCH
directory should be periodically checked for scratch
files remaining from abnormally terminated jobs.
QCSCRATCH
defaults
to the working directory if not explicitly set. Please see se
ction 2.6 for
details on saving temporary files and consult your systems ad
ministrator.


QCLOCALSCR On certain platforms, such as Linux clusters, it
is sometimes preferable to
write the temporary files to a disk local to the node.
QCLOCALSCR
spec-
ifies this directory. The temporary files will be copied to
QCSCRATCH
at
the end of the job, unless the job is terminated abnormally. I
n such cases
Q-Chem
will attempt to remove the files in
QCLOCALSCR
, but may not
be able to due to access restrictions. Please specify this va
riable only if
required
*

On 10/02/2014 02:08 PM, Lee-Ping Wang wrote:

Hi Ralph,

I’ve been troubleshooting this issue and communicating with Blue Waters
support.  It turns out that Q-Chem and OpenMPI are both trying to open
sockets, and I get different error messages depending on which one fails.

As an aside, I don’t know why Q-Chem needs sockets of its own to
communicate between ranks; shouldn’t OpenMPI be taking care of all
that?  (I’m unfamiliar with this part of the Q-Chem code base, maybe
it’s trying to duplicate some functionality?)

The Blue Waters support has indicated that there’s a problem with their
realm-specific IP addressing (RSIP) for the compute nodes, which they’re
working on fixing.  I also tried running the same Q-Chem / OpenMPI job
on a management node which I think has the same hardware (but not the
RSIP), and the problem went away.  So I think I’ll shelve this problem
for now, until Blue Waters support gets back to me with the fix. :)

Thanks,

-Lee-Ping

*From:*users [mailto:users-boun...@open-mpi.org] *On Behalf Of *Lee-Ping
Wang
*Sent:* Tuesday, September 30, 2014 1:15 PM
*To:* Open MPI Users
*Subject:* Re: [OMPI users] General question about running single-node jobs.

Hi Ralph,

Thanks.  I'll add some print statements to the code and try to figure
out precisely where the failure is happening.

- Lee-Ping

On Sep 30, 2014, at 12:06 PM, Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>> wrote:



On Sep 30, 2014, at 11:19 AM, Lee-Ping Wang <leep...@stanford.edu
<mailto:leep...@stanford.edu>> wrote:



Hi Ralph,

  If so, then I should be able to (1) locate where the port
number is defined in the code, and (2) randomize the port number
every time it's called to work around the issue.  What do you think?

That might work, depending on the code. I'm not sure what it is
trying to connect to, and if that code knows how to handle arbitrary
connections

Th

Re: [OMPI users] About debugging and asynchronous communication

2014-09-18 Thread Gus Correa

There is no guarantee that the messages will be received in the same
order that they were sent.
Use tags or another mechanism to match the messages on send and recv ends.

On 09/18/2014 10:42 AM, XingFENG wrote:

I have found some thing strange.

Basically, in my codes, processes send and receive messages to/from
others with variable lengths asynchronously. When sending messages, a
process would first send the length of message and then the content of
the message. When receiving, a process would first receive the length.
Then, it allocate the buffer and receive content of message.

However, at some time (say, after 150708 times of communication ), some
process would receive a wrong length(say 170 instead of 445) and the
process exits abnormally. Anyone has similar experience?

On Thu, Sep 18, 2014 at 10:07 PM, XingFENG > wrote:

Thank you for your reply! I am still working on my codes. I would
update the post when I fix bugs.

On Thu, Sep 18, 2014 at 9:48 PM, Nick Papior Andersen
> wrote:

I just checked, if the tests return "Received" for all messages
it will not go into memory burst.
Hence doing MPI_Test will be enough. :)

Hence, if at any time the mpi-layer is notified about the
success of a send/recv it will clean up. This makes sense :)

See the updated code.

2014-09-18 13:39 GMT+02:00 Tobias Kloeffel
>:

ok i have to wait until tomorrow, they have some problems
with the network...




On 09/18/2014 01:27 PM, Nick Papior Andersen wrote:

I am not sure whether test will cover this... You should
check it...


I here attach my example script which shows two working
cases, and one not workning (you can check the memory
usage simultaneously and see that the first two works, the
last one goes ballistic in memory).

Just check it with test to see if it works...


2014-09-18 13:20 GMT+02:00 XingFENG
>:

Thanks very much for your reply!

To Sir Jeff Squyres:

I think it fails due to truncation errors. I am now
logging information of each send and receive to find
out the reason.




To Sir Nick Papior Andersen:

Oh, wait (mpi_wait) is never called in my codes.

What I do is to call MPI_Irecv once. Then MPI_Test is
called several times to check whether new messages are
available. If new messages are available, some
functions to process these messages are called.

I will add the wait function and check the running
results.

On Thu, Sep 18, 2014 at 8:47 PM, Nick Papior Andersen
>
wrote:

In complement to Jeff, I would add that using
asynchronous messages REQUIRES that you wait
(mpi_wait) for all messages at some point. Even
though this might not seem obvious it is due to
memory allocation "behind the scenes" which are
only de-allocated upon completion through a wait
statement.


2014-09-18 12:36 GMT+02:00 Jeff Squyres (jsquyres)
>:

On Sep 18, 2014, at 2:43 AM, XingFENG
> wrote:

> a. How to get more information about errors?
I got errors like below. This says that
program exited abnormally in function
MPI_Test(). But is there a way to know more
about the error?
>
> *** An error occurred in MPI_Test
> *** on communicator MPI_COMM_WORLD
> *** MPI_ERR_TRUNCATE: message truncated
> *** MPI_ERRORS_ARE_FATAL: your MPI job will
now abort

For the purpose of this discussion, let's take
a simplification that you are sending and
receiving the same datatypes (e.g., you're
sending MPI_INT and you're receiving MPI_INT).

This error means that you tried to receive
message with too small a buffer.


Re: [OMPI users] compilation problem with ifort

2014-09-04 Thread Gus Correa

Hi Elie

I really think you need to direct your questions to the EPW and QE lists 
or developers.

This is clearly a problem in their configuration scripts and makefiles,
which they should address.
Otherwise, since it works here it should work for you also,
assuming you follow the same recipe that I did and sent to you.
There are differences is in compiler versions,
maybe also in MPI versions, maybe in the Linux distribution,
but configuration scripts are meant to take care of such differences.
They are also the right forum for questions that are specific to EPW and 
QE, and don't belong to the OpenMPI list.


I don't know how to help you further.

***

As for your question about libraries.

Compilers in principle link the executables to shared libraries,
when they are available.  These belong to the computer where
the compilation happened. Hence, the executable portability to
another computer requires the same shared libraries (version, etc)
located in the same directories as the original computer.
(Well, there are ways to get around this, but this would only
make things make things even more confusing to you.)
That is why the epw.x executable I created here is useless
to you.

My Intel compiler is newer than yours, and it also has the MKL
libraries.
However, the QE configure script found the MKL BLAS and LAPACK
libraries in your case, but it didn't find on my case (it seems
to have found the lapack and blas libraries from Linux).
I.e, my configure ended up with this:

The following libraries have been found:
  BLAS_LIBS= -lblas
  LAPACK_LIBS= -llapack
  FFT_LIBS=
Please check if this is what you expect.


Why, it is a question to be posed to the EPW and QE developers.

Nevertheless, the EPW and QE code seems to come with code for
all the required libraries (blas, lapack, fft) and to build
them.  At least that is what seems to have happened on my computer.
So, I don't think you need any other libraries.

Good luck,
Gus Correa


On 09/04/2014 04:17 PM, Elio Physics wrote:

Dear Gus,

Firstly I really need to thank you for the effort you are doing to help
me and write all these e-mails and in details explaining every step.
Secondly, I did all what you wrote; the EPW is indeed inside the QE
espresso but I still get the same annoying error. I actually deleted all
the tar files and the files themselves and started afresh...

However I still did not tackle the LIBRARIES ISSUE..I did not quite
understand what you said about libraries..How do I know the path of the
openmpi libraries...Sorry I am really "dumb" in Fortran...Can you just
explain ONLY that part please in more details.

Another thing when configure was successful, at the end there were those
lines:

"The following libraries have been found:
   BLAS_LIBS=-L/opt/intel/mkl/10.0.011/lib/em64t -lmkl_em64t
   LAPACK_LIBS= -L/opt/intel/mkl/10.0.011/lib/em64t -lmkl_em64t
   FFT_LIBS=
Please check if this is what you expect.

If any libraries are missing, you may specify a list of directories
to search and retry, as follows:
   ./configure LIBDIRS="list of directories, separated by spaces" "

Do I need other libraries?

Thanks a lot for your efforts

ELIO MOUJAES

 > Date: Thu, 4 Sep 2014 12:48:44 -0400
 > From: g...@ldeo.columbia.edu
 > To: us...@open-mpi.org
 > Subject: Re: [OMPI users] compilation problem with ifort
 >
 > Hi Elie
 >
 > The executable generated in my computer will be useless to you,
 > because these days most if not all libraries linked to an executable are
 > dynamic/shared libraries.
 > You won't have the same in your computer, or the equivalent will be
 > located in different places, may be from different versions, etc.
 > (E.g. your Intel compiler libraries will be from a different version,
 > in a different location, and likewise for OpenMPI libraries etc.)
 > Take any executable that you may have in your computer and do "ldd
 > exectuable_name" to see the list of shared libraries.
 >
 > The error you reported suggests a misconfiguration of Makefiles,
 > or better, a mispositioning of directories.
 >
 > **
 >
 > First thing I would try is to start fresh.
 > Delete or move the old directory trees,
 > download everything again on blank directories,
 > and do the recipe all over again.
 > Leftovers of previous compilations are often a hurdle,
 > so you do yourself a favor by starting over from scratch.
 >
 > **
 > Second *really important* item to check:
 >
 > The top directories of QE and EPW *must* follow this hierarchy:
 >
 > espresso-4.0.3
 > |-- EPW-3.0.0
 >
 > Is this what you have?
 > The EPW web site just hints this in their recipe step 3.
 > The Makefiles will NOT work if this directory hierarchy is incorrect.
 >
 > The error you reported in your first email *suggests* that the Makefiles
 > in the EPW tarball are not finding the Makefiles in the QE ta

Re: [OMPI users] compilation problem with ifort

2014-09-04 Thread Gus Correa

Hi Elie

The executable generated in my computer will be useless to you,
because these days most if not all libraries linked to an executable are
dynamic/shared libraries.
You won't have the same in your computer, or the equivalent will be
located in different places, may be from different versions, etc.
(E.g. your Intel compiler libraries will be from a different version,
in a different location, and likewise for OpenMPI libraries etc.)
Take any executable that you may have in your computer and do "ldd 
exectuable_name" to see the list of shared libraries.


The error you reported suggests a misconfiguration of Makefiles,
or better, a mispositioning of directories.

**

First thing I would try is to start fresh.
Delete or move the old directory trees,
download everything again on blank directories,
and do the recipe all over again.
Leftovers of previous compilations are often a hurdle,
so you do yourself a favor by starting over from scratch.

**
Second *really important* item to check:

The top directories of QE and EPW *must* follow this hierarchy:

espresso-4.0.3
|-- EPW-3.0.0

Is this what you have?
The EPW web site just hints this in their recipe step 3.
The Makefiles will NOT work if this directory hierarchy is incorrect.

The error you reported in your first email *suggests* that the Makefiles
in the EPW tarball are not finding the Makefiles in the QE tarball,
which indicates that the the directories may not have a correct relative 
location.


I.e. the EPW top directory must be right under the QE top directory.

**

Third thing, is that you have to follow the recipe strictly (and on
the EPW web site there seems to be typos and omissions):

1) untar the QE tarball:

tar -zxf espresso-4.0.3.tar.gz

2) move the EPW tarball to the QE top directory produced by step 1 
above, something like this:


mv EPW-3.0.0.tar.gz espresso-4.0.3

3) untar the EPW tarball you copied/moved in step 2 above,
something like this:

cd espresso-4.0.3
tar -zxf  EPW-3.0.0.tar.gz

4) Set up your OpenMPI environment (assuming you are using OpenMPI
and that it is not installed in a standard location such as /usr/local):


[bash/sh]
export PATH=/your/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/your/openmpi/lib:$LD_LIBRARY_PATH

[tcsh/csh]
setenv PATH /your/openmpi/bin:$PATH
setenv LD_LIBRARY_PATH /your/openmpi/lib:$LD_LIBRARY_PATH

5) configure espresso-4.0.3, i.e, assuming you already are in the
espresso-4.0.3, do:

./configure CC=icc F77=ifort

(assuming you are using Intel compilers, and that you compiled OpenMPI 
with them, if you did

not, say, if you used gcc and gfortran, use CC=gcc FC=gfortran instead)

6) Run "make" on the top EPW directory:

cd EPW-3.0.0
make

When you configure QE it doesn't compile anything.
It just generates/sets up a bunch of Makefiles in the QE directory tree.

When you do "make" on the EPW-3.0.0 directory the top Makefile just
says (cd src; make).
If you look into the "src" subdirectory you will see that the Makefile
therein points to library and include directories two levels above,
which means that they are in the *QE* directory tree:

*
IFLAGS   = -I../../include
MODFLAGS = -I./ -I../../Modules -I../../iotk/src \
   -I../../PW -I../../PH -I../../PP
LIBOBJS  = ../../flib/ptools.a ../../flib/flib.a \
   ../../clib/clib.a ../../iotk/src/libiotk.a
W90LIB   = ../../W90/libwannier.a
**

Hence, if your QE directory is not immediately above your EPW directory
everything will fail, because the EPW Makefile won't be able to find
the bits and parts of QE that it needs.
And this is *exactly what the error message in your first email showed*,
a bunch of object files that were not found.

***

Sorry, but I cannot do any better than this.
I hope this helps,
Gus Correa

On 09/03/2014 08:59 PM, Elio Physics wrote:

Ray and Gus,

Thanks a lot for your help. I followed Gus' steps. I still have the same
problem for the compilation (I didnt check the libraries part though!).
The executables for quantum espresso work pretty fine. I have got them
in espresso-4.0.3/bin:
dynmat.x  iotk  iotk_print_kinds.x  iotk.x  matdyn.x  ph.x  pw.x  q2r.x.
The problem are the EPW executables and I dont understand why.

Gus do me a favor: can u send me all the EPW executables that you have
produced, in the epw.x? I guess this resolves the problem for the moment.

Regards

ELIO

 > Date: Wed, 3 Sep 2014 19:45:32 -0400
 > From: g...@ldeo.columbia.edu
 > To: us...@open-mpi.org
 > Subject: Re: [OMPI users] compilation problem with ifort
 >
 > Hi Elio
 >
 > For what it is worth, I followed the instructions on
 > the EPW web site, and the program compiled flawlessly.
 > Sorry, I don't know how to use/run it,
 > don't have the time to learn it, and won't even try.
 >
 > **
 >
 > 1) Environment:
 >
 > If your MPI/OpenMPI is not installed in a standard location,
 > you need to setup these environment variables:
 >

Re: [OMPI users] compilation problem with ifort

2014-09-03 Thread Gus Correa

Hi Ray, Elio, list

Hmmm ... somehow I didn't need to do "make all" in the QE top
directory before doing "make" in the EPW top directory.
Please, see the email that I just sent to the list.
I stopped in the QE configure step as per the recipe in the EPW site.
I presume the latter "make" takes care of the former, or not?

Besides epw.x in the EPW bin directory,
I ended up with several executables in the QE bin directory as well:
dynmat.x  iotk  iotk_print_kinds.x  iotk.x  matdyn.x  ph.x  pw.x  q2r.x

I would guess the relative location of the top QE and
top EPW directory (which per the recipe is right below the top QE)
plays a role.

Anyway, phonons are not my playground,
just trying to help two-cent-wise,
although this is not really an MPI or OpenMPI issue,
more or a Makefile/configure issue specific to QE and EPW.

Thanks,
Gus Correa

On 09/03/2014 07:39 PM, Ray Sheppard wrote:

Hi Elio and everyone,
   I went to the EPW website and their instructions seem lacking with
respect to The Quantum-Expresso 4.0.3 requirement.  The EPW folks want
to leverage Quantum Expresso intermediate object files.  By knowing how
it builds and telling you where to put their package, they can then
navigate relative to their make to link the files they want.
   Unfortunately, their instructions end with ./configure.  I think if
you look, you will see the Expresso object files were never built.
Instead, you should look up the complete installation instructions from
the Quantum Expresso folks. It might be as simple as "make all" but I
can guarantee there is more to be done.  Once you check that it
actually works, you can finish with the EPW specific instructions.  Of
course, these are just my two cents :)
 Ray

On 9/3/2014 7:10 PM, Jonathan Dursi (SN) wrote:




  Original Message

*From: *Elio Physics
*Sent: *Wednesday, September 3, 2014 6:48 PM
*To: *Open MPI Users
*Reply To: *Open MPI Users
*Subject: *Re: [OMPI users] compilation problem with ifort


I have already done all of the steps you mentioned. I have installed
the older version of quantum espresso, configured it and followed all
the steps on the EPW website when I got that error in the last steo;
In fact I do have the latest version of quantum espresso but since I
work with electron phonon and EPW seemed really promising and less
time consuming, I decided to give it a try.

The reason I have asked my question in this forum because once I had a
similar "compiler" issue (not the same as this one) and when i asked
on the Quantum Espresso (QE) website, one of the answers was, this is
not the right since this is a compiler problem not a QE issue so I was
really trying to avoid such answers.

Anyhow, I guess you are absolutely right. I will try to e-mail the EPW
people and explain the situation; after all they should be able to
help. Thanks for your advice and time.

ELIO MOUJAESS
University of Rondonia
Brasil

> Date: Wed, 3 Sep 2014 18:19:25 -0400
> From: g...@ldeo.columbia.edu
> To: us...@open-mpi.org
> Subject: Re: [OMPI users] compilation problem with ifort
>
> It is hard to tell why, but the object files (yes a2f.o, etc)
> seem not to have been compiled from the corresponding source files
> (a2f.f90 or similar).
>
> In general the executable (your epw.x) is compiled only after all
> the pre-requisite object files (the .o) and modules (the .mod)
> have been compiled already.
> In many cases there is not only one Makefile, but a chain/tree of
> them, to compile the code in the source directory tree (under src).
>
> Also, it is a bit awkward that you don't seem to
> have configured anything,
> i.e., telling where MPI was installed, etc,
> but that may just not be in your email.
>
> Phonons is not my league, just trying to help, but afraid I may
> not take you in the right direction.
>
> Did you do the installation as per the EPW web site? (I just found it):
> http://epw.org.uk/Main/DownloadAndInstall
> It seems to require QuantumExpresso.
> Did you get it, configure it, etc?
>
> Do they have a mailing list or bulletin board where you could get
> specific help for their software?
> (Either on EPW or on QuantumExpresso (which seems to be required):
> http://www.quantum-espresso.org/)
> That would probably be the right forum to ask your questions.
>
> My two cents,
> Gus Correa
>
>
> On 09/03/2014 05:51 PM, Elio Physics wrote:
> > This was the first error yes. What do you mean other files are
missing?
> > Do you mean the atom.o, basic_algebra_routines.o...? Well the f90
> > files present in the src subdirectory start from a2f.90
> > , allocate_epwq.o,...and so on... I am not also sure why there is that
> > slash "\" just before the "a2f.o" Is there a way to know what is
> > going on? I mean what are the first steps?
> >

Re: [OMPI users] compilation problem with ifort

2014-09-03 Thread Gus Correa

Hi Elio

For what it is worth, I followed the instructions on
the EPW web site, and the program compiled flawlessly.
Sorry, I don't know how to use/run it,
don't have the time to learn it, and won't even try.

**

1) Environment:

If your MPI/OpenMPI is not installed in a standard location,
you need to setup these environment variables:

[bash/sh]
export PATH=/your/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/your/openmpi/lib:$LD_LIBRARY_PATH

[tcsh/csh]
setenv PATH /your/openmpi/bin:$PATH
setenv LD_LIBRARY_PATH /your/openmpi/lib:$LD_LIBRARY_PATH

**

2) Configuring QE:

In step 2, I separated the final configure step, to set the compilers to 
Intel.

I used OpenMPI compiled with intel, hence I
configured QuantumEspresso with Intel compilers:

./configure CC=icc FC=ifort

NOTE: You may need to use tar zxf ... instead of tar xfz ...

**

3) Untar EPW in the correct location

Step 3 should be as below.
(You need to untar EPW inside the espresso-4.0.3 directory.
Could this have been the problem for you?):


cp EPW-3.0.0.tar.gz espresso-4.0.3/
cd espresso-4.0.3/
tar zxf EPW-3.0.0.tar.gz

**

4) Run make to compile EPW

The compilation in step 4 takes a while, is long, but works fine,
and produces the epw.x executable as expected.

**

5) Old version issue, perhaps?

Also, you said you "installed the older version of quantum espresso".
Could this have been the problem (older version)?
Did you try the latest QE (4.0.3), and the latest EPW (3.0.0),
as per the recipe on the EPW web site?

http://epw.org.uk/Main/DownloadAndInstall

**

I hope this helps,
Gus Correa




On 09/03/2014 06:48 PM, Elio Physics wrote:

I have already done all of the steps you mentioned. I have installed the
older version of quantum espresso, configured it and followed all the
steps on the EPW website when I got that error in the last steo; In fact
I do have the latest version of quantum espresso but since I work with
electron phonon and EPW seemed really promising and less time consuming,
I decided to give it a try.

The reason I have asked my question in this forum because once I had a
similar "compiler" issue (not the same as this one) and when i asked on
the Quantum Espresso (QE) website, one of the answers was, this is not
the right since this is a compiler problem not a QE issue so I was
really trying to avoid such answers.

Anyhow, I guess you are absolutely right. I will try to e-mail the EPW
people and explain the situation; after all they should be able to help.
Thanks for your advice and time.

ELIO MOUJAESS
University of Rondonia
Brasil

 > Date: Wed, 3 Sep 2014 18:19:25 -0400
 > From: g...@ldeo.columbia.edu
 > To: us...@open-mpi.org
 > Subject: Re: [OMPI users] compilation problem with ifort
 >
 > It is hard to tell why, but the object files (yes a2f.o, etc)
 > seem not to have been compiled from the corresponding source files
 > (a2f.f90 or similar).
 >
 > In general the executable (your epw.x) is compiled only after all
 > the pre-requisite object files (the .o) and modules (the .mod)
 > have been compiled already.
 > In many cases there is not only one Makefile, but a chain/tree of
 > them, to compile the code in the source directory tree (under src).
 >
 > Also, it is a bit awkward that you don't seem to
 > have configured anything,
 > i.e., telling where MPI was installed, etc,
 > but that may just not be in your email.
 >
 > Phonons is not my league, just trying to help, but afraid I may
 > not take you in the right direction.
 >
 > Did you do the installation as per the EPW web site? (I just found it):
 > http://epw.org.uk/Main/DownloadAndInstall
 > It seems to require QuantumExpresso.
 > Did you get it, configure it, etc?
 >
 > Do they have a mailing list or bulletin board where you could get
 > specific help for their software?
 > (Either on EPW or on QuantumExpresso (which seems to be required):
 > http://www.quantum-espresso.org/)
 > That would probably be the right forum to ask your questions.
 >
 > My two cents,
 > Gus Correa
 >
 >
 > On 09/03/2014 05:51 PM, Elio Physics wrote:
 > > This was the first error yes. What do you mean other files are missing?
 > > Do you mean the atom.o, basic_algebra_routines.o...? Well the f90
 > > files present in the src subdirectory start from a2f.90
 > > , allocate_epwq.o,...and so on... I am not also sure why there is that
 > > slash "\" just before the "a2f.o" Is there a way to know what is
 > > going on? I mean what are the first steps?
 > >
 > > Thank you
 > >
 > > ELIO MOUJAES
 > > Univeristy of Rondonia
 > > Brazil
 > >
 > > > Date: Wed, 3 Sep 2014 17:43:44 -0400
 > > > From: g...@ldeo.columbia.edu
 > > > To: us...@open-mpi.org
 > > > Subject: Re: [OMPI users] compilation problem with ifort
 >

Re: [OMPI users] compilation problem with ifort

2014-09-03 Thread Gus Correa

It is hard to tell why, but the object files (yes a2f.o, etc)
seem not to have been compiled from the corresponding source files 
(a2f.f90 or similar).


In general the executable (your epw.x) is compiled only after all
the pre-requisite object files (the .o) and modules (the .mod)
have been compiled already.
In many cases there is not only one Makefile, but a chain/tree of
them, to compile the code in the source directory tree (under src).

Also, it is a bit awkward that you don't seem to
have configured anything,
i.e., telling where MPI was installed, etc,
but that may just not be in your email.

Phonons is not my league, just trying to help, but afraid I may
not take you in the right direction.

Did you do the installation as per the EPW web site? (I just found it):
http://epw.org.uk/Main/DownloadAndInstall
It seems to require QuantumExpresso.
Did you get it, configure it, etc?

Do they have a mailing list or bulletin board where you could get 
specific help for their software?

(Either on EPW or on QuantumExpresso (which seems to be required):
 http://www.quantum-espresso.org/)
That would probably be the right forum to ask your questions.

My two cents,
Gus Correa


On 09/03/2014 05:51 PM, Elio Physics wrote:

This was the first error yes. What do you mean other files are missing?
Do you mean the atom.o, basic_algebra_routines.o...? Well the f90
files present in the src subdirectory start from a2f.90
, allocate_epwq.o,...and so on... I am not also sure why there is that
slash "\" just before the "a2f.o" Is there a way to know what is
going on? I mean what are the first steps?

Thank you

ELIO MOUJAES
Univeristy of Rondonia
Brazil

 > Date: Wed, 3 Sep 2014 17:43:44 -0400
 > From: g...@ldeo.columbia.edu
 > To: us...@open-mpi.org
 > Subject: Re: [OMPI users] compilation problem with ifort
 >
 > Was the error that you listed the *first* error?
 >
 > Apparently various object files are missing from the
 > ../../Modules/ directory, and were not compiled,
 > suggesting something is amiss even before the
 > compilation of the executable (epw.x).
 >
 > On 09/03/2014 05:20 PM, Elio Physics wrote:
 > > Dear all,
 > >
 > > I am really a beginner in Fortran and Linux. I was trying to compile a
 > > software (EPW). Everything was going fine (or maybe this is what I
think):
 > >
 > > mpif90 -o epw.x ../../Modules/atom.o
 > > ../../Modules/basic_algebra_routines.o ../../Modules/cell_base.o
 > > ../../Modules/check_stop.o ../../Modules/clocks.o
 > > ../../Modules/constraints_module.o ../../Modules/control_flags.o
 > > ../../Modules/descriptors.o ../../Modules/dspev_drv.o
 > > ../../Modules/electrons_base.o ../../Modules/error_handler.o
 > > ../../Modules/exc_t.o ../../Modules/fft_base.o
 > > ../../Modules/fft_parallel.o ../../Modules/fft_scalar.o
 > > ../../Modules/fft_types.o ../../Modules/functionals.o
 > > ../../Modules/input_parameters.o ../../Modules/io_files.o
 > > ../../Modules/io_global.o ../../Modules/ions_base.o
../../Modules/kind.o
 > > ../../Modules/metagga.o
 > > ..\ a2f.o
 > > allocate_epwq.o bcast_epw_input.o broyden.o close_epw.o constants_epw.o
 > > create_mesh.o create_mesh_mp.o createkmap.o dasmio.o deallocate_epw.o
 > > deallocate_eliashberg.o distribution.o dmebloch2wan.o dmewan2bloch.o
 > > dvanqq2.o dvqpsi_us3.o dvqpsi_us_only3.o dynbloch2wan.o dynwan2bloch.o
 > > eliashberg.o
 > >
 > > Then I get the following error:
 > > ifort: error #10236: File not found: 'a2f.o'
 > > ifort: error #10236: File not found: 'allocate_epwq.o'
 > > ifort: error #10236: File not found: 'bcast_epw_input.o'
 > > ifort: error #10236: File not found: 'broyden.o'
 > > ifort: error #10236: File not found: 'close_epw.o'
 > > ifort: error #10236: File not found: 'constants_epw.o'
 > > ifort: error #10236: File not found: 'create_mesh.o'
 > > ifort: error #10236: File not found: 'create_mesh_mp.o'
 > > ifort: error #10236: File not found: 'createkmap.o'
 > > ifort: error #10236: File not found: 'dasmio.o'
 > > ifort: error #10236: File not found: 'deallocate_epw.o'
 > > ifort: error #10236: File not found: 'deallocate_eliashberg.o'
 > > ifort: error #10236: File not found: 'distribution.o'
 > > ifort: error #10236: File not found: 'dmebloch2wan.o'
 > > ifort: error #10236: File not found: 'dmewan2bloch.o'
 > > ifort: error #10236: File not found: 'dvanqq2.o'
 > > ifort: error #10236: File not found: 'dvqpsi_us3.o'
 > > ifort: error #10236: File not found: 'dvqpsi_us_only3.o'
 > > ifort: error #10236: File not found: 'dynbloch2wan.o'
 > > ifort: error #10236: File not found: 'dynwan2bloch.o'
 > > ifort: error #10236: File not foun

Re: [OMPI users] compilation problem with ifort

2014-09-03 Thread Gus Correa

Was the error that you listed the *first* error?

Apparently various object files are missing from the
../../Modules/ directory, and were not compiled,
suggesting something is amiss even before the
compilation of the executable (epw.x).

On 09/03/2014 05:20 PM, Elio Physics wrote:

Dear all,

I am really a beginner in Fortran and Linux. I was trying to compile a
software (EPW). Everything was going fine (or maybe this is what I think):

mpif90 -o epw.x ../../Modules/atom.o
../../Modules/basic_algebra_routines.o ../../Modules/cell_base.o
../../Modules/check_stop.o ../../Modules/clocks.o
../../Modules/constraints_module.o ../../Modules/control_flags.o
../../Modules/descriptors.o ../../Modules/dspev_drv.o
../../Modules/electrons_base.o ../../Modules/error_handler.o
../../Modules/exc_t.o ../../Modules/fft_base.o
../../Modules/fft_parallel.o ../../Modules/fft_scalar.o
../../Modules/fft_types.o ../../Modules/functionals.o
../../Modules/input_parameters.o ../../Modules/io_files.o
../../Modules/io_global.o ../../Modules/ions_base.o ../../Modules/kind.o
../../Modules/metagga.o
..\ a2f.o
allocate_epwq.o bcast_epw_input.o broyden.o close_epw.o constants_epw.o
create_mesh.o create_mesh_mp.o createkmap.o dasmio.o deallocate_epw.o
deallocate_eliashberg.o distribution.o dmebloch2wan.o dmewan2bloch.o
dvanqq2.o dvqpsi_us3.o dvqpsi_us_only3.o dynbloch2wan.o dynwan2bloch.o
eliashberg.o

Then I get the following error:
ifort: error #10236: File not found:  'a2f.o'
ifort: error #10236: File not found:  'allocate_epwq.o'
ifort: error #10236: File not found:  'bcast_epw_input.o'
ifort: error #10236: File not found:  'broyden.o'
ifort: error #10236: File not found:  'close_epw.o'
ifort: error #10236: File not found:  'constants_epw.o'
ifort: error #10236: File not found:  'create_mesh.o'
ifort: error #10236: File not found:  'create_mesh_mp.o'
ifort: error #10236: File not found:  'createkmap.o'
ifort: error #10236: File not found:  'dasmio.o'
ifort: error #10236: File not found:  'deallocate_epw.o'
ifort: error #10236: File not found:  'deallocate_eliashberg.o'
ifort: error #10236: File not found:  'distribution.o'
ifort: error #10236: File not found:  'dmebloch2wan.o'
ifort: error #10236: File not found:  'dmewan2bloch.o'
ifort: error #10236: File not found:  'dvanqq2.o'
ifort: error #10236: File not found:  'dvqpsi_us3.o'
ifort: error #10236: File not found:  'dvqpsi_us_only3.o'
ifort: error #10236: File not found:  'dynbloch2wan.o'
ifort: error #10236: File not found:  'dynwan2bloch.o'
ifort: error #10236: File not found:  'eliashberg.o'
ifort: error #10236: File not found:  'eliashbergcom.
make[1]: *** [epw] Error 1
make[1]: Leaving directory
`/home_cluster/fis718/eliemouj/espresso-4.0.3/EPW-3.0.0/src'
make: *** [epw] Error 2

I reckon that there is an error in the Makefile. However the Makefile
content is just:
"default: epw

all: epw

epw:
 (cd src ; make )
 (cd bin ; ln -fs ../src/epw.x . )

clean:
 cd src ; rm -f *.o *.mod *.F90 *~

release:
 cd ../ ; cp -r EPW EPW-release; cd EPW-release ; \
 rm -f src/*.o src/*.mod src/*.F90 src/*~ ; \
 rm bin/*.x ; \
 rm -rf examples/*/epw/out/* examples/*/epw/tmp/* \
  examples/*/phonons/out/* examples/*/phonons/tmp/* \
  examples/*/phonons/save/* ; \
 rm -rf .svn */.svn */*/*.svn */*/*/*.svn */*/*/*/*.svn
 cd .. ; tar cfz EPW/EPW-release.tgz EPW-release ; \
 rm -rf EPW-release ; cd EPW "

Please can anyone help me and guide me how to fix this error step by
step as I am no FOrtran or Linux professional

Regards

ELIO MOUJAES
University of Rondonia
Brazil






___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/09/25253.php





Re: [OMPI users] building openmpi 1.8.1 with intel 14.0.1

2014-08-21 Thread Gus Correa

Hi Peter

If I remember right from my compilation of OMPI on a Mac
years ago, you need to have X-Code installed, in case you don't.

If vampir-trace is the only problem,
you can disable it when you configure OMPI (--disable-vt).

My two cents,
Gus Correa


On 08/21/2014 03:35 PM, Bosler, Peter Andrew wrote:

Good afternoon,

I’m having trouble configuring OpenMPI for use with the Intel compilers.
  I run the command “./configure —prefix=/opt/openmpi/intel CC=icc
CXX=icpc FC=ifort 2>&1 | tee ~/openmpi-config.out” and I notice three
problems:

 1. I get two instances of “Report this to
http://www.open-mpi.org/community/help” with regard to netinet/in.h
and netinit/tcp.h in the output (attached)
 2. I receive a note about Vampire Trace being broken and finally a
failed configure warning
 3. Configure ultimately fails because it failed to build GNU libltdl.

I’m running Mac OS X 10.9.4 on a 3.5 Ghz 6-core Intel Xeon E5 with Intel
compilers version 14.0.1.  The OpenMPI version I’m trying to build is
1.8.1.

My environment is set with LD_LIBRARY_PATH=/opt/intel/lib/intel64

As an aside, if there are any configuration options for OpenMPI that
will take special advantage of the Xeon processor, I would love to know
more about them.

Thank you very much for your time.

Pete Bosler




___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/25122.php





Re: [OMPI users] Newbie query - mpirun will not run if it's previously been killed with Control-C

2014-08-07 Thread Gus Correa

On 08/07/2014 11:49 AM, Ralph Castain wrote:


On Aug 7, 2014, at 8:47 AM, Reuti <re...@staff.uni-marburg.de
<mailto:re...@staff.uni-marburg.de>> wrote:


Am 07.08.2014 um 17:28 schrieb Gus Correa:


I guess Control-C will kill only the mpirun process.
You may need to kill the (two) jules.exe processes separately,
say, with kill -9.
ps -u "yourname"
will show what you have running.


Shouldn't Open MPI clean this up in a proper way when Control-C is
pressed?


So far as I know, it does...



How about processes in D state, waiting for a slow/busy NFS server?
Could this prevent Control-C to do the right thing?

Gus Correa



But maybe there is something left in /tmp like
"openmpi-sessions-...@..." which needs to be removed.

-- Reuti



On 08/07/2014 11:16 AM, Jane Lewis wrote:

Hi all,

This is a really simple problem (I hope) where I’ve introduced MPI to a
complex numerical model which I have to kill occasionally with Control-C
as I don’t want it running forever.

I have only used mpi_init(), mpi_comm_size(), mpi_comm_rank() and
mpi_finalize()– there are no send/receive calls going on at the moment –
and I only have two instances. My startup command is:

#/bin/bash

/opt/openmpi/bin/mpirun  -np 2 -hostfile hostfile jules.exe

where hostfile has one entry : localhost

The result of terminating the process with Control-C at the command
prompt from where I launched it, is that I am then unable to run it
again. I get the

“mpirun has exited due to process rank 0 with PID 10094 on node
metclcv10.local exiting improperly. There are two reasons this could
occur:…” error each time despite checking running processes for
stragglers, closing my terminal, or changing node.

I have spent several hours searching for an answer to this, if it’s
already somewhere then please point me in the right direction.

many thanks in advance

Jane

For info:

#ompi_info -v ompi full --parsable

package:Open MPI root@centos-6-3.localdomain
<mailto:root@centos-6-3.localdomain> Distribution

ompi:version:full:1.6.2

ompi:version:svn:r27344

ompi:version:release_date:Sep 18, 2012

orte:version:full:1.6.2

orte:version:svn:r27344

orte:version:release_date:Sep 18, 2012

opal:version:full:1.6.2

opal:version:svn:r27344

opal:version:release_date:Sep 18, 2012

mpi-api:version:full:2.1

ident:1.6.2

I’m using centos-6-3 and FORTRAN.

Jane Lewis

Deputy Technical Director, Reading e-Science Centre

Department of Meteorology

University of Reading, UK

Tel: +44 (0)118 378 5173

http://www.resc.reading.ac.uk <http://www.resc.reading.ac.uk/>



___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/08/24938.php



___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/08/24939.php



___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2014/08/24940.php




___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/24941.php





Re: [OMPI users] Newbie query - mpirun will not run if it's previously been killed with Control-C

2014-08-07 Thread Gus Correa

On 08/07/2014 11:28 AM, Gus Correa wrote:

I guess Control-C will kill only the mpirun process.
You may need to kill the (two) jules.exe processes separately,
say, with kill -9.
ps -u "yourname"
will show what you have running.



Something may have been left behind by Control-C,
although as Reuti and Ralph said that is unusual.
Besides the jules.exe and mpiexec,
check for the orted process also,
which may need to be killed manually.
Just in case: Better not run MPI programs as root,
but as a regular user.




On 08/07/2014 11:16 AM, Jane Lewis wrote:

Hi all,

This is a really simple problem (I hope) where I’ve introduced MPI to a
complex numerical model which I have to kill occasionally with Control-C
as I don’t want it running forever.

I have only used mpi_init(), mpi_comm_size(), mpi_comm_rank() and
mpi_finalize()– there are no send/receive calls going on at the moment –
and I only have two instances. My startup command is:

#/bin/bash

/opt/openmpi/bin/mpirun  -np 2 -hostfile hostfile jules.exe

where hostfile has one entry : localhost

The result of terminating the process with Control-C at the command
prompt from where I launched it, is that I am then unable to run it
again. I get the

“mpirun has exited due to process rank 0 with PID 10094 on node
metclcv10.local exiting improperly. There are two reasons this could
occur:…” error each time despite checking running processes for
stragglers, closing my terminal, or changing node.

I have spent several hours searching for an answer to this, if it’s
already somewhere then please point me in the right direction.

many thanks in advance

Jane

For info:

#ompi_info -v ompi full --parsable

package:Open MPI root@centos-6-3.localdomain Distribution

ompi:version:full:1.6.2

ompi:version:svn:r27344

ompi:version:release_date:Sep 18, 2012

orte:version:full:1.6.2

orte:version:svn:r27344

orte:version:release_date:Sep 18, 2012

opal:version:full:1.6.2

opal:version:svn:r27344

opal:version:release_date:Sep 18, 2012

mpi-api:version:full:2.1

ident:1.6.2

I’m using centos-6-3 and FORTRAN.

Jane Lewis

Deputy Technical Director, Reading e-Science Centre

Department of Meteorology

University of Reading, UK

Tel: +44 (0)118 378 5173

http://www.resc.reading.ac.uk <http://www.resc.reading.ac.uk/>



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/08/24938.php



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/08/24939.php




Re: [OMPI users] Newbie query - mpirun will not run if it's previously been killed with Control-C

2014-08-07 Thread Gus Correa

I guess Control-C will kill only the mpirun process.
You may need to kill the (two) jules.exe processes separately,
say, with kill -9.
ps -u "yourname"
will show what you have running.


On 08/07/2014 11:16 AM, Jane Lewis wrote:

Hi all,

This is a really simple problem (I hope) where I’ve introduced MPI to a
complex numerical model which I have to kill occasionally with Control-C
as I don’t want it running forever.

I have only used mpi_init(), mpi_comm_size(), mpi_comm_rank() and
mpi_finalize()– there are no send/receive calls going on at the moment –
and I only have two instances. My startup command is:

#/bin/bash

/opt/openmpi/bin/mpirun  -np 2 -hostfile hostfile jules.exe

where hostfile has one entry : localhost

The result of terminating the process with Control-C at the command
prompt from where I launched it, is that I am then unable to run it
again. I get the

“mpirun has exited due to process rank 0 with PID 10094 on node
metclcv10.local exiting improperly. There are two reasons this could
occur:…” error each time despite checking running processes for
stragglers, closing my terminal, or changing node.

I have spent several hours searching for an answer to this, if it’s
already somewhere then please point me in the right direction.

many thanks in advance

Jane

For info:

#ompi_info -v ompi full --parsable

package:Open MPI root@centos-6-3.localdomain Distribution

ompi:version:full:1.6.2

ompi:version:svn:r27344

ompi:version:release_date:Sep 18, 2012

orte:version:full:1.6.2

orte:version:svn:r27344

orte:version:release_date:Sep 18, 2012

opal:version:full:1.6.2

opal:version:svn:r27344

opal:version:release_date:Sep 18, 2012

mpi-api:version:full:2.1

ident:1.6.2

I’m using centos-6-3 and FORTRAN.

Jane Lewis

Deputy Technical Director, Reading e-Science Centre

Department of Meteorology

University of Reading, UK

Tel: +44 (0)118 378 5173

http://www.resc.reading.ac.uk 



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/24938.php





Re: [OMPI users] How to keep multiple installations at same time

2014-08-05 Thread Gus Correa

Thanks for not saying this turn that Environment Modules
is "outdated and not maintained".
That might mislead the OMPI list audience,
which has a big intersection with Environment Modules users.

On May 16, 2014, at 4:07 PM, Maxime Boissonneault 
<maxime.boissonneault_at_[hidden]> wrote:

> Instead of using the outdated and not maintained Module environment,
> why not use Lmod : https://www.tacc.utexas.edu/tacc-projects/lmod
>
> It is a drop-in replacement for Module environment that supports all
> of their features and much, much more, such as :
> - module hierarchies
> - module properties and color highlighting (we use it to higlight
> bioinformatic modules or tools for example)
> - module caching (very useful for a parallel filesystem with
> tons of modules)
> - path priorities (useful to make sure personal modules
> take precendence over system modules)
> - export module tree to json
>
> It works like a charm, understand both TCL and Lua modules
> and is  actively developped and debugged. There are litteraly new
> features every month or so. If it does not do what you want, odds are
> that the developper will add it shortly (I've had it happen).
>
> Maxime
>

Some of the features introduced by LMod seem to be in the
works in the Environment modules as well (cache, hierarchies),
judging from recent mailing list postings.

I am not sure all of them are essential, though.
For instance, I am skeptical that most users,
which are scientifically trained in one way or another,
would not understand the stack nature of the the enviroment modules
and make mistakes like the one you mentioned:

> module load gcc openmpi_gcc
> module unload gcc
> module load intel

Unless they were never told about it.

Hierarchies may simplify the naming convention, but may also work
as a straitjacket, reflecting the sys admin hierarchical choices,
but potentially limiting the user choices.
In one national lab that I have access to,
navigating their module hierarchy, and specially getting around the
official one to do what you need/want, is a pain.

Anyway, this is the OMPI list, not a place for advocacy of either
package, so I am going to stop here.

I just wanted to set the record straight that:

- the Enviroment Modules package is not dead,
- it has a large user base, and
- it is sooo good that among other things it opened the road for
the imitators!  :)

Thank you,
Gus Correa

On 08/05/2014 03:51 PM, Maxime Boissonneault wrote:



The Environment Modules package user base is not negligible,
including many universities, research centers, national labs,
ans private companies, in the US and around the world.
How does the user base of LMod compare?


The user base certainly is much larger for Environment Modules than LMod.
But, as a user of both Lmod and Environment Modules, I can tell you the
following :

Regardless of any virtues that LMod may have,
currently I don't see any reason to switch to LMod,
install everything over again

Nothing needs reinstalling. Lmod understands Tcl modules and can work
fine with your old module tree.

, troubleshoot it,
learn Lua, migrate my modules from Tcl,

Again, migration to Lua is not required. Tcl modules gets converted on
the fly.

educate my users and convince them to use a new
package to achieve the same exact thing that they currently have,

Very little education has to be done. The commands are the same :
module avail
module load/add
module unload/remove
module use
...

and in the end gain little if any
relevant/useful/new functionality.

If you do not want to make any changes, in the way you organize modules,
then don't. You will also get no benefit from changing to Lmod in that
situation.

If you do want to use new features, then there are plenty. Most notably is
- the possibility to organize modules in hierarchy   (which you do not
HAVE to do, but in my opinion, is much more intuitive).
- the possibility to cache the module structure (and avoid reading it
from a parallel filesystem every time a user type a module command).
- the possibility to color-code modules so that users can find what they
want easier out of hundreds of modules

IF you do use hierarchy, you get the added benefit of avoiding user
mistakes such as

"
module load gcc openmpi_gcc
module unload gcc
module load intel

... why is my MPI not working!
"

IF you do use hierarchy, you get the added benefit of not having silly
module names such as
fftw/3.3_gcc4.8_openmpi1.6.3
fftw/3.3_gcc4.6_openmpi1.8.1
...

Again, you do NOT have to, but the benefits much outweight the changes
that need to be made to get them.

My 2 cents,

Maxime Boissonneault



My two cents of opinion
Gus Correa


On 08/05/2014 12:54 PM, Ralph Castain wrote:

Check the repo - hasn't been touched in a very long time

On Aug 5, 2014, at 9:42 AM, Fabricio Cannini <fcann...@gmail.com> wrote:


On 05-08-2014 13:10, Ralph Castain wrote:

Since modules isn

Re: [OMPI users] How to keep multiple installations at same time

2014-08-05 Thread Gus Correa

Hi Ralph and list

I am no developer, but my impression is that,
paraphrasing Mark Twain,
the reports about the death of the
Environment Modules (http://modules.sourceforge.net/)
package have been exaggerated.
That presumed death/obsolescence has been
repeated a few times on the OMPI list,
which has a big intersection with the Environment Modules
actual and potential users,
a statement which IMHO is inaccurate, to say the least.

The Environment Modules repository has 2014 entries:

http://sourceforge.net/p/modules/git/ci/master/tree/

The Environment Modules mailing list is active as well:

https://lists.sourceforge.net/lists/listinfo/modules-interest
http://sourceforge.net/p/modules/mailman/modules-interest/

The Environment Modules package user base is not negligible,
including many universities, research centers, national labs,
ans private companies, in the US and around the world.
How does the user base of LMod compare?

Regardless of any virtues that LMod may have,
currently I don't see any reason to switch to LMod,
install everything over again, troubleshoot it,
learn Lua, migrate my modules from Tcl,
educate my users and convince them to use a new
package to achieve the same exact thing that they currently have,
and in the end gain little if any
relevant/useful/new functionality.

My two cents of opinion
Gus Correa


On 08/05/2014 12:54 PM, Ralph Castain wrote:

Check the repo - hasn't been touched in a very long time

On Aug 5, 2014, at 9:42 AM, Fabricio Cannini <fcann...@gmail.com> wrote:


On 05-08-2014 13:10, Ralph Castain wrote:

Since modules isn't a supported s/w package any more, you might consider using 
LMOD instead:

https://www.tacc.utexas.edu/tacc-projects/lmod


Modules isn't supported anymore? :O

Could you please send a link about it ?
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/24918.php


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/24919.php





Re: [OMPI users] How to keep multiple installations at same time

2014-08-05 Thread Gus Correa

Hi Ahsan

Besides Andrew's excellent suggestion for the runtime environment.

For the installation, Open MPI configuration scripts
and Makefiles support VPATH.
Hence, you can create two separate directories to
*build* it with Gnu and Intel compilers.
Then you launch configure, make, make install from
each of these dirctories, using the appropriate compilers,
and pointing to two distinct *installation directories*
(with configure -prefix).


My two cents,
Gus Correa

On 08/04/2014 11:54 PM, Andrew Caird wrote:

Hi Ahsan,

We, and I think many people, use the Environment Modules software,

http://modules.sourceforge.net , to do this.


I hope that helps.

--andy


On Aug 4, 2014, at 11:47 PM, Syed Ahsan Ali <ahsansha...@gmail.com> wrote:


I want to compile openmpi with both intel and gnu compilers. How can I install 
both at the same time and then specify which one to use during job submission.


Regards
Ahsan
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/24905.php

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/08/24906.php





Re: [OMPI users] openmpi 1.8.1 gfortran not working

2014-08-04 Thread Gus Correa

>> I have the following env variables set export OMPI_CC=gcc echo
>> $OMPI_CC export OMPI_CXX=g++ echo $OMPI_CXX export OMPI_F77=gfortran
>> echo $OMPI_F77 export OMPI_FC=gfortran echo $OMPI_FC

Dan

Have you tried to set/export the compilers without the "OMPI_" prefix?
(CC, CXX, FC)
Then "make distclean; configure; make; make install".

Gus Correa


On 08/04/2014 04:10 PM, Dan Shell wrote:

Ralph
Ok
I will give that a try
Thanks
Dan Shell


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Monday, August 04, 2014 3:11 PM
To: Open MPI Users
Subject: Re: [OMPI users] openmpi 1.8.1 gfortran not working

I know there were some lingering issues in 1.8.1 - you might want to try the
latest nightly 1.8 tarball as I believe things have been fixed now.


On Aug 4, 2014, at 11:09 AM, Dan Shell <dshellwirel...@gmail.com> wrote:


  openmpi

I have the following env variables set export OMPI_CC=gcc echo
$OMPI_CC export OMPI_CXX=g++ echo $OMPI_CXX export OMPI_F77=gfortran
echo $OMPI_F77 export OMPI_FC=gfortran echo $OMPI_FC

I run the configure script
See fortran section below
Looks like mpifort should be configure From what I can tell make
install is fine When I goto the command line and type mpifort I get
the wrapper.txt error
message:
  "No underlying compiler was specified in the wrapper compiler data file"
mpifort.wrapper.txt is in the right place

gfortran is installed correctly on unbuntu 10 linux PC

Any help is greatly appreciated
Dan Shell

** Fortran compiler
checking for gfortran... gfortran
checking whether we are using the GNU Fortran compiler... yes checking
whether gfortran accepts -g... yes checking whether ln -s works... yes
checking if Fortran compiler works... yes checking for extra arguments
to build a shared library... none needed checking for Fortran flag to
compile .f files... none checking for Fortran flag to compile .f90
files... none checking to see if Fortran compilers need additional
linker flags... none checking  external symbol convention... single
underscore checking if C and Fortran are link compatible... yes
checking to see if Fortran compiler likes the C++ exception flags...
skipped (no C++ exceptions flags) checking to see if mpifort compiler
needs additional linker flags... none checking if Fortran compiler
supports CHARACTER... yes checking size of Fortran CHARACTER... 1
checking for C type corresponding to CHARACTER... char checking
alignment of Fortran CHARACTER... 1 checking for corresponding KIND
value of CHARACTER... C_SIGNED_CHAR checking KIND value of Fortran
C_SIGNED_CHAR... 1 checking if Fortran compiler supports LOGICAL...
yes checking size of Fortran LOGICAL... 4 checking for C type
corresponding to LOGICAL... int checking alignment of Fortran
LOGICAL... 4 checking for corresponding KIND value of LOGICAL... C_INT
checking KIND value of Fortran C_INT... 4 checking if Fortran compiler
supports LOGICAL*1... yes checking size of Fortran LOGICAL*1... 1
checking for C type corresponding to LOGICAL*1... char checking
alignment of Fortran LOGICAL*1... 1 checking for corresponding KIND
value of LOGICAL*1... C_SIGNED_CHAR checking KIND value of Fortran
C_SIGNED_CHAR... (cached) 1 checking if Fortran compiler supports
LOGICAL*2... yes checking size of Fortran LOGICAL*2... 2 checking for
C type corresponding to LOGICAL*2... short checking alignment of
Fortran LOGICAL*2... 2 checking for corresponding KIND value of
LOGICAL*2... C_SHORT checking KIND value of Fortran C_SHORT... 2
checking if Fortran compiler supports LOGICAL*4... yes checking size
of Fortran LOGICAL*4... 4 checking for C type corresponding to
LOGICAL*4... int checking alignment of Fortran LOGICAL*4... 4 checking
for corresponding KIND value of LOGICAL*4... C_INT checking KIND value
of Fortran C_INT... (cached) 4 checking if Fortran compiler supports
LOGICAL*8... yes checking size of Fortran LOGICAL*8... 8 checking for
C type corresponding to LOGICAL*8... long long checking alignment of
Fortran LOGICAL*8... 8 checking for corresponding KIND value of
LOGICAL*8... C_LONG_LONG checking KIND value of Fortran C_LONG_LONG...
8 checking if Fortran compiler supports INTEGER... yes checking size
of Fortran INTEGER... 4 checking for C type corresponding to
INTEGER... int checking alignment of Fortran INTEGER... 4 checking for
corresponding KIND value of INTEGER... C_INT checking KIND value of
Fortran C_INT... (cached) 4 checking if Fortran compiler supports
INTEGER*1... yes checking size of Fortran INTEGER*1... 1 checking for
C type corresponding to INTEGER*1... char checking alignment of
Fortran INTEGER*1... 1 checking for corresponding KIND value of
INTEGER*1... C_SIGNED_CHAR checking KIND value of Fortran
C_SIGNED_CHAR... (cached) 1 checking if Fortran compiler supports
INTEGER*2... yes checking size of Fortran INTEGER*2... 2 checking for
C type corresponding to INTEGER*2... short checking alignment of
Fortra

Re: [OMPI users] Configuring openib on openmpi 1.8.1

2014-07-30 Thread Gus Correa

Also, just in case, make sure the mpirun, ompi_info,
and other Open MPI (OMPI) commands you are using are the ones
that you installed.
I.e., not something from an RPM, or that may have come bundled with a 
compiler or other software.

Those may not have infiniband enabled, and in any case should not be mixed.
The OMPI implementations should be the same on all machines as well.
Running "which mpirun" on those machines may help.
These user enviroment problems often cause confusion.

My two cents,
Gus Correa

On 07/30/2014 09:56 AM, Ralph Castain wrote:

Does "polaris" have the same rpm's as the host where you checked in your
prior email?

Try adding "--level 9" to your ompi_info command line - the MCA param
system changed somewhat and the params may just not be getting shown by
default


On Jul 30, 2014, at 2:35 AM, Chaitra Kumar <chaitragku...@gmail.com
<mailto:chaitragku...@gmail.com>> wrote:


The command: 'ompi_info --param btl openib' doesnt return any openib
component.

When I try to use command like: ' mpirun *--mca btl self,sm,openib* ...'
it throws an error:
--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  polaris
Framework: btl
Component: openib
--

Regards,
Chaitra




On Wed, Jul 30, 2014 at 2:40 PM, Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>> wrote:

According to your output, you *do* have the IB components available:


 MCA btl: openib (MCA v2.0, API v2.0, Component
v1.8.1)


What made you think that you don't have them?


On Jul 30, 2014, at 12:10 AM, Chaitra Kumar
<chaitragku...@gmail.com <mailto:chaitragku...@gmail.com>> wrote:


Hi Howard,

The attached file "config,out" has the output of configure.

Output of ompi_info command:
 Package: Open MPI padmanac@polaris-4 Distribution
Open MPI: 1.8.1
  Open MPI repo revision: r31483
   Open MPI release date: Apr 22, 2014
Open RTE: 1.8.1
  Open RTE repo revision: r31483
   Open RTE release date: Apr 22, 2014
OPAL: 1.8.1
  OPAL repo revision: r31483
   OPAL release date: Apr 22, 2014
 MPI API: 3.0
Ident string: 1.8.1
  Prefix: /home/padmanac/openmpi181
 Configured architecture: x86_64-unknown-linux-gnu
  Configure host: polaris-4
   Configured by: padmanac
   Configured on: Tue Jul 29 11:41:12 PDT 2014
  Configure host: polaris-4
Built by: padmanac
Built on: Tue Jul 29 11:57:53 PDT 2014
  Built host: polaris-4
  C bindings: yes
C++ bindings: yes
 Fort mpif.h: yes (all)
Fort use mpi: yes (limited: overloading)
   Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: no
 Fort mpi_f08 compliance: The mpi_f08 module was not built
  Fort mpi_f08 subarrays: no
   Java bindings: no
  Wrapper compiler rpath: runpath
  C compiler: gcc
 C compiler absolute: /opt/gcc/bin/gcc
  C compiler family name: GNU
  C compiler version: 4.8.2
C++ compiler: g++
   C++ compiler absolute: /opt/gcc/bin/g++
   Fort compiler: gfortran
   Fort compiler abs: /opt/gcc/bin/gfortran
 Fort ignore TKR: no
   Fort 08 assumed shape: no
  Fort optional args: no
  Fort BIND(C) (all): no
  Fort ISO_C_BINDING: no
 Fort SUBROUTINE BIND(C): no
   Fort TYPE,BIND(C): no
 Fort T,BIND(C,name="a"): no
Fort PRIVATE: no
  Fort PROTECTED: no
   Fort ABSTRACT: no
   Fort ASYNCHRONOUS: no
  Fort PROCEDURE: no
 Fort f08 using wrappers: no
 C profiling: yes
   C++ profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: no
  C++ exceptions: no
  Thread support: posix (MPI_THREAD_MULTIPLE: no, OPAL
support: yes,
  OMPI progress: no, ORTE progress: yes,
Event lib:
  yes)
   Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
 MPI parameter check: runtime
Memory profiling support: no
 

Re: [OMPI users] mpifort wrapper.txt

2014-07-29 Thread Gus Correa

On 07/29/2014 07:20 AM, Jeff Squyres (jsquyres) wrote:

On Jul 27, 2014, at 3:39 PM, Dan Shell <dshellwirel...@gmail.com> wrote:


I have been looking at the openmpi doc page and would like some pointers on how 
to implement the wrapper.txt file with  mpifort.


I'm not sure what you're asking here...?


I have the wrapper .txt file how does mpifort use the wrapper.txt file.


Presuming you had a Fortran compiler available when you configured/built Open 
MPI, Open MPI should have created an mpifort-wrapper-data.txt file for you and 
installed it under $pkgdatadir.  I.e., you shouldn't have to create anything.

Did Open MPI not do this?

Please send the output listed here:

http://www.open-mpi.org/community/help/


Do I create a script copy the fortran wrapper.txt commands in the file to a
mkmf?  Not reall sure on how to proceed.  Any help would be appreciated.
See info below
Dan Shell

  make -f Make_lib_FMS
mpifort -Duse_netCDF -Duse_netCDF3 -Duse_libMPI -DUSE_OCEAN_BGC -DENABLE_ODA 
-DSPMD -DLAND_BND_TRACERS -I/usr/include 
-I/root/Desktop/NEW_MOM/openmpi/include -I/root/Desktop/NEW_MOM/newmom/netcdf 
-I/root/Desktop/NEW_MOM/newmom/netcdf/include   -fcray-pointer  -g  
-fdefault-real-8 -O2 -Waliasing -ffree-line-length-none -fno-range-check  -c 
-I/root/Desktop/NEW_MOM/newmom/src/shared/include 
-I/root/Desktop/NEW_MOM/newmom/src/shared/mpp/include
/root/Desktop/NEW_MOM/newmom/src/shared/mpp/mpp_data.F90
--
No underlying compiler was specified in the wrapper compiler data file
(e.g., mpicc-wrapper-data.txt)




The error message is complaining about mpicc, not mpifort.
I wonder if this may be due to a Makefile misconfiguration again.
My two cents,
Gus Correa


Re: [OMPI users] Trying to use openmpi with MOM getting a compile error

2014-07-25 Thread Gus Correa

On 07/25/2014 03:02 PM, Jeff Squyres (jsquyres) wrote:

On Jul 25, 2014, at 1:14 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:


Change the mkmf.template file and replace the Fortran
compiler name (gfortran) by the Open MPI (OMPI) Fortran compiler wrapper: 
mpifortran (or mpif90 if it still exists
in OMPI 1.8.1),


mpifort is the preferred Fortran compiler wrapper name in the 1.8.x series.
mpif90 still exists, but we'll likely remove that name in some future 
release

(not before 1.9.x, of course).




Hi Jeff

Oops! Sorry for my misspelling of mpifort.
(Intel must love this name choice! :) )
Well, hopefully Dan Shell found the right compiler wrapper.

I haven't got beyond the ol' mpif90 and OMPI 1.6.5.
Just waiting for 1.8.2 to be out in the sun to update.

Gus Correa


Re: [OMPI users] Trying to use openmpi with MOM getting a compile error

2014-07-25 Thread Gus Correa

Hi Dan

This is not an Open MPI problem.
This is most likely a problem with the MOM Makefile,
which seems to be missing your Open MPI include directory.

Change the mkmf.template file and replace the Fortran
compiler name (gfortran) by the Open MPI (OMPI) Fortran compiler 
wrapper: mpifortran (or mpif90 if it still exists

in OMPI 1.8.1), perhaps using the full path to it.
The mpifortran/mpif90 compiler wrapper knows exactly where to find
the Open MPI include files, the libraries, etc.
You may need to comment out or remove spurious entries
mkmf.template pointing to other MPI implementations (e.g.
to MPICH libraries and include files).

Then rebuild the Makefile and compile MOM again.

I hope this helps.
Gus Correa


On 07/25/2014 12:37 PM, Dan Shell wrote:

OpenMOM-mpi
I am trying to compile MOM and have installed openmpi 1.8.1 getting an
installation error below
Looking for some help in openmpi to make sure the mpif.h is loaded correctly
Should I use an older version of openmpi?
Thank You
Dan Shell

gfortran  -Duse_netCDF -Duse_netCDF3 -Duse_libMPI -DUSE_OCEAN_BGC
-DENABLE_ODA -DSPMD -DLAND_BND_TRACERS -c
-I/root/Desktop/NEW_MOM/newmom/src/shared/include
-I/root/Desktop/NEW_MOM/newmom/src/shared/mpp/include
/root/Desktop/NEW_MOM/newmom/src/shared/mpp/mpp_data.F90
/root/Desktop/NEW_MOM/newmom/src/shared/mpp/mpp_data.F90:23:

include 
1
Error: Unclassifiable statement at (1)
/root/Desktop/NEW_MOM/newmom/src/shared/mpp/include/mpp_data_mpi.inc:8.31:
 Included at
/root/Desktop/NEW_MOM/newmom/src/shared/mpp/mpp_data.F90:45:

integer :: stat(MPI_STATUS_SIZE)
1
Error: Symbol 'mpi_status_size' at (1) has no IMPLICIT type
/root/Desktop/NEW_MOM/newmom/src/shared/mpp/mpp_data.F90:28.16:

   public :: stat, mpp_stack, ptr_stack, status, ptr_status, sync, ptr_sync
 1
Error: The module or main program array 'stat' at (1) must have constant
shape
make: *** [mpp_data.o] Error 1
if ( 2 ) then
echo Make failed to create  lib_FMS.a
Make failed to create  lib_FMS.a



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/07/24876.php





Re: [OMPI users] configure fails to detect missing libcrypto

2014-07-24 Thread Gus Correa

Hi

In case this matters,
ldd on my Torque 4.2.5 libtorque.so shows:
libcrypto.so.10 => /usr/lib64/libcrypto.so.10
plus a few kerberos crypto libraries, etc.
May be because I built Torque with PAM module support?
Anyway, I have built OMPI up to 1.6.5 with that Torque support
with no problem.
Could your libcrypto be in an an unusual location?
Maybe you need to load a Torque environment module to add it to your 
LD_LIBRARY_PATH before you build OMPI?


Gus Correa


On 07/24/2014 05:18 PM, Jeff Hammond wrote:

That could be the case.  I've reported the missing libcrypto issue to
NERSC already.  But neither Intel MPI nor MVAPICH care about libcrypto
and they are both supporting PBS, so I'm not entirely convinced that
PBS depends upon it.

Thanks,

Jeff

Subject: Re: [OMPI users] configure fails to detect missing libcrypto
From: Ralph Castain (rhc_at_[hidden])
Date: 2014-07-24 17:12:16

Previous message: Jeff Hammond: "[OMPI users] configure fails to
detect missing libcrypto"
In reply to: Jeff Hammond: "[OMPI users] configure fails to detect
missing libcrypto"
I'm not aware of our tm module requiring libcrypto - is this something
specific to your PBS install, so it wants to pull libcrypto in when we
link against the Torque lib?

On Jul 24, 2014, at 1:49 PM, Jeff Hammond <jeff.science_at_[hidden]> wrote:

I am trying to build Open MPI SVN trunk on NERSC Babbage with PBS
support. configure completes without error but the build fails
because libcrypto.so is missing.

I consider it a desirable property that configure detect all the
necessary dependencies for a build to complete, rather than defer
errors to the compilation phase.

I will file a Trac ticket as soon as my account is reset (in-progress).

Making all in mca/plm/tm
make[2]: Entering directory
`/chos/global/u1/j/jhammond/MPI/ompi-trunk/build-intel/orte/mca/plm/tm'
CC plm_tm_component.lo
CC plm_tm_module.lo
CCLD mca_plm_tm.la
ld: cannot find -lcrypto
make[2]: *** [mca_plm_tm.la] Error 1
make[2]: Leaving directory
`/chos/global/u1/j/jhammond/MPI/ompi-trunk/build-intel/orte/mca/plm/tm'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory
`/chos/global/u1/j/jhammond/MPI/ompi-trunk/build-intel/orte'
make: *** [all-recursive] Error 1

Thanks,

Jeff

--
Jeff Hammond
jeff.science_at_[hidden]
http://jeffhammond.github.io/
___
users mailing list
users_at_[hidden]
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/07/24864.php






Re: [OMPI users] Problem moving from 1.4 to 1.6

2014-06-27 Thread Gus Correa
If you don't have control over the MPI version/versions/implementations 
installed, you probably can still verify if your environment is 
consistently pointing to the same MPI implementation and version.


It is not uncommon to have more than one implementation and version
installed on a computer, on a cluster, or worse, different 
versions+implementations on different cluster nodes.

Mixed-up environment variables can produce very confusing results.

Commands such as:

which mpiexec
which mpicc
which mpif90

and also

mpiexec --version
mpicc --show
etc

may help diagnose that.

Likewise,

env |grep PATH

and

env |grep LD_LIBRARY_PATH

may hint if you have a mixed environment and mixed MPI implementations 
and versions.


I hope this helps,
Gus Correa

PS - BTW, unless your company's policies forbid,
you can install OpenMPI on a user directory, say, your /home directory. 
 This will work if that directory is shared across the cluster (e.g. 
via NFS), and as long as you set your PATH and LD_LIBRARY_PATH to point 
to its bin and lib subdirectories.


https://www.open-mpi.org/faq/?category=running#adding-ompi-to-path

On 06/27/2014 01:56 PM, Jeffrey A Cummings wrote:

I appreciate your response and I understand the logic behind your
suggestion, but you and the other regular expert contributors to this
list are frequently working under a misapprehension.  Many of your
openMPI users don't have any control over what version of openMPI is
available on their system.  I'm stuck with whatever version my IT people
choose to bless, which in general is the (possibly old and/or moldy)
version that is bundled with some larger package (i.e., Rocks, Linux).
  The fact that I'm only now seeing this 1.4 to 1.6 problem illustrates
the situation I'm in.  I really need someone to did into their memory
archives to see if they can come up with a clue for me.

Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org



From: Gus Correa <g...@ldeo.columbia.edu>
To: Open MPI Users <us...@open-mpi.org>,
Date: 06/27/2014 01:45 PM
Subject: Re: [OMPI users] Problem moving from 1.4 to 1.6
Sent by: "users" <users-boun...@open-mpi.org>




It may be easier to install the latest OMPI from the tarball,
rather than trying to sort out the error.

http://www.open-mpi.org/software/ompi/v1.8/

The packaged built of (somewhat old) OMPI 1.6.2 that came with
Linux may not have built against the same IB libraries, hardware,
and configuration you have.
[The error message reference to udapl is ominous.]

 > The mpirun command line contains the argument '--mca btl ^openib', which
 > I thought told mpi to not look for the ib interface.

As you said, the mca parameter above tells OMPI not to use openib,
although it may not be the only cause of the problem.
If you want to use openib switch to
--mca btl openib,sm,self

Another thing to check is whether there is a mixup of enviroment
variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI
version you may have installed.

My two cents,
Gus Correa

On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:
 > We have recently upgraded our cluster to a version of Linux which comes
 > with openMPI version 1.6.2.
 >
 > An application which ran previously (using some version of 1.4) now
 > errors out with the following messages:
 >
 >  librdmacm: Fatal: no RDMA devices found
 >  librdmacm: Fatal: no RDMA devices found
 >  librdmacm: Fatal: no RDMA devices found
 >
 >
--
 >  WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
 >  This may be a real error or it may be an invalid entry in the
 > uDAPL
 >  Registry which is contained in the dat.conf file. Contact your
 > local
 >  System Administrator to confirm the availability of the
 > interfaces in
 >  the dat.conf file.
 >
 >
--
 >  [tupile:25363] 2 more processes have sent help message
 > help-mpi-btl-udapl.txt / dat_ia_open fail
 >  [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to
 > 0 to see all help / error messages
 >
 > The mpirun command line contains the argument '--mca btl ^openib', which
 > I thought told mpi to not look for the ib interface.
 >
 > Can anyone suggest what the problem might be?  Did the relevant syntax
 > change between versions 1.4 and 1.6?
 >
 >
 > Jeffrey A. Cummings
 > Engineering Specialist
 > Performance Modeling and Analysis Department
 > Systems An

Re: [OMPI users] Problem moving from 1.4 to 1.6

2014-06-27 Thread Gus Correa

It may be easier to install the latest OMPI from the tarball,
rather than trying to sort out the error.

http://www.open-mpi.org/software/ompi/v1.8/

The packaged built of (somewhat old) OMPI 1.6.2 that came with
Linux may not have built against the same IB libraries, hardware,
and configuration you have.
[The error message reference to udapl is ominous.]

> The mpirun command line contains the argument '--mca btl ^openib', which
> I thought told mpi to not look for the ib interface.

As you said, the mca parameter above tells OMPI not to use openib,
although it may not be the only cause of the problem.
If you want to use openib switch to
--mca btl openib,sm,self

Another thing to check is whether there is a mixup of enviroment 
variables, PATH and LD_LIBRARY_PATH perhaps pointing to the old OMPI 
version you may have installed.


My two cents,
Gus Correa

On 06/27/2014 12:53 PM, Jeffrey A Cummings wrote:

We have recently upgraded our cluster to a version of Linux which comes
with openMPI version 1.6.2.

An application which ran previously (using some version of 1.4) now
errors out with the following messages:

 librdmacm: Fatal: no RDMA devices found
 librdmacm: Fatal: no RDMA devices found
 librdmacm: Fatal: no RDMA devices found

--
 WARNING: Failed to open "OpenIB-cma" [DAT_INTERNAL_ERROR:].
 This may be a real error or it may be an invalid entry in the
uDAPL
 Registry which is contained in the dat.conf file. Contact your
local
 System Administrator to confirm the availability of the
interfaces in
 the dat.conf file.

--
 [tupile:25363] 2 more processes have sent help message
help-mpi-btl-udapl.txt / dat_ia_open fail
 [tupile:25363] Set MCA parameter "orte_base_help_aggregate" to
0 to see all help / error messages

The mpirun command line contains the argument '--mca btl ^openib', which
I thought told mpi to not look for the ib interface.

Can anyone suggest what the problem might be?  Did the relevant syntax
change between versions 1.4 and 1.6?


Jeffrey A. Cummings
Engineering Specialist
Performance Modeling and Analysis Department
Systems Analysis and Simulation Subdivision
Systems Engineering Division
Engineering and Technology Group
The Aerospace Corporation
571-307-4220
jeffrey.a.cummi...@aero.org


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/06/24721.php





Re: [OMPI users] openib segfaults with Torque

2014-06-11 Thread Gus Correa

If that could help Greg,
on the compute nodes I normally add this to /etc/security/limits.conf:

*   -   memlock -1
*   -   stack   -1
*   -   nofile  32768

and

ulimit -n 32768
ulimit -l unlimited
ulimit -s unlimited

to either /etc/init.d/pbs_mom or to /etc/sysconfig/pbs_mom (which
should be sourced by the former).
Other values are possible, of course.

My recollection is that the boilerplate init scripts that
come with Torque don't change those limits.

I suppose this makes the pbs_mom child processes,
including the user job script and whatever processes it starts
(mpiexec, etc), to inherit those limits.
Or not?

Gus Correa


On 06/11/2014 06:20 PM, Jeff Squyres (jsquyres) wrote:

+1

On Jun 11, 2014, at 6:01 PM, Ralph Castain <r...@open-mpi.org>
  wrote:


Yeah, I think we've seen that somewhere before too...


On Jun 11, 2014, at 2:59 PM, Joshua Ladd <jladd.m...@gmail.com> wrote:


Agreed. The problem is not with UDCM. I don't think something is wrong with the 
system. I think his Torque is imposing major constraints on the maximum size 
that can be locked into memory.

Josh


On Wed, Jun 11, 2014 at 5:49 PM, Nathan Hjelm <hje...@lanl.gov> wrote:
Probably won't help to use RDMACM though as you will just see the
resource failure somewhere else. UDCM is not the problem. Something is
wrong with the system. Allocating a 512 entry CQ should not fail.

-Nathan

On Wed, Jun 11, 2014 at 05:03:31PM -0400, Joshua Ladd wrote:

I'm guessing it's a resource limitation issue coming from Torque.

H...I found something interesting on the interwebs that looks awfully
similar:
http://www.supercluster.org/pipermail/torqueusers/2008-February/006916.html

Greg, if the suggestion from the Torque users doesn't resolve your issue (
"...adding the following line 'ulimit -l unlimited' to pbs_mom and
restarting pbs_mom." ) doesn't work, try using the RDMACM CPC (instead of
UDCM, which is a pretty recent addition to the openIB BTL.) by setting:

-mca btl_openib_cpc_include rdmacm

Josh

On Wed, Jun 11, 2014 at 4:04 PM, Jeff Squyres (jsquyres)
<jsquy...@cisco.com> wrote:

  Mellanox --

  What would cause a CQ to fail to be created?

  On Jun 11, 2014, at 3:42 PM, "Fischer, Greg A."
  <fisch...@westinghouse.com> wrote:

  > Is there any other work around that I might try?  Something that
  avoids UDCM?
  >
  > -Original Message-
  > From: Fischer, Greg A.
  > Sent: Tuesday, June 10, 2014 2:59 PM
  > To: Nathan Hjelm
  > Cc: Open MPI Users; Fischer, Greg A.
  > Subject: RE: [OMPI users] openib segfaults with Torque
  >
  > [binf316:fischega] $ ulimit -m
  > unlimited
  >
  > Greg
  >
  > -Original Message-
  > From: Nathan Hjelm [mailto:hje...@lanl.gov]
  > Sent: Tuesday, June 10, 2014 2:58 PM
  > To: Fischer, Greg A.
  > Cc: Open MPI Users
  > Subject: Re: [OMPI users] openib segfaults with Torque
  >
  > Out of curiosity what is the mlock limit on your system? If it is too
  low that can cause ibv_create_cq to fail. To check run ulimit -m.
  >
  > -Nathan Hjelm
  > Application Readiness, HPC-5, LANL
  >
  > On Tue, Jun 10, 2014 at 02:53:58PM -0400, Fischer, Greg A. wrote:
  >> Yes, this fails on all nodes on the system, except for the head node.
  >>
  >> The uptime of the system isn't significant. Maybe 1 week, and it's
  received basically no use.
  >>
  >> -Original Message-
  >> From: Nathan Hjelm [mailto:hje...@lanl.gov]
  >> Sent: Tuesday, June 10, 2014 2:49 PM
  >> To: Fischer, Greg A.
  >> Cc: Open MPI Users
  >> Subject: Re: [OMPI users] openib segfaults with Torque
  >>
  >>
  >> Well, thats interesting. The output shows that ibv_create_cq is
  failing. Strange since an identical call had just succeeded (udcm
  creates two completion queues). Some questions that might indicate where
  the failure might be:
  >>
  >> Does this fail on any other node in your system?
  >>
  >> How long has the node been up?
  >>
  >> -Nathan Hjelm
  >> Application Readiness, HPC-5, LANL
  >>
  >> On Tue, Jun 10, 2014 at 02:06:54PM -0400, Fischer, Greg A. wrote:
  >>> Jeff/Nathan,
  >>>
  >>> I ran the following with my debug build of OpenMPI 1.8.1 - after
  opening a terminal on a compute node with "qsub -l nodes 2 -I":
  >>>
  >>>  mpirun -mca btl openib,self -mca btl_base_verbose 100 -np 2
  >>> ring_c &> output.txt
  >>>
  >>&g

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Gus Correa

On 06/06/2014 01:05 PM, Ralph Castain wrote:

You can always add --display-allocation to the cmd line to see what we
thought we received.

If you configure OMPI with --enable-debug, you can set --mca
ras_base_verbose 10 to see the details




Hi John

On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
script.  This will list the nodes (multiple times according to the 
number of cores requested).

I find this useful documentation,
along with job number, work directory, etc.
"man qsub" will show you all the PBS_* environment variables
available to the job.
For instance, you can echo them using a Torque
'prolog' script, if the user
didn't do it. That will appear in the Torque STDOUT file.

From outside the job script, "qstat -n" (and variants, say, with -u 
username)

will list the nodes allocated to each job,
again multiple times as per the requested cores.

"tracejob job_number" will show similar information.


If you configured Torque --with-cpuset,
there is more information about the cpuset allocated to the job
in /dev/cpuset/torque/jobnumber (on the first node listed above, called 
"mother superior" in Torque parlance).

This mostly matter if there is more than one job running on a node.
However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
whatever.  As Ralph said, Open MPI does that.

I believe Open MPI doesn't use the cpuset info from Torque.
(Ralph, please correct me if I am wrong.)

My two cents,
Gus Correa



On Jun 6, 2014, at 10:01 AM, Reuti <re...@staff.uni-marburg.de
<mailto:re...@staff.uni-marburg.de>> wrote:


Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):


OK, so at the least, how can I get the node and slots/node info that
is passed from PBS?

I ask because I’m trying to troubleshoot a problem w/ PBS and the
build of OpenMPI 1.6 I noted.  If I submit a 24-process simple job
through PBS using a script which has:

/usr/local/openmpi/bin/orterun -n 24 --hostfile
/home/sasso/TEST/hosts.file --mca orte_rsh_agent rsh --mca btl
openib,tcp,self --mca orte_base_help_aggregate 0 -x PATH -x
LD_LIBRARY_PATH /home/sasso/TEST/simplempihello.exe


Using the --hostfile on your own would mean to violate the granted
slot allocation by PBS. Just leave this option out. How do you submit
your job?

-- Reuti



And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the
first 16 being host node0001 and the last 8 being node0002), it
appears that 24 MPI tasks try to start on node0001 instead of getting
distributed as 16 on node0001 and 8 on node0002.   Hence, I am
curious what is being passed by PBS.

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph
Castain
Sent: Friday, June 06, 2014 12:31 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler
passes to OpenMPI

We currently only get the node and slots/node info from PBS - we
don't get any task placement info at all. We then use the mpirun cmd
options and built-in mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't
really seen a reason to do so - the OMPI mappers are generally more
flexible than anything in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE)
<john1.sa...@ge.com <mailto:john1.sa...@ge.com>> wrote:


For the PBS scheduler and using a build of OpenMPI 1.6 built against
PBS include files + libs, is there a way to determine (perhaps via
some debugging flags passed to mpirun) what job placement parameters
are passed from the PBS scheduler to OpenMPI?  In particular, I am
talking about task placement info such as nodes to place on, etc.
  Thanks!

 --john
___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users





Re: [OMPI users] intermittent segfaults with openib on ring_c.c

2014-06-04 Thread Gus Correa

Hi Greg

From your original email:

>> [binf102:fischega] $ mpirun -np 2 --mca btl openib,self ring_c

This may not fix the problem,
but have you tried to add the shared memory btl to your mca parameter?

mpirun -np 2 --mca btl openib,sm,self ring_c

As far as I know, sm is the preferred transport layer for intra-node
communication.

Gus Correa


On 06/04/2014 11:13 AM, Ralph Castain wrote:

Thanks!! Really appreciate your help - I'll try to figure out what went
wrong and get back to you

On Jun 4, 2014, at 8:07 AM, Fischer, Greg A. <fisch...@westinghouse.com
<mailto:fisch...@westinghouse.com>> wrote:


I re-ran with 1 processor and got more information. How about this?
Core was generated by `ring_c'.
Program terminated with signal 11, Segmentation fault.
#0  opal_memory_ptmalloc2_int_malloc (av=0x2b48f6300020,
bytes=47592367980728) at
../../../../../openmpi-1.8.1/opal/mca/memory/linux/malloc.c:4098
4098  bck->fd = unsorted_chunks(av);
(gdb) bt
#0  opal_memory_ptmalloc2_int_malloc (av=0x2b48f6300020,
bytes=47592367980728) at
../../../../../openmpi-1.8.1/opal/mca/memory/linux/malloc.c:4098
#1  0x2b48f2a15e38 in opal_memory_ptmalloc2_malloc
(bytes=47592367980576) at
../../../../../openmpi-1.8.1/opal/mca/memory/linux/malloc.c:3433
#2  0x2b48f2a15b36 in opal_memory_linux_malloc_hook
(sz=47592367980576, caller=0x2b48f63000b8) at
../../../../../openmpi-1.8.1/opal/mca/memory/linux/hooks.c:691
#3  0x2b48f2374b90 in vasprintf () from /lib64/libc.so.6
#4  0x2b48f2354148 in asprintf () from /lib64/libc.so.6
#5  0x2b48f26dc7d1 in orte_oob_base_get_addr (uri=0x2b48f6300020)
at ../../../../openmpi-1.8.1/orte/mca/oob/base/oob_base_stubs.c:234
#6  0x2b48f53e7d4a in orte_rml_oob_get_uri () at
../../../../../openmpi-1.8.1/orte/mca/rml/oob/rml_oob_contact.c:36
#7  0x2b48f26fa181 in orte_routed_base_register_sync (setup=32 '
') at ../../../../openmpi-1.8.1/orte/mca/routed/base/routed_base_fns.c:301
#8  0x2b48f4bbcccf in init_routes (job=4130340896,
ndat=0x2b48f63000b8) at
../../../../../openmpi-1.8.1/orte/mca/routed/binomial/routed_binomial.c:705
#9  0x2b48f26c615d in orte_ess_base_app_setup
(db_restrict_local=32 ' ') at
../../../../openmpi-1.8.1/orte/mca/ess/base/ess_base_std_app.c:245
#10 0x2b48f45b069f in rte_init () at
../../../../../openmpi-1.8.1/orte/mca/ess/env/ess_env_module.c:146
#11 0x2b48f26935ab in orte_init (pargc=0x2b48f6300020,
pargv=0x2b48f63000b8, flags=8) at
../../openmpi-1.8.1/orte/runtime/orte_init.c:148
#12 0x2b48f1739d38 in ompi_mpi_init (argc=1, argv=0x7fffebf0d1f8,
requested=8, provided=0x0) at
../../openmpi-1.8.1/ompi/runtime/ompi_mpi_init.c:464
#13 0x2b48f1760a37 in PMPI_Init (argc=0x2b48f6300020,
argv=0x2b48f63000b8) at pinit.c:84
#14 0x004024ef in main (argc=1, argv=0x7fffebf0d1f8) at
ring_c.c:19
*From:*users [mailto:users-boun...@open-mpi.org]*On Behalf Of*Ralph
Castain
*Sent:*Wednesday, June 04, 2014 11:00 AM
*To:*Open MPI Users
*Subject:*Re: [OMPI users] intermittent segfaults with openib on ring_c.c
Does the trace go any further back? Your prior trace seemed to
indicate an error in our OOB framework, but in a very basic place.
Looks like it could be an uninitialized variable, and having the line
number down as deep as possible might help identify the source
On Jun 4, 2014, at 7:55 AM, Fischer, Greg A.
<fisch...@westinghouse.com <mailto:fisch...@westinghouse.com>> wrote:


Oops, ulimit was set improperly. I generated a core file, loaded it in
GDB, and ran a backtrace:
Core was generated by `ring_c'.
Program terminated with signal 11, Segmentation fault.
#0  opal_memory_ptmalloc2_int_malloc (av=0x2b8e4fd00020,
bytes=47890224382136) at
../../../../../openmpi-1.8.1/opal/mca/memory/linux/malloc.c:4098
4098  bck->fd = unsorted_chunks(av);
(gdb) bt
#0  opal_memory_ptmalloc2_int_malloc (av=0x2b8e4fd00020,
bytes=47890224382136) at
../../../../../openmpi-1.8.1/opal/mca/memory/linux/malloc.c:4098
#1  0x in ?? ()
Is that helpful?
Greg
*From:*Fischer, Greg A.
*Sent:*Wednesday, June 04, 2014 10:17 AM
*To:*'Open MPI Users'
*Cc:*Fischer, Greg A.
*Subject:*RE: [OMPI users] intermittent segfaults with openib on ring_c.c
I recompiled with “—enable-debug” but it doesn’t seem to be providing
any more information or a core dump. I’m compiling ring.c with:
mpicc ring_c.c -g -traceback -o ring_c
and running with:
mpirun -np 4 --mca btl openib,self ring_c
and I’m getting:
[binf112:05845] *** Process received signal ***
[binf112:05845] Signal: Segmentation fault (11)
[binf112:05845] Signal code: Address not mapped (1)
[binf112:05845] Failing at address: 0x10
[binf112:05845] [ 0] /lib64/libpthread.so.0(+0xf7c0)[0x2b2fa44d57c0]
[binf112:05845] [ 1]
//_ib/intel-12.1.0.233/toolset/openmpi-1.8.1/lib/libopen-pal.so.6(opal_memory_ptmalloc2_int_malloc+0x4b3)[0x2b2fa4ff2b03]
[binf112:05845] [ 2]
//_ib/intel-12.1.0.233/toolset/openmpi-1.8.1/lib/libopen-pal.so.6(opal_memory_ptmalloc2_m

Re: [OMPI users] openmpi configuration error?

2014-05-21 Thread Gus Correa

Hi Ben

One of the ranks (52) called MPI_Abort.
This may be a bug in the code, or a problem with the setup
(e.g. a missing or incorrect input file).
For instance, the CCTM Wiki says:
"AERO6 expects emissions inputs for 13 new PM species. CCTM will crash 
if any emitted PM species is not included in the emissions input file"

I am not familiar to CCTM, so these are just guesses.

It doesn't look like an MPI problem, though.

You may want to check any other logs that the CCTM code may
produce, for any clue on where it fails.
Otherwise, you could compile with -g -traceback (and remove any
optimization options in FFLAGS, FCFLAGS, CFLAGS, etc.)
It may also have a -DDEBUG or similar that can be turned on
in the CPPFLAGS, which in many models makes a more verbose log.
This *may* tell you where it fails (source file, subroutine and line),
and may help understand why it fails.
If it dumps a core file, you can trace the failure point with
a debugger.

I hope this helps,
Gus

On 05/21/2014 03:20 PM, Ben Lash wrote:

I used a different build of netcdf 4.1.3, and the code seems to run now.
I have a totally different, non-mpi related error in part of it, but
there's no way for the list to help, I mostly just wanted to report that
this particular problem seems to be solved for the record. It doesn't
seem to fail quite as gracefully anymore, but I'm still getting enough
of the error messages to know what's going on.

MPI_ABORT was invoked on rank 52 in communicator MPI_COMM_WORLD
with errorcode 0.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],52] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],54] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],55] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-158.davinci.rice.edu:12459 <http://cn-158.davinci.rice.edu:12459>]
[[63355,0],1]-[[63355,1],15] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-158.davinci.rice.edu:12459 <http://cn-158.davinci.rice.edu:12459>]
[[63355,0],1]-[[63355,1],17] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],56] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],53] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],51] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
[cn-099.davinci.rice.edu:26185 <http://cn-099.davinci.rice.edu:26185>]
[[63355,0],4]-[[63355,1],57] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
forrtl: error (78): process killed (SIGTERM)
Image  PCRoutineLineSource



[cn-158.davinci.rice.edu:12459 <http://cn-158.davinci.rice.edu:12459>]
[[63355,0],1]-[[63355,1],16] mca_oob_tcp_msg_recv: readv failed:
Connection reset by peer (104)
--
mpiexec has exited due to process rank 49 with PID 26187 on
node cn-099 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).
--
forrtl: error (78): process killed (SIGTERM)
Image  PCRoutineLineSource
CCTM_V5g_Linux2_x  007FEA29  Unknown   Unknown  Unknown
CCTM_V5g_Linux2_x  007FD3A0  Unknown   Unknown  Unknown
CCTM_V5g_Linux2_x  007BA9A2  Unknown   Unknown  Unknown
CCTM_V5g_Linux2_x  00759288  Unknown   Unknown  Un

Re: [OMPI users] openmpi configuration error?

2014-05-21 Thread Gus Correa

Hi Ben

My guess is that your sys admins may have built NetCDF
with parallel support, pnetcdf, and the latter with OpenMPI,
which could explain the dependency.
Ideally, they should have built it again with the latest default OpenMPI 
(1.6.5?)


Check if there is a NetCDF module that either doesn't have any
dependence on MPI, or depends on the current Open MPI that
you are using (1.6.5 I think).
A  'module show netcdf/bla/bla'
on the available netcdf modules will tell.

If the application code is old as you said, it probably doesn't use
any pnetcdf. In addition, it should work even with NetCDF 3.X.Y,
which probably doesn't have any pnetcdf built in.
Newer netcdf (4.Z.W > 4.1.3) should also work, and in this case
pick one that requires the default OpenMPI, if available.

Just out of curiosity, besides netcdf/4.1.3, did you load openmpi/1.6.5?
Somehow the openmpi/1.6.5 should have been marked
to conflict with 1.4.4.
Is it?
Anyway, you may want to do a 'which mpiexec' to see which one is
taking precedence in your environment (1.6.5 or 1.4.4)
Probably 1.6.5.

Does the code work now, or does it continue to fail?

I hope this helps,
Gus Correa



On 05/21/2014 02:36 PM, Ben Lash wrote:

Yep, there is is.

[bl10@login2 USlogsminus10]$ module show netcdf/4.1.3
---
/opt/apps/modulefiles/netcdf/4.1.3:

module   load openmpi/1.4.4-intel
prepend-path PATH
/opt/apps/netcdf/4.1.3/bin:/opt/apps/netcdf/4.1.3/deps/hdf5/1.8.7/bin
prepend-path LD_LIBRARY_PATH
/opt/apps/netcdf/4.1.3/lib:/opt/apps/netcdf/4.1.3/deps/hdf5/1.8.7/lib:/opt/apps/netcdf/4.1.3/deps/szip/2.1/lib

prepend-path MANPATH /opt/apps/netcdf/4.1.3/share/man
---



On Wed, May 21, 2014 at 1:34 PM, Douglas L Reeder <d...@centurylink.net
<mailto:d...@centurylink.net>> wrote:

Ben,

The netcdf/4.1.3 module maybe loading the openmpi/1.4.4 module. Can
you do module show the netcdf module file to to see if there is a
module load openmpi command.

Doug Reeder

On May 21, 2014, at 12:23 PM, Ben Lash <b...@rice.edu
<mailto:b...@rice.edu>> wrote:


I just wanted to follow up for anyone else who got a similar
problem - module load netcdf/4.1.3 *also* loaded openmpi/1.4.4.
<http://1.4.4./> Don't ask me why. My code doesn't seem to fail as
gracefully but otherwise works now. Thanks.


On Sat, May 17, 2014 at 6:02 AM, Jeff Squyres (jsquyres)
<jsquy...@cisco.com <mailto:jsquy...@cisco.com>> wrote:

Ditto -- Lmod looks pretty cool.  Thanks for the heads up.


On May 16, 2014, at 6:23 PM, Douglas L Reeder
<d...@centurylink.net <mailto:d...@centurylink.net>> wrote:

> Maxime,
>
> I was unaware of Lmod. Thanks for bringing it to my attention.
>
> Doug
> On May 16, 2014, at 4:07 PM, Maxime Boissonneault
<maxime.boissonnea...@calculquebec.ca
<mailto:maxime.boissonnea...@calculquebec.ca>> wrote:
>
>> Instead of using the outdated and not maintained Module
environment, why not use Lmod :
https://www.tacc.utexas.edu/tacc-projects/lmod
>>
>> It is a drop-in replacement for Module environment that
supports all of their features and much, much more, such as :
>> - module hierarchies
>> - module properties and color highlighting (we use it to
higlight bioinformatic modules or tools for example)
>> - module caching (very useful for a parallel filesystem
with tons of modules)
>> - path priorities (useful to make sure personal modules
take precendence over system modules)
>> - export module tree to json
>>
>> It works like a charm, understand both TCL and Lua modules
and is actively developped and debugged. There are litteraly
new features every month or so. If it does not do what you
want, odds are that the developper will add it shortly (I've
had it happen).
>>
>> Maxime
>>
>> Le 2014-05-16 17:58, Douglas L Reeder a écrit :
>>> Ben,
>>>
>>> You might want to use module (source forge) to manage
paths to different mpi implementations. It is fairly easy to
set up and very robust for this type of problem. You would
remove contentious application paths from you standard PATH
and then use module to switch them in and out as needed.
>>>
>>> Doug Reeder
>>> On May 16, 2014, at 3:39 PM, Ben Lash <b...@rice.edu
<mailto:b...@rice.edu>> wrote:
>>>
>>>>

Re: [OMPI users] openmpi configuration error?

2014-05-16 Thread Gus Correa

On 05/16/2014 07:09 PM, Ben Lash wrote:

The $PATH and $LD_LIBRARY_PATH seem to be correct, as does module list.
I will try to hear back from our particular cluster people, otherwise I
will try using the latest version. This is old government software,
significant parts are written in fortran77 for example, typically
upgrading to a new version breaks it. It was looking for mpich, hence
the link, but a long time ago I gave it openmpi instead as recommended
and that worked, so I suppose it's less persnickety about the mpi
version than some other things. The most current version installed
is openmpi/1.6.5-intel(default). Thanks again.



We have code here that has been recompiled (some with modifications, 
some not) with OpenMPI since 1.2.8 with no problems.
MPI is a standard, both OpenMPI and MPICH follow the standard (except 
perhaps for very dusty corners or latest greatest MPI 3 features).

If your code compiled with 1.4.4, it should (better) do with 1.6.5.
Fortran77 shouldn't be an issue.

I agree, the PATH and LD_LIBRARY_PATH point to the "retired" directory.
Many things may have happened, though, say, the "retired" directory may 
not be complete, or may not have been installed on all cluster nodes,
or (if not really re-installed) probably points to the original 
(pre-retirement) directories that no longer exist.

Rather than sorting this out, I think you have a better shot using
Open MPI 1.6.5.
Just load the module and try to recompile the code.
(Probably just
module swap openmpi/1.4.4-intel openmpi/1.6.5-intel)

You may need to tweak with the Makefile, if it hardwires
the MPI wrappers/binary location, or the library and include paths.
Some do, some don't.

Gus Correa



[bl10@login2 ~]$ echo $PATH
/home/bl10/rlib/deps/bin:/opt/apps/netcdf/4.1.3/bin:/opt/apps/netcdf/4.1.3/deps/hdf5/1.8.7/bin:/opt/apps/openmpi/retired/1.4.4-intel/bin:/opt/apps/pgi/11.7/linux86-64/11.7/bin:/opt/apps/python3/3.2.1/bin:/opt/apps/intel/2013.1.039/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/apps/moab/current/bin:/projects/dsc1/apps/cmaq/deps/ioapi-kiran/3.1/bin:/home/bl10/bin

[bl10@login2 ~]$ echo $LD_LIBRARY_PATH
/home/bl10/rlib/deps/lib:/projects/dsc1/apps/cmaq/deps/netcdf/4.1.3-intel/lib:/opt/apps/netcdf/4.1.3/lib:/opt/apps/netcdf/4.1.3/deps/hdf5/1.8.7/lib:/opt/apps/netcdf/4.1.3/deps/szip/2.1/lib:/opt/apps/openmpi/retired/1.4.4-intel/lib:/opt/apps/intel/2011.0.013/mkl/lib/intel64:/opt/apps/intel/2013.1.039/mkl/lib/intel64:/opt/apps/intel/2013.1.039/lib/intel64

[bl10@login2 ~]$ module list
Currently Loaded Modulefiles:
   1) intel/2013.1.039  2) python3/3.2.1 3) pgi/11.7
  4) openmpi/1.4.4-intel   5) netcdf/4.1.3
[bl10@login2 ~]$




On Fri, May 16, 2014 at 5:46 PM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:

On 05/16/2014 06:26 PM, Ben Lash wrote:

I'm not sure I have the ability to implement a different module
management system, I am using a university cluster. We have a module
system, and I am beginning to suspect that maybe it wasn't updated
during the upgrade. I have
module list
..other modulesopenmpi/1.4.4
Perhaps this is still trying to go to the old source location?
How would
I check? Is there an easy way around it if it is wrong? Thanks
again!


Most likely the module openmpi/1.4.4 is out of date.
You can check it with
echo $PATH
If it doesn't point to the "retired" directory, then it is probably
out of date.

Why don't you try to recompile the code
with the current Open MPI installed in the cluster?

module avail
will show everyting, and you can pick the latest, load it,
and try to recompile the program with that.

Gus Correa


On Fri, May 16, 2014 at 5:07 PM, Maxime Boissonneault
<maxime.boissonneault@__calculquebec.ca
<mailto:maxime.boissonnea...@calculquebec.ca>
<mailto:maxime.boissonneault@__calculquebec.ca
<mailto:maxime.boissonnea...@calculquebec.ca>>> wrote:

 Instead of using the outdated and not maintained Module
environment,
 why not use Lmod :
https://www.tacc.utexas.edu/__tacc-projects/lmod
<https://www.tacc.utexas.edu/tacc-projects/lmod>

 It is a drop-in replacement for Module environment that
supports all
 of their features and much, much more, such as :
 - module hierarchies
 - module properties and color highlighting (we use it to
higlight
 bioinformatic modules or tools for example)
 - module caching (very useful for a parallel filesystem
with tons of
 modules)
 - path priorities (useful to make sure personal modules take
 precendence over system modules)
 - export modu

Re: [OMPI users] openmpi configuration error?

2014-05-16 Thread Gus Correa

On 05/16/2014 06:26 PM, Ben Lash wrote:

I'm not sure I have the ability to implement a different module
management system, I am using a university cluster. We have a module
system, and I am beginning to suspect that maybe it wasn't updated
during the upgrade. I have
module list
..other modulesopenmpi/1.4.4
Perhaps this is still trying to go to the old source location? How would
I check? Is there an easy way around it if it is wrong? Thanks again!



Most likely the module openmpi/1.4.4 is out of date.
You can check it with
echo $PATH
If it doesn't point to the "retired" directory, then it is probably out 
of date.


Why don't you try to recompile the code
with the current Open MPI installed in the cluster?

module avail
will show everyting, and you can pick the latest, load it,
and try to recompile the program with that.

Gus Correa



On Fri, May 16, 2014 at 5:07 PM, Maxime Boissonneault
<maxime.boissonnea...@calculquebec.ca
<mailto:maxime.boissonnea...@calculquebec.ca>> wrote:

Instead of using the outdated and not maintained Module environment,
why not use Lmod : https://www.tacc.utexas.edu/tacc-projects/lmod

It is a drop-in replacement for Module environment that supports all
of their features and much, much more, such as :
- module hierarchies
- module properties and color highlighting (we use it to higlight
bioinformatic modules or tools for example)
- module caching (very useful for a parallel filesystem with tons of
modules)
- path priorities (useful to make sure personal modules take
precendence over system modules)
- export module tree to json

It works like a charm, understand both TCL and Lua modules and is
actively developped and debugged. There are litteraly new features
every month or so. If it does not do what you want, odds are that
the developper will add it shortly (I've had it happen).

Maxime

Le 2014-05-16 17:58, Douglas L Reeder a écrit :

Ben,

You might want to use module (source forge) to manage paths to
different mpi implementations. It is fairly easy to set up and
very robust for this type of problem. You would remove contentious
application paths from you standard PATH and then use module to
switch them in and out as needed.

Doug Reeder
On May 16, 2014, at 3:39 PM, Ben Lash <b...@rice.edu
<mailto:b...@rice.edu>> wrote:


My cluster has just upgraded to a new version of MPI, and I'm
using an old one. It seems that I'm having trouble compiling due
to the compiler wrapper file moving (full error here:
http://pastebin.com/EmwRvCd9)
"Cannot open configuration file
/opt/apps/openmpi/1.4.4-intel/share/openmpi/mpif90-wrapper-data.txt"

I've found the file on the cluster at
 /opt/apps/openmpi/retired/1.4.4-intel/share/openmpi/mpif90-wrapper-data.txt
How do I tell the old mpi wrapper where this file is?
I've already corrected one link to mpich ->
/opt/apps/openmpi/retired/1.4.4-intel/, which is in the software
I'm trying to recompile's lib folder
(/home/bl10/CMAQv5.0.1/lib/x86_64/ifort). Thanks for any ideas. I
also tried changing $pkgdatadir based on what I read here:

http://www.open-mpi.org/faq/?category=mpi-apps#default-wrapper-compiler-flags


Thanks.

--Ben L
___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users




___
users mailing list
us...@open-mpi.org  <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
-
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Ph. D. en physique


___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users




--
--Ben L


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users





Re: [OMPI users] openmpi configuration error?

2014-05-16 Thread Gus Correa

Hi Ben

I guess you are not particularly interested on Modules or LMod either.
You probably don't administer this cluster,
but are just trying to have MPI working, right?


Are you trying to use Open MPI or MPICH?
Your email mentions both.
Let's assume it is Open MPI.

Is recompiling your code with the new version of Open MPI installed
in the cluster an option for you, or not?
If it is, that may be the simplest and best solution.

If the cluster has a system administrator, you may ask him about
the best way to set up your environment variables.

In any case, for your Open MPI to work you need your PATH and 
LD_LIBRARY_PATH to be correctly set.
If the system administator migrated OpenMPI 1.4.4 to a "retired" 
directory, you may need to adjust PATH and LD_LIBRARY_PATH

to point to the "retired" directories (if they are still functional).

echo $PATH
echo $LD_LIBRARY_PATH

may show what you have.

Does the cluster use modules, perhaps?
What do you get from

module list

?

As a workaround you're using bash, you could add these to .bashrc

export PATH=/opt/apps/openmpi/retired/1.4.4-intel/bin:$PATH
export 
LD_LIBRARY_PATH=/opt/apps/openmpi/retired/1.4.4-intel/lib:$LD_LIBRARY_PATH


if csh to .cshrc

setenv PATH /opt/apps/openmpi/retired/1.4.4-intel/bin:$PATH
setenv LD_LIBRARY_PATH 
/opt/apps/openmpi/retired/1.4.4-intel/lib:$LD_LIBRARY_PATH


I hope this helps,
Gus Correa



On 05/16/2014 05:39 PM, Ben Lash wrote:

My cluster has just upgraded to a new version of MPI, and I'm using an
old one. It seems that I'm having trouble compiling due to the compiler
wrapper file moving (full error here: http://pastebin.com/EmwRvCd9)
"Cannot open configuration file
/opt/apps/openmpi/1.4.4-intel/share/openmpi/mpif90-wrapper-data.txt"

I've found the file on the cluster at
  /opt/apps/openmpi/retired/1.4.4-intel/share/openmpi/mpif90-wrapper-data.txt
How do I tell the old mpi wrapper where this file is?
I've already corrected one link to mpich ->
/opt/apps/openmpi/retired/1.4.4-intel/, which is in the software I'm
trying to recompile's lib folder
(/home/bl10/CMAQv5.0.1/lib/x86_64/ifort). Thanks for any ideas. I also
tried changing $pkgdatadir based on what I read here:
http://www.open-mpi.org/faq/?category=mpi-apps#default-wrapper-compiler-flags

Thanks.

--Ben L


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users





Re: [OMPI users] Question about scheduler support

2014-05-16 Thread Gus Correa

++1

The original issue was that OMPI builds support for slurm
and loadlevler by default, and this was not desirable (or desired).

That is a non-issue.
If you don't want slurm and loadleveler support,
just configure OMPI

--with-slurm=no --with-loadleveler=no

All other supported schedulers can be dismissed by similar configure 
flags, if one is strict about having a slim OMPI installation.
For those who love the minutiae, 'configure --help' will show all 
options, including those above,

and I am surprised that this was not noticed first place (and used)
by those complaining of the default support
to slurm and loadleveler on OMPI.

So, why the fuss if the solution is so simple?

I am happy with the way OMPI builds.
For me the main goal is to provide a reliable and flexible MPI build
to support parallel programs, not to fiddle with the build system.
Given the small number
of complaints about the OMPI build system on this list so far
(was there any before this one?),
I would guess most OMPI users also are happy with its build system.
We have GigE, Infiniband, and Torque: OMPI picks them up and
works perfectly with them.
We don't have Open-MX or Knem, but if we had, I would like them to be
discovered by OMPI and used.
As Bert Lance would say:
"If it ain't broke, don't fix it."
But not only it is not broken, it works like a charm.

Why switch to a different tool chain?
Is it wise, safe, widely available on the OMPI installed base, 
convenient to the final user?
Quite frankly this is the first time I see so much fuss about OMPI's 
build system.


Gus Correa

On 5/16/2014 3:00 PM, Martin Siegert wrote:

+1 even if cmake would make life easier for the developpers, you may
want to consider those sysadmins/users who actually need to compile
and install the software. And for those cmake is a nightmare.
Everytime
I run into a software package that uses cmake it makes me cringe.
gromacs is the perfect example - it has become orders of magnitudes
more complicated to compile just because it now uses cmake. I still
have not succeeded cross compiling (compiling on a machine with a
different processor than the code will later run on) gromacs. This
was
trivial before they switched to cmake.
Another example: want to add RPATH to the executables/libraries?
Just set LDFLAGS='-Wl,-rpath,/usr/local/xyz/lib64' with autotools.
With cmake? Really complicated.

Please, just say no.

Cheers,
Martin

On Fri, May 16, 2014 at 08:33:15PM +, Hjelm, Nathan T wrote:

+1 the bootstrapping issue is 50% of the reason I will never use
CMake for any production code.

vygr:~ hjelmn$ type -p cmake
vygr:~ hjelmn$

Nada, zilch, nothing on standard OS X install. I do not want to put
an extra requirement on my users. Nor do I want something as
simple-minded as CMake. autotools works great for me.

-Nathan


From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain
[r...@open-mpi.org]
Sent: Friday, May 16, 2014 2:07 PM
To: Open MPI Users
Subject: Re: [OMPI users] Question about scheduler support

On May 16, 2014, at 1:03 PM, Fabricio Cannini
<fcann...@gmail.com<mailto:fcann...@gmail.com>> wrote:

Em 16-05-2014 10:06, Jeff Squyres (jsquyres) escreveu:
On May 15, 2014, at 8:00 PM, Fabricio Cannini
<fcann...@gmail.com<mailto:fcann...@gmail.com>>
wrote:

Nobody is disagreeing that one could find a way to make CMake
work - all we are saying is that (a) CMake has issues too, just
like autotools, and (b) we have yet to see a compelling reason to
undertake the transition...which would have to be a *very*
compelling one.

I was simply agreeing with Maxime about why it could work. ;)

But if you and the other devels are fine with it, i'm fine too.

FWIW, simply for my own curiosity's sake, if someone could confirm
deny whether cmake:

1. Supports the following compiler suites: GNU (that's a given, I
assume), Clang, OS X native (which is variants of GNU and Clang),
Absoft, PGI, Intel, Cray, HP-UX, Oracle Solaris (Linux and Solaris),
Tru64, Microsoft Visual, IBM BlueGene (I think that's gcc, but am
not entirely sure).  (some of these matter mainly to hwloc, not
necessarily OMPI)

Not 100% confirmed, but this is good evidence that cmake does indeed
supports all these suites. See the file list:

http://fr2.rpmfind.net//linux/RPM/centos/6.5/x86_64/Packages/cmake-2.6.4-5.el6.x86_64.html


http://fr2.rpmfind.net//linux/RPM/dag/redhat/el6/x86_64/extras/cmake-2.8.8-1.el6.rfx.x86_64.html


http://fr2.rpmfind.net//linux/RPM/opensuse/factory/aarch64/aarch64/cmake-3.0.0~rc4-2.1.aarch64.html


2. Bootstrap a tarball such that an end user does not need to have
cmake installed.

What do you mean by 'bootstrapping a tarball' ?

If someone doesn't have cmake installed and downloads a tarball that
was built from a CMake-based project, can they configure/build that
tarball? Or do they have to install cmake first?


Re: [OMPI users] Question about scheduler support

2014-05-15 Thread Gus Correa

Hi List

Sorry, but I confess I am having a hard time to understand
all the fuss about this.

At least in OMPI 1.6.5 there are already
two configure options that just knock out support for slurm and
loadleveler if they are set to "no", hopefully for the joy of everybody
that want lean and mean OMPI installations.
(Maybe those options are available in OMPI 1.8.1 also?
I haven't tried it.)

Just configure:

--with-slurm=no --with-loadleveler=no

One could go on and on and make a comprehensive list of all options
that one wants in/out, and configure with/without all of them.

**

Isn't this level of simplicity, effectiveness, and of
standard use of configure options, good enough for us?

Works for me.

**

IMHO, what would help is to very clearly document
on the "configure --help" output what is the default value of
*all* options.

This would allow system administrators and other users to safely
make informed choices (or just let the defaults go, if they prefer).
Although I must say right now "configure --help" is not that bad at all. 
I am not frustrated with it. It may need only a few tweaks.


Currently the --with-slurm and --with-loadleveler options
are clearly documented as having "yes" as the default.
However, other options don't have so clearly documented defaults.
In particular -with-tm Torque (which seems to depend on its libraries 
and headers being found or not by configure).

By contrast --with-sge clearly says "no" is the default.

This covers pretty much all free/open source schedulers,
correct me if I am wrong, please.

LSF seems not to have a clearly documented default also.
But LSF is for the rich. I am out.

My 2 cents, 2nd edition, out of print.
Bye, thanks, regards.
Gus Correa


On 05/15/2014 09:35 AM, Jeff Squyres (jsquyres) wrote:

These are all good points -- thanks for the feedback.

Just to be clear: my point about the menu system was to generate file that 
could be used for subsequent installs, very specifically targeted at those who 
want/need scriptable installations.

One possible scenario could be: you download OMPI, run the menu installer so 
that you can see the options that are available, pick the ones you want, 
generate the file, and then use it in automated installs (e.g., ./configure 
--only-build-this-stuff=file_I_created_from_menu_installer).

I am presuming that the generated file will be text, so it could be built/edited by hand, 
too.  This is heavily inspired by the Linux kernel's "make menuconfig": it 
generates a .config file that you can use for subsequent builds.

We can also look at the possibility of providing a template file that lists all options 
that are available in that particular tarball (similar to the Linux kernel "make 
config").

This, BTW, is one part where we need to build some new infrastructure: to 
register and discover all available options (history has shown that just 
maintaining a text file for a project the size of Open MPI is not feasible -- 
it'll get stale/out of date).

Other large projects do this kind of thing; we need to see if there's any 
ideas/code we can borrow rather than completely re-inventing the wheel.

Again, these are just my thoughts after having thought about this for only 
about 30 minutes -- so this is likely quite rough and may not even resemble 
what we finally end up with.


On May 15, 2014, at 8:51 AM, Noam Bernstein <noam.bernst...@nrl.navy.mil> wrote:


I’m not sure how this would apply to other options, but for the scheduler, what 
about no scheduler-related options defaults to everything enabled (like 
before), but having any explicit scheduler enable option disables by default 
all the other schedulers? Multiple explicit enable options would enable all the 
requested schedulers, and disable everything else.


Noam
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users







Re: [OMPI users] Question about scheduler support

2014-05-14 Thread Gus Correa

On 05/14/2014 04:25 PM, Maxime Boissonneault wrote:

Hi,
I was compiling OpenMPI 1.8.1 today and I noticed that pretty much every
single scheduler has its support enabled by default at configure (except
the one I need, which is Torque). Is there a reason for that ? Why not
have a single scheduler enabled and require to specify it at configure
time ?

Is there any reason for me to build with loadlever or slurm if we're
using torque ?

Thanks,

Maxime Boisssonneault


Hi Maxime

I haven't tried 1.8.1 yet.
However, for all previous versions of OMPI I tried, up to 1.6.5,
all it took to configure it with Torque support was to point configure
to the Torque installation directory (which is non-standard in my case):

--with-tm=/opt/torque/bla/bla

My two cents,
Gus Correa



Re: [OMPI users] users Digest, Vol 2881, Issue 4

2014-05-07 Thread Gus Correa

On 05/06/2014 09:49 PM, Ralph Castain wrote:


On May 6, 2014, at 6:24 PM, Clay Kirkland > wrote:


 Got it to work finally.  The longer line doesn't work.
192.168.0.0/1
But if I take off the -mca oob_tcp_if_include 192.168.0.0/16
 part then everything works from
every combination of machines I have.


Interesting - I'm surprised, but glad it worked



Could it be perhaps 192.168.0.0/24 (instead of /16)?
The ifconfig output says the netmask is 255.255.255.0.



And as to any MPI having trouble, in my original posting I stated that
I installed lam mpi
on the same hardware and it worked just fine.   Maybe you guys should
look at what they
do and copy it.   Virtually every machine I have used in the last 5
years has multiple nic
interfaces and almost all of them are set up to use only 1
interface.   It seems odd to have
a product that is designed to lash together multiple machines and have
it fail with default
install on generic machines.


Actually, we are the "lam mpi" guys :-)

There clearly is a bug in the connection logic, but a little hint will
work it thru until we can resolve it.



  But software is like that some time and I want to thank you  much
for all the help.   Please
take my criticism with a grain of salt.   I love MPI, I just want to
see it work.   I have been
using it for 20 some years to synchronize multiple machines for I/O
testing and it is one
slick product for that.   It has helped us find many bugs in shared
files systems.  Thanks
again,


No problem!






On Tue, May 6, 2014 at 7:45 PM, > wrote:

Send users mailing list submissions to
us...@open-mpi.org 

To subscribe or unsubscribe via the World Wide Web, visit
http://www.open-mpi.org/mailman/listinfo.cgi/users
or, via email, send a message with subject or body 'help' to
users-requ...@open-mpi.org 

You can reach the person managing the list at
users-ow...@open-mpi.org 

When replying, please edit your Subject line so it is more specific
than "Re: Contents of users digest..."


Today's Topics:

   1. Re: users Digest, Vol 2881, Issue 2 (Ralph Castain)


--

Message: 1
Date: Tue, 6 May 2014 17:45:09 -0700
From: Ralph Castain >
To: Open MPI Users >
Subject: Re: [OMPI users] users Digest, Vol 2881, Issue 2
Message-ID: <4b207e61-952a-4744-9a7b-0704c4b0d...@open-mpi.org
>
Content-Type: text/plain; charset="us-ascii"

-mca btl_tcp_if_include 192.168.0.0/16 
-mca oob_tcp_if_include 192.168.0.0/16 

should do the trick. Any MPI is going to have trouble with your
arrangement - just need a little hint to help figure it out.


On May 6, 2014, at 5:14 PM, Clay Kirkland
> wrote:

>  Someone suggested using some network address if all machines
are on same subnet.
> They are all on the same subnet, I think.   I have no idea what
to put for a param there.
> I tried the ethernet address but of course it couldn't be that
simple.  Here are my ifconfig
> outputs from a couple of machines:
>
> [root@RAID MPI]# ifconfig -a
> eth0  Link encap:Ethernet  HWaddr 00:25:90:73:2A:36
>   inet addr:192.168.0.59  Bcast:192.168.0.255
 Mask:255.255.255.0
>   inet6 addr: fe80::225:90ff:fe73:2a36/64 Scope:Link
>   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>   RX packets:17983 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:9952 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:1000
>   RX bytes:26309771 (25.0 MiB)  TX bytes:758940 (741.1 KiB)
>   Interrupt:16 Memory:fbde-fbe0
>
> eth1  Link encap:Ethernet  HWaddr 00:25:90:73:2A:37
>   inet6 addr: fe80::225:90ff:fe73:2a37/64 Scope:Link
>   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>   RX packets:56 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:1000
>   RX bytes:3924 (3.8 KiB)  TX bytes:468 (468.0 b)
>   Interrupt:17 Memory:fbee-fbf0
>
>  And from one that I can't get to work:
>
> [root@centos ~]# ifconfig -a
> eth0  Link encap:Ethernet  HWaddr 00:1E:4F:FB:30:34
>   inet6 addr: fe80::21e:4fff:fefb:3034/64 Scope:Link
>   UP 

Re: [OMPI users] Where is the error? (MPI program in fortran)

2014-04-17 Thread Gus Correa

Hi Oscar

As Ralph suggested, the problem is indeed a memory access violation,
a typical violation of array bounds.
Not really an MPI or OpenMPI problem to be addressed
by this mailing list.

Your ran2 function has a memory violation bug.
It declares dimension ir(1000),
but, the algorithm generates indexes j for that array above 1000.
Here is a sample:

[1,1]: j =   72
[1,1]: j =  686
[1,1]: j = 1353

By the way, although a comment in the program says so,
other than the name, ran2 certainly isn't the ran2
algorithm in Numerical Recipes.
If you want to use a random number generator from Num. Rec.,
the books and the algorithms are available online
(ran2 is on p. 272, ch. 7, Numerical Recipes in Fortran 77 or 90):

http://www.nr.com/oldverswitcher.html

As a general suggestion, you may get fewer bugs in Fortran if you drop
all implicit variable declarations in the program code
and replace them by explicit declarations
(and add "implicit none" to all program units, to play safe).
Implicit variable declarations are a big source of bugs.

I hope this helps,
Gus Correa

PS - If you are at UFBA, send my hello to Milton Porsani, please.

On 04/17/2014 02:01 PM, Oscar Mojica wrote:

Hello guys

I used the command

ulimit -s unlimited

and got

stack size  (kbytes, -s) unlimited

but when I ran the program got the same error. So I used the gdb
debugger, I compiled using

mpif90 -g -o mpivfsa_versao2.f  exe

I ran the program and then I ran gdb with both the executable and the
core file name as arguments and got the following

Program received signal SIGSEGV, Segmentation fault.
0x2b59b54c in free () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) backtrace
#0  0x2b59b54c in free () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00406801 in inv_grav3d_vfsa () at mpivfsa_versao2.f:131
#2  0x00406b88 in main (argc=1, argv=0x7fffe387) at
mpivfsa_versao2.f:9
#3  0x2b53976d in __libc_start_main () from
/lib/x86_64-linux-gnu/libc.so.6
#4  0x00401399 in _start ()

These are the lines

9use mpi
131   deallocate(zv,xrec,yrec,xprm,yprm)

I think the problem is not memory, the problem is related to MPI

Which could be the error?

_Oscar Fabian Mojica Ladino_
Geologist M.S. in  Geophysics


 > From: o_moji...@hotmail.com
 > Date: Wed, 16 Apr 2014 15:17:51 -0300
 > To: us...@open-mpi.org
 > Subject: Re: [OMPI users] Where is the error? (MPI program in fortran)
 >
 > Gus
 > It is a single machine and i have installed Ubuntu 12.04 LTS. I left
my computer in the college but I will try to follow your advice when I
can and tell you about it.
 >
 > Thanks
 >
 > Enviado desde mi iPad
 >
 > > El 16/04/2014, a las 14:17, "Gus Correa" <g...@ldeo.columbia.edu>
escribió:
 > >
 > > Hi Oscar
 > >
 > > This is a long shot, but maybe worth trying.
 > > I am assuming you're using Linux, or some form or Unix, right?
 > >
 > > You may try to increase the stack size.
 > > The default in Linux is often too small for large programs.
 > > Sometimes this may cause a segmentation fault, even if the
 > > program is correct.
 > >
 > > You can check what you have with:
 > >
 > > ulimit -a (bash)
 > >
 > > or
 > >
 > > limit (csh or tcsh)
 > >
 > > Then set it to a larger number or perhaps to unlimited,
 > > e.g.:
 > >
 > > ulimit -s unlimited
 > >
 > > or
 > >
 > > limit stacksize unlimited
 > >
 > > You didn't say anything about the computer(s) you are using.
 > > Is this a single machine, a cluster, something else?
 > >
 > > Anyway, resetting the statck size may depend a bit on what you
 > > have in /etc/security/limits.conf,
 > > and whether it allows you to increase the stack size.
 > > If it is a single computer that you have root access, you may
 > > do it yourself.
 > > There are other limits worth increasing (number of open files,
 > > max locked memory).
 > > For instance, this could go in limits.conf:
 > >
 > > * - memlock -1
 > > * - stack -1
 > > * - nofile 4096
 > >
 > > See 'man limits.conf' for details.
 > >
 > > If it is a cluster, and this should be set on all nodes,
 > > and you may need to ask your system administrator to do it.
 > >
 > > I hope this helps,
 > > Gus Correa
 > >
 > >> On 04/16/2014 11:24 AM, Gus Correa wrote:
 > >>> On 04/16/2014 08:30 AM, Oscar Mojica wrote:
 > >>> How would be the command line to compile with the option -g ? What
 > >>> debugger can I use?
 > >>> Thanks
 > >>>
 > >>
 > >> Replace any optimization flags (-O2, or similar) by -g.
 > >> Check if 

Re: [OMPI users] Where is the error? (MPI program in fortran)

2014-04-16 Thread Gus Correa

Hi Oscar

This is a long shot, but maybe worth trying.
I am assuming you're using Linux, or some form or Unix, right?

You may try to increase the stack size.
The default in Linux is often too small for large programs.
Sometimes this may cause a segmentation fault, even if the
program is correct.

You can check what you have with:

ulimit -a(bash)

or

limit (csh or tcsh)

Then set it to a larger number or perhaps to unlimited,
e.g.:

ulimit -s unlimited

or

limit stacksize unlimited

You didn't say anything about the computer(s) you are using.
Is this a single machine, a cluster, something else?

Anyway, resetting the statck size may depend a bit on what you
have in /etc/security/limits.conf,
and whether it allows you to increase the stack size.
If it is a single computer that you have root access, you may
do it yourself.
There are other limits worth increasing (number of open files,
max locked memory).
For instance, this could go in limits.conf:

*   -   memlock -1
*   -   stack   -1
*   -   nofile  4096

See 'man limits.conf' for details.

If it is a cluster, and this should be set on all nodes,
and you may need to ask your system administrator to do it.

I hope this helps,
Gus Correa

On 04/16/2014 11:24 AM, Gus Correa wrote:

On 04/16/2014 08:30 AM, Oscar Mojica wrote:

How would be the command line to compile with the option -g ? What
debugger can I use?
Thanks



Replace any optimization flags (-O2, or similar) by -g.
Check if your compiler has the -traceback flag or similar
(man compiler-name).

The gdb debugger is normally available on Linux (or you can install it
with yum, apt-get, etc).  An alternative is ddd, with a GUI (can also be
installed from yum, etc).
If you use a commercial compiler you may have a debugger with a GUI.


Enviado desde mi iPad


El 15/04/2014, a las 18:20, "Gus Correa" <g...@ldeo.columbia.edu>
escribió:

Or just compiling with -g or -traceback (depending on the compiler) will
give you more information about the point of failure
in the error message.


On 04/15/2014 04:25 PM, Ralph Castain wrote:
Have you tried using a debugger to look at the resulting core file? It
will probably point you right at the problem. Most likely a case of
overrunning some array when #temps > 5




On Tue, Apr 15, 2014 at 10:46 AM, Oscar Mojica <o_moji...@hotmail.com
<mailto:o_moji...@hotmail.com>> wrote:

Hello everybody

I implemented a parallel simulated annealing algorithm in fortran.
  The algorithm is describes as follows

1. The MPI program initially generates P processes that have rank
0,1,...,P-1.
2. The MPI program generates a starting point and sends it  for all
processes set T=T0
3. At the current temperature T, each process begins to execute
iterative operations
4. At end of iterations, process with rank 0 is responsible for
collecting the solution obatined by
5. Each process at current temperature and broadcast the best
solution of them among all participating
process
6. Each process cools the temperatue and goes back to step 3, until
the maximum number of temperatures
is reach

I compiled with: mpif90 -o exe mpivfsa_version2.f
and run with: mpirun -np 4 ./exe in a single machine

So I have 4 processes, 1 iteration per temperature and for example
15 temperatures. When I run the program
with just 5 temperatures it works well, but when the number of
temperatures is higher than 5 it doesn't write the
ouput files and I get the following error message:


[oscar-Vostro-3550:06740] *** Process received signal ***
[oscar-Vostro-3550:06741] *** Process received signal ***
[oscar-Vostro-3550:06741] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06741] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06741] Failing at address: 0xad6af
[oscar-Vostro-3550:06742] *** Process received signal ***
[oscar-Vostro-3550:06740] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06740] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06740] Failing at address: 0xad6af
[oscar-Vostro-3550:06742] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06742] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06742] Failing at address: 0xad6af
[oscar-Vostro-3550:06740] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f49ee2224a0]
[oscar-Vostro-3550:06740] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f49ee26f54c]
[oscar-Vostro-3550:06740] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06740] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06740] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)
[0x7f49ee20d76d]
[oscar-Vostro-3550:06742] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f6877fdc4a0]
[oscar-Vostro-3550:06742] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f687802954c]
[oscar-Vostro-355

Re: [OMPI users] Where is the error? (MPI program in fortran)

2014-04-16 Thread Gus Correa

On 04/16/2014 08:30 AM, Oscar Mojica wrote:

How would be the command line to compile with the option -g ? What debugger can 
I use?
Thanks



Replace any optimization flags (-O2, or similar) by -g.
Check if your compiler has the -traceback flag or similar
(man compiler-name).

The gdb debugger is normally available on Linux (or you can install it 
with yum, apt-get, etc).  An alternative is ddd, with a GUI (can also be 
installed from yum, etc).

If you use a commercial compiler you may have a debugger with a GUI.


Enviado desde mi iPad


El 15/04/2014, a las 18:20, "Gus Correa" <g...@ldeo.columbia.edu> escribió:

Or just compiling with -g or -traceback (depending on the compiler) will
give you more information about the point of failure
in the error message.


On 04/15/2014 04:25 PM, Ralph Castain wrote:
Have you tried using a debugger to look at the resulting core file? It
will probably point you right at the problem. Most likely a case of
overrunning some array when #temps > 5




On Tue, Apr 15, 2014 at 10:46 AM, Oscar Mojica <o_moji...@hotmail.com
<mailto:o_moji...@hotmail.com>> wrote:

Hello everybody

I implemented a parallel simulated annealing algorithm in fortran.
  The algorithm is describes as follows

1. The MPI program initially generates P processes that have rank
0,1,...,P-1.
2. The MPI program generates a starting point and sends it  for all
processes set T=T0
3. At the current temperature T, each process begins to execute
iterative operations
4. At end of iterations, process with rank 0 is responsible for
collecting the solution obatined by
5. Each process at current temperature and broadcast the best
solution of them among all participating
process
6. Each process cools the temperatue and goes back to step 3, until
the maximum number of temperatures
is reach

I compiled with: mpif90 -o exe mpivfsa_version2.f
and run with: mpirun -np 4 ./exe in a single machine

So I have 4 processes, 1 iteration per temperature and for example
15 temperatures. When I run the program
with just 5 temperatures it works well, but when the number of
temperatures is higher than 5 it doesn't write the
ouput files and I get the following error message:


[oscar-Vostro-3550:06740] *** Process received signal ***
[oscar-Vostro-3550:06741] *** Process received signal ***
[oscar-Vostro-3550:06741] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06741] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06741] Failing at address: 0xad6af
[oscar-Vostro-3550:06742] *** Process received signal ***
[oscar-Vostro-3550:06740] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06740] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06740] Failing at address: 0xad6af
[oscar-Vostro-3550:06742] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06742] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06742] Failing at address: 0xad6af
[oscar-Vostro-3550:06740] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f49ee2224a0]
[oscar-Vostro-3550:06740] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f49ee26f54c]
[oscar-Vostro-3550:06740] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06740] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06740] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f49ee20d76d]
[oscar-Vostro-3550:06742] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f6877fdc4a0]
[oscar-Vostro-3550:06742] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f687802954c]
[oscar-Vostro-3550:06742] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06742] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06742] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f6877fc776d]
[oscar-Vostro-3550:06742] [ 5] ./exe() [0x401399]
[oscar-Vostro-3550:06742] *** End of error message ***
[oscar-Vostro-3550:06740] [ 5] ./exe() [0x401399]
[oscar-Vostro-3550:06740] *** End of error message ***
[oscar-Vostro-3550:06741] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fa6c4c6e4a0]
[oscar-Vostro-3550:06741] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7fa6c4cbb54c]
[oscar-Vostro-3550:06741] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06741] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06741] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fa6c4c5976d]
[oscar-Vostro-3550:06741] [ 5] ./exe() [0x401399]
[oscar-Vostro-3550:06741] *** End of error message ***
--
mpirun noticed that process rank 0 with PID 6917 on node
oscar-Vostro-3550 exited on signal 11 (Segmentation fault).
--
2 total processes killed 

Re: [OMPI users] Where is the error? (MPI program in fortran)

2014-04-15 Thread Gus Correa

Or just compiling with -g or -traceback (depending on the compiler) will
give you more information about the point of failure
in the error message.

On 04/15/2014 04:25 PM, Ralph Castain wrote:

Have you tried using a debugger to look at the resulting core file? It
will probably point you right at the problem. Most likely a case of
overrunning some array when #temps > 5




On Tue, Apr 15, 2014 at 10:46 AM, Oscar Mojica > wrote:

Hello everybody

I implemented a parallel simulated annealing algorithm in fortran.
  The algorithm is describes as follows

1. The MPI program initially generates P processes that have rank
0,1,...,P-1.
2. The MPI program generates a starting point and sends it  for all
processes set T=T0
3. At the current temperature T, each process begins to execute
iterative operations
4. At end of iterations, process with rank 0 is responsible for
collecting the solution obatined by
5. Each process at current temperature and broadcast the best
solution of them among all participating
process
6. Each process cools the temperatue and goes back to step 3, until
the maximum number of temperatures
is reach

I compiled with: mpif90 -o exe mpivfsa_version2.f
and run with: mpirun -np 4 ./exe in a single machine

So I have 4 processes, 1 iteration per temperature and for example
15 temperatures. When I run the program
with just 5 temperatures it works well, but when the number of
temperatures is higher than 5 it doesn't write the
ouput files and I get the following error message:


[oscar-Vostro-3550:06740] *** Process received signal ***
[oscar-Vostro-3550:06741] *** Process received signal ***
[oscar-Vostro-3550:06741] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06741] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06741] Failing at address: 0xad6af
[oscar-Vostro-3550:06742] *** Process received signal ***
[oscar-Vostro-3550:06740] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06740] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06740] Failing at address: 0xad6af
[oscar-Vostro-3550:06742] Signal: Segmentation fault (11)
[oscar-Vostro-3550:06742] Signal code: Address not mapped (1)
[oscar-Vostro-3550:06742] Failing at address: 0xad6af
[oscar-Vostro-3550:06740] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f49ee2224a0]
[oscar-Vostro-3550:06740] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f49ee26f54c]
[oscar-Vostro-3550:06740] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06740] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06740] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f49ee20d76d]
[oscar-Vostro-3550:06742] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f6877fdc4a0]
[oscar-Vostro-3550:06742] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f687802954c]
[oscar-Vostro-3550:06742] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06742] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06742] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f6877fc776d]
[oscar-Vostro-3550:06742] [ 5] ./exe() [0x401399]
[oscar-Vostro-3550:06742] *** End of error message ***
[oscar-Vostro-3550:06740] [ 5] ./exe() [0x401399]
[oscar-Vostro-3550:06740] *** End of error message ***
[oscar-Vostro-3550:06741] [ 0]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fa6c4c6e4a0]
[oscar-Vostro-3550:06741] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7fa6c4cbb54c]
[oscar-Vostro-3550:06741] [ 2] ./exe() [0x406742]
[oscar-Vostro-3550:06741] [ 3] ./exe(main+0x34) [0x406ac9]
[oscar-Vostro-3550:06741] [ 4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fa6c4c5976d]
[oscar-Vostro-3550:06741] [ 5] ./exe() [0x401399]
[oscar-Vostro-3550:06741] *** End of error message ***
--
mpirun noticed that process rank 0 with PID 6917 on node
oscar-Vostro-3550 exited on signal 11 (Segmentation fault).
--
2 total processes killed (some possibly by mpirun during cleanup)

If there is a segmentation fault in no case it must work .
I checked the program and didn't find the error. Why does the
program work with five temperatures?
Could someone help me to find the error and answer my question please.

The program and the necessary files to run it  are attached

Thanks


_Oscar Fabian Mojica Ladino_
Geologist M.S. in  Geophysics

___
users mailing list
us...@open-mpi.org 
http://www.open-mpi.org/mailman/listinfo.cgi/users





Re: [OMPI users] mpirun runs in serial even I set np to several processors

2014-04-15 Thread Gus Correa

Hi Djordje

That is great news.
Congrats for making it work!

Just out of curiosity: What did the trick?
Did you install Open MPI from source, or did you sort out
the various MPI flavors which were previously installed on your system?

Now the challenge is to add OpenMP and run WRF
in hybrid mode just for fun!  :)

Best,
Gus Correa

***

PS: Parallel computing, MPI, and OpenMP, tutorials at LLNL:

https://computing.llnl.gov/tutorials/parallel_comp/
https://computing.llnl.gov/tutorials/mpi/
https://computing.llnl.gov/tutorials/openMP/

Ch. 5 in the first tutorial gives an outline of the various
parallel programming models, and the basic ideas behind MPI and OpenMP.

**

Wild guesses based on other models (climate, not weather):

Most likely WRF uses the domain decomposition technique to solve
the dynamics' PDEs, exchanging sub-domain boundary data via MPI.
[Besides the dynamics, each process will also
compute thermodynamics, radiation effects, etc,
which may also require data exchange with neighbors.]
Each MPI process takes care of/computes on a subdomain,
and exchanges boundary data with those processes assigned
to neighbor subdomains, with the whole group contributing to
solve the PDEs in the global domain.
[This uses MPI point-to-point functions like MPI_Send/MPI_Recv.]
There may be some additional global calculations also, say,
to ensure conservation of mass, energy, momentum, etc,
which may involve all MPI processes.
[This may use MPI collective functions like MPI_Reduce.]

http://en.wikipedia.org/wiki/Domain_decomposition_methods

Besides, WRF  probably can split computation on
loops across different threads via OpenMP.
[Well, there is more to OpenMP than just loop splitting,
but loop splitting is the most common.]
You need to provide physical processors for those threads,
which is typically done by setting the environment variable 
OMP_NUM_THREADS (e.g. in bash: 'export OMP_NUM_THREADS=4').


In hybrid (MPI + OpenMP mode) you use both, but must be careful
to provide enough processors for all MPI processes and OpenMP threads.
Say, for 3 MPI processes, each one launching two OpenMP threads,
you could do (if you turned both on when you configured WRF):

export OMP_NUM_THREADS=2
mpirun -np 3 ./wrf.exe

for a total of 6 processors.

Better not oversubscribe processors.
If your computer has 4 cores, use -np 2 instead of 3 in the lines above.

For a small number of processors (and/or a small global domain), you 
will probably get better performance if you assign

all processors to MPI processes, and simply do not use OpenMP.

Finally, if you do:
export OMP_NUM_THREADS=1
mpiexec -np 4 ./wrf.exe
WRF will run in MPI mode, even if you configured it hybrid.
[At least this is what it is supposed to do.]

I hope this helps,
Gus Correa

On 04/15/2014 01:59 PM, Djordje Romanic wrote:

Hi,

It is working now. It shows:

starting wrf task0  of4
  starting wrf task1  of4
  starting wrf task2  of4
  starting wrf task3  of4
-
Thank you so much!!! You helped me a lot! Finally :) And plus I know the
difference between OpenMP and Open MPI (well, to be honest not
completely, but more than i knew before). :D

Thanks,

Djordje




On Tue, Apr 15, 2014 at 11:57 AM, Gus Correa <g...@ldeo.columbia.edu
<mailto:g...@ldeo.columbia.edu>> wrote:

Hi Djordje

"locate mpirun" shows items labled "intel", "mpich", and "openmpi",
maybe more.
Is it Ubuntu or Debian?

Anyway, if you got this mess from somebody else,
instead of sorting it out,
it may save you time and headaches installing Open MPI from
source.
Since it is a single machine, there are no worries about
having an homogeneous installation for several computers (which
could be done if needed, though).

0. Make sure you have gcc, g++, and gfortran installed,
including any "devel" packages that may exist.
[apt-get or yum should tell you]
If something is missing, install it.

1. Download the Open MPI (a.k.a OMPI) tarball to a work directory
of your choice,
say /home/djordje/inst/openmpi/1.8 (create the directory if needed),
and untar the tarball (tar -jxvf ...)

http://www.open-mpi.org/__software/ompi/v1.8/
<http://www.open-mpi.org/software/ompi/v1.8/>

2. Configure it to be installed in yet another directory under
your home, say /home/djordje/sw/openmpi/1.8 (with --prefix).

cd /home/djordje/inst/openmpi/1.8

./configure --prefix=/home/djordje/sw/__openmpi/1.8 CC=gcc, CXX=g++,
FC=gfortran

[Not sure if with 1.8 there is a separate F77 interface, if there is
add F77=gfortran to the configure command line above.
Also, I am using OMPI 1.6.5,
but my recollection is that Jeff would phase off mpif90 and mpif77 in

Re: [OMPI users] mpirun runs in serial even I set np to several processors

2014-04-15 Thread Gus Correa

Hi Djordje

"locate mpirun" shows items labled "intel", "mpich", and "openmpi", 
maybe more.

Is it Ubuntu or Debian?

Anyway, if you got this mess from somebody else,
instead of sorting it out,
it may save you time and headaches installing Open MPI from
source.
Since it is a single machine, there are no worries about
having an homogeneous installation for several computers (which
could be done if needed, though).

0. Make sure you have gcc, g++, and gfortran installed,
including any "devel" packages that may exist.
[apt-get or yum should tell you]
If something is missing, install it.

1. Download the Open MPI (a.k.a OMPI) tarball to a work directory
of your choice,
say /home/djordje/inst/openmpi/1.8 (create the directory if needed),
and untar the tarball (tar -jxvf ...)

http://www.open-mpi.org/software/ompi/v1.8/

2. Configure it to be installed in yet another directory under
your home, say /home/djordje/sw/openmpi/1.8 (with --prefix).

cd /home/djordje/inst/openmpi/1.8

./configure --prefix=/home/djordje/sw/openmpi/1.8 CC=gcc, CXX=g++,
FC=gfortran

[Not sure if with 1.8 there is a separate F77 interface, if there is
add F77=gfortran to the configure command line above.
Also, I am using OMPI 1.6.5,
but my recollection is that Jeff would phase off mpif90 and mpif77 in
favor of a single mpifortran of sorts.  Please check the OMPI README file.]

Then do

make
make install

3. Setup your environment variables PATH and LD_LIBRARY_PATH
to point to *this* Open MPI installation ahead of anything else.
This is easily done in your .bashrc or .tcshrc/.cshrc file,
depending on which shell you use

.bashrc :
export PATH=/home/djordje/sw/openmpi/1.8/bin:$PATH
export LD_LIBRARY_PATH=/home/djordje/sw/openmpi/1.8/lib:$LD_LIBRARY_PATH

.tcshrc/.cshrc:

setenv PATH /home/djordje/sw/openmpi/1.8/bin:$PATH
setenv LD_LIBRARY_PATH /home/djordje/sw/openmpi/1.8/lib:$LD_LIBRARY_PATH

4. Logout, login again (or open a new terminal), and check if you
get the right mpirun, etc:

which mpicc
which mpif90
which mpirun

They should point to items in /home/djordje/sw/openmpi/1.8/bin

5. Rebuild WRF from scratch.

6. Check if WRF got the libraries right:

ldd wrf.exe

This should show mpi libraries in /home/djordje/sw/openmpi/1.8/lib

7. Run WRF
mpirun -np 4 wrf.exe

I hope this helps,
Gus Correa




On 04/14/2014 08:21 PM, Djordje Romanic wrote:

Hi,

Thanks for this guys. I think I might have two MPI implementations
installed because 'locate mpirun' gives (see bold lines) :
-
/etc/alternatives/mpirun
/etc/alternatives/mpirun.1.gz
*/home/djordje/Build_WRF/LIBRARIES/mpich/bin/mpirun*
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/intel/4.1.1.036/linux-x86_64/bin/mpirun
<http://4.1.1.036/linux-x86_64/bin/mpirun>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/intel/4.1.1.036/linux-x86_64/bin64/mpirun
<http://4.1.1.036/linux-x86_64/bin64/mpirun>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/intel/4.1.1.036/linux-x86_64/ia32/bin/mpirun
<http://4.1.1.036/linux-x86_64/ia32/bin/mpirun>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/intel/4.1.1.036/linux-x86_64/intel64/bin/mpirun
<http://4.1.1.036/linux-x86_64/intel64/bin/mpirun>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/openmpi/1.4.3/linux-x86_64-2.3.4/gnu4.5/bin/mpirun
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/openmpi/1.4.3/linux-x86_64-2.3.4/gnu4.5/share/man/man1/mpirun.1
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/openmpi/1.6.4/linux-x86_64-2.3.4/gnu4.6/bin/mpirun
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/openmpi/1.6.4/linux-x86_64-2.3.4/gnu4.6/share/man/man1/mpirun.1
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/bin/mpirun
<http://8.2.0.0/linux64_2.6-x86-glibc_2.3.4/bin/mpirun>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/bin/mpirun.mpich
<http://8.2.0.0/linux64_2.6-x86-glibc_2.3.4/bin/mpirun.mpich>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/bin/mpirun.mpich2
<http://8.2.0.0/linux64_2.6-x86-glibc_2.3.4/bin/mpirun.mpich2>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/bin/mpirun
<http://8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/bin/mpirun>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/bin/mpirun.mpich
<http://8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/bin/mpirun.mpich>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/bin/mpirun.mpich2
<http://8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/bin/mpirun.mpich2>
/home/djordje/StarCCM/Install/STAR-CCM+8.06.007/mpi/platform/8.2.0.0/linux64_2.6-x86-glibc_2.3.4/ia32/lib/linux_amd64/libmpirun.so
<http://8.2.0.0/linux64_2.6-x86-gl

Re: [OMPI users] mpirun runs in serial even I set np to several processors

2014-04-14 Thread Gus Correa

Apologies for stirring even more the confusion by mispelling
"Open MPI" as "OpenMPI".
"OMPI" doesn't help either, because all OpenMP environment
variables and directives start with "OMP".
Maybe associating the names to
"message passing" vs. "threads" would help?

Djordje:

'which mpif90' etc show everything in /usr/bin.
So, very likely they were installed from packages
(yum, apt-get, rpm ...),right?
Have you tried something like
"yum list |grep mpi"
to see what you have?

As Dave, Jeff and Tom said, this may be a mixup of different
MPI implementations at compilation (mpicc mpif90) and runtime (mpirun).
That is common, you may have different MPI implementations installed.

Other possibilities that may tell what MPI you have:

mpirun --version
mpif90 --show
mpicc --show

Yet another:

locate mpirun
locate mpif90
locate mpicc

The ldd didn't show any MPI libraries, maybe they are static libraries.

An alternative is to install Open MPI from source,
and put it in a non-system directory
(not /usr/bin, not /usr/local/bin, etc).

Is this a single machine or a cluster?
Or perhaps a set of PCs that you have access to?
If it is a cluster, do you have access to a filesystem that is
shared across the cluster?
On clusters typically /home is shared, often via NFS.

Gus Correa

On 04/14/2014 05:15 PM, Jeff Squyres (jsquyres) wrote:

Maybe we should rename OpenMP to be something less confusing --
perhaps something totally unrelated, perhaps even non-sensical.
That'll end lots of confusion!

My vote: OpenMP --> SharkBook

It's got a ring to it, doesn't it?  And it sounds fearsome!



On Apr 14, 2014, at 5:04 PM, "Elken, Tom" <tom.el...@intel.com> wrote:


That’s OK.  Many of us make that mistake, though often as a typo.
One thing that helps is that the correct spelling of Open MPI has a space in it,

but OpenMP does not.

If not aware what OpenMP is, here is a link: http://openmp.org/wp/

What makes it more confusing is that more and more apps.

offer the option of running in a hybrid mode, such as WRF,
with OpenMP threads running over MPI ranks with the same executable.
And sometimes that MPI is Open MPI.


Cheers,
-Tom

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Djordje Romanic
Sent: Monday, April 14, 2014 1:28 PM
To: Open MPI Users
Subject: Re: [OMPI users] mpirun runs in serial even I set np to several 
processors

OK guys... Thanks for all this info. Frankly, I didn't know these diferences 
between OpenMP and OpenMPI. The commands:
which mpirun
which mpif90
which mpicc
give,
/usr/bin/mpirun
/usr/bin/mpif90
/usr/bin/mpicc
respectively.

A tutorial on how to compile WRF 
(http://www.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php) provides 
a test program to test MPI. I ran the program and it gave me the output of 
successful run, which is:
-
C function called by Fortran
Values are xx = 2.00 and ii = 1
status = 2
SUCCESS test 2 fortran + c + netcdf + mpi
-
It uses mpif90 and mpicc for compiling. Below is the output of 'ldd ./wrf.exe':


 linux-vdso.so.1 =>  (0x7fff584e7000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x7f4d160ab000)
 libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 
(0x7f4d15d94000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f4d15a97000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f4d15881000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f4d154c1000)
 /lib64/ld-linux-x86-64.so.2 (0x7f4d162e8000)
 libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 
(0x7f4d1528a000)



On Mon, Apr 14, 2014 at 4:09 PM, Gus Correa <g...@ldeo.columbia.edu> wrote:
Djordje

Your WRF configure file seems to use mpif90 and mpicc (line 115 & following).
In addition, it also seems to have DISABLED OpenMP (NO TRAILING "I")
(lines 109-111, where OpenMP stuff is commented out).
So, it looks like to me your intent was to compile with MPI.

Whether it is THIS MPI (OpenMPI) or another MPI (say MPICH, or MVAPICH,
or Intel MPI, or Cray, or ...) only your environment can tell.

What do you get from these commands:

which mpirun
which mpif90
which mpicc

I never built WRF here (but other people here use it).
Which input do you provide to the command that generates the configure
script that you sent before?
Maybe the full command line will shed some light on the problem.


I hope this helps,
Gus Correa


On 04/14/2014 03:11 PM, Djordje Romanic wrote:
to get help :)



On Mon, Apr 14, 2014 at 3:11 PM, Djordje Romanic <djord...@gmail.com
<mailto:djord...@gmail.com>> wrote:

 Yes, but I was hoping to get. :)


 On Mon, Apr 14, 2014 at 3:02 PM, Jeff Squyres (jsquyres)
 <jsquy...@cisco.com <mailto:jsquy...@cisco.com>> wrote:

 

  1   2   3   4   5   >