Re: [OMPI users] btl_openib_receive_queues & mca-btl-openib-device-params.ini

[Sasso, John (GE Power & Water, Non-GE)]

I did some code-digging and I found the answer.

If the MCA parameter btl_openib_receive_queues is not spec'd on the mpirun 
command line and not specified in 
MPI_HOME/share/openmpi/mca-btl-openib-device-params.ini  (via receive_queues 
parameter), then OpenMPI derives the default setting from the code in 
ompi/mca/btl/openib/btl_openib_mca.c.  If you look in that code you will see:

snprintf(default_qps, 100,

"P,128,256,192,128:S,%u,1024,1008,64:S,%u,1024,1008,64:S,%u,1024,1008,64",
mid_qp_size,   // For S,%u,1024,1008,64 where %u == 2048
(uint32_t)mca_btl_openib_module.super.btl_eager_limit,
(uint32_t)mca_btl_openib_module.super.btl_max_send_size);

So for OpenMPI 1.6.5 we have:


P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64

So you can examine the code and see how the %u's are filled in to get the 
resultant default btl_openib_receive_queues.

Nice if this was documented!

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Shamis, Pavel
Sent: Thursday, November 05, 2015 2:41 PM
To: Open MPI Users
Subject: Re: [OMPI users] btl_openib_receive_queues & 
mca-btl-openib-device-params.ini

Below is from ompi_info:

$ ompi_info --all | grep btl_openib_receive
 MCA btl: parameter "btl_openib_receive_queues" (current value: 
,
 data source: default value)

This tunning does make sense for me.



#receive_queues = P,128,256,192,128:S,65536,256,192,128

Most likely it was not updated for a long time.
I'm pretty sure the ompi_info version is the one that actually used.



So comparing the two above, they don't coincide, in particular for the SRQ 
settings.  Thus, I gather the settings reported by ompi_info above are those 
that are actually used, and if so, I am wondering where/how they are derived?

The idea behind this parameters is coming from here:
http://www.open-mpi.org/papers/euro-pvmmpi-2007-ib/euro-pvmmpi-2007-ib.pdf
And later on XRC support was introduced:
http://www.open-mpi.org/papers/euro-pvmmpi-2008-xrc/euro-pvmmpi-2008-xrc.pdf

Best,
Pasha

[OMPI users] btl_openib_receive_queues & mca-btl-openib-device-params.ini

[Sasso, John (GE Power & Water, Non-GE)]

Inquiring about how btl_openib_receive_queues actually gets its default 
setting, since what I am seeing is not joving with documentation.  We are using 
OpenMPI 1.6.5, but I gather the version is moot.

Below is from ompi_info:

$ ompi_info --all | grep btl_openib_receive
 MCA btl: parameter "btl_openib_receive_queues" (current value: 
,
 data source: default value)


And based on the IB HCA part ID, this is the entry in 
mca-btl-openib-device-params.ini which is applicable:

# A.k.a. ConnectX
[Mellanox Hermon]
vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708,0x03ba,0x15b3,0x119f
vendor_part_id = 
25408,25418,25428,25448,26418,26428,26438,26448,26468,26478,26488,4099,4103,4100
use_eager_rdma = 1
mtu = 2048
max_inline_data = 128


Yet according to the doc in mca-btl-openib-device-params.ini, the default 
setting if receive_queues is not specified should be:

#receive_queues = P,128,256,192,128:S,65536,256,192,128

So comparing the two above, they don't coincide, in particular for the SRQ 
settings.  Thus, I gather the settings reported by ompi_info above are those 
that are actually used, and if so, I am wondering where/how they are derived?

--john

[OMPI users] Finding amount of pinned memory & regions

[Sasso, John (GE Power & Water, Non-GE)]

Pardon if this has been addressed already, but I could not find the answer 
after going through the OpenMPI FAQ and doing Google searches of the 
open-mpi.org site.

We are in the process of analyzing and troubleshooting MPI jobs of increasingly 
large scale (OpenMPI 1.6.5).  At a sufficiently large scale (# cores) a job 
will end up failing with errors similar to:

[y][[56933,1],1904][connect/btl_openib_connect_oob.c:867:rml_recv_cb] error 
in endpoint reply start connect
[x:29318] 853 more processes have sent help message 
help-mpi-btl-openib-cpc-base.txt / ibv_create_qp failed

So I know we are running into some memory limitation (educated guess) when 
queue pairs are being created to support such a huge mesh.  We are now 
investigating using the XRC transport to decrease memory consumption.

Anyways, my questions are:


1.   How do we determine HOW MUCH memory is being pinned by an MPI job on a 
node?  (If pmap, what exactly are we looking for?)

2.   How do we determine WHERE these pinned memory regions are?

We are running RedHat 6.x.  Thank you!

--john

[OMPI users] Why 3 threads per MPI task in pure MPI job?

[Sasso, John (GE Power & Water, Non-GE)]

I stumbled upon something while using 'ps -eFL' to view threads of processes, 
and Google searches have failed to answer my question.  This question holds for 
OpenMPI 1.6.x and even OpenMPI 1.4.x.

For a program which is pure MPI (built and run using OpenMPI) and does not 
implement Pthreads or OpenMP, why is it that each MPI task appears as having 3 
threads:

UID  PID  PPID   LWP  C NLWPSZ   RSS PSR STIME TTY  TIME CMD
sasso  20512 20493 20512 993 187849 582420 14 11:01 ?   00:26:37 
/home/sasso/mpi_example.exe
sasso  20512 20493 20588  03 187849 582420 11 11:01 ?   00:00:00 
/home/sasso/mpi_example.exe
sasso  20512 20493 20599  03 187849 582420 12 11:01 ?   00:00:00 
/home/sasso/mpi_example.exe

whereas if I compile and run a non-MPI program, 'ps -eFL' shows it running as a 
single thread?

Granted the CPU utilization (C) for 2 of the 3 threads is zero, but the threads 
are bound to different processors (11,12,14).   I am curious as to why this is, 
and no complaining that there is a problem.  Thanks!

--john

Re: [OMPI users] LAM/MPI -> OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

As far as I know, no MPI-IO is done in their LAM/MPI-based apps

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Rob Latham
Sent: Friday, February 27, 2015 11:22 AM
To: us...@open-mpi.org
Subject: Re: [OMPI users] LAM/MPI -> OpenMPI



On 02/27/2015 09:40 AM, Ralph Castain wrote:

>> Yeah, any other recommendations I can give to convince the 
>> powers-that-be that immediate sun-setting of LAM/MPI would be great.
>>  Sometimes I feel like I am trying to fit a square peg in a round 
>> holeL
>
> Other than the fact that LAM/MPI no longer is supported, the only real 
> rationale would be that OMPI has a lot of enhancements in terms of 
> binding options and other features, supports thru MPI-3, etc.

Does this application do any I/O?  I was curious so I dug around in LAM's 
suversion repository.  Last change to ROMIO, the MPI-IO implementation, was 
this one:

r10377 | brbarret | 2007-07-02 21:53:06

so you're missing out on 8 years of I/O related bug fixes and optimizations.


==rob

--
Rob Latham
Mathematics and Computer Science Division Argonne National Lab, IL USA 
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/02/26410.php

[OMPI users] LAM/MPI -> OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

Unfortunately, we have a few apps which use LAM/MPI instead of OpenMPI (and 
this is something I have NO control over).   I have been making an effort to 
try and convince those who handle such apps to move over to LAM/MPI as it is 
(as I understand it) no longer supported and end-of-life.   In fact, the 
website www.lam-mpi-org is no longer accessible as the 
domain name expired on 2/20/2015.

Aside from a change of library paths in the Makefiles, I am assuming that 
hardly any code changes would be needed for them to rebuild and run the code 
with OpenMPI.  Comments on this?

Yeah, any other recommendations I can give to convince the powers-that-be that 
immediate sun-setting of LAM/MPI would be great.   Sometimes I feel like I am 
trying to fit a square peg in a round hole :(

   --john

[OMPI users] OpenMPI build and OFED versions - rebuild necessary?

[Sasso, John (GE Power & Water, Non-GE)]

On RHEL 5 hosts which have OFED 1.5 installed, we have builds of OpenMPI 1.4.5 
and 1.6.5 in place.

On RHEL 6 hosts we have OFED 2.4 installed.  Will we need to rebuild OpenMPI 
1.4.5 and 1.6.5, or can our existing builds of such still work on the RHEL 6 
hosts?

--john

Re: [OMPI users] Determine IB transport type of OpenMPI job

[Sasso, John (GE Power & Water, Non-GE)]

Thank you!  Unless I missed something, I haven’t seen this in the OpenMPI FAQ.  
If it isn’t in there, it would be nice to include it.  Thanks!

--john

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Joshua Ladd
Sent: Friday, January 09, 2015 3:22 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determine IB transport type of OpenMPI job

Open MPI's openib BTL only supports RC transport.

Best,

Josh

Sent from my iPhone

On Jan 9, 2015, at 9:03 AM, "Sasso, John (GE Power & Water, Non-GE)" 
<john1.sa...@ge.com<mailto:john1.sa...@ge.com>> wrote:
For a multi-node job using OpenMPI 1.6.5 over InfiniBand where the OFED library 
is used, is there a way to tell what IB transport type is being used (RC, UC, 
UD, etc)?

---john

___
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26152.php

[OMPI users] Determine IB transport type of OpenMPI job

[Sasso, John (GE Power & Water, Non-GE)]

For a multi-node job using OpenMPI 1.6.5 over InfiniBand where the OFED library 
is used, is there a way to tell what IB transport type is being used (RC, UC, 
UD, etc)?

---john

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

I will be testing w/ 1.6.5 and update accordingly.  Thanks much!


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Sunday, June 08, 2014 7:40 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

Sorry about the comment re cpus-per-proc - confused this momentarily with 
another user also using Torque. I confirmed that this works fine with 1.6.5, 
and would guess you are hitting some bug in 1.6.0. Can you update?


On Jun 6, 2014, at 12:20 PM, Ralph Castain <r...@open-mpi.org> wrote:

> You might want to update to 1.6.5, if you can - I'll see what I can 
> find
> 
> On Jun 6, 2014, at 12:07 PM, Sasso, John (GE Power & Water, Non-GE) 
> <john1.sa...@ge.com> wrote:
> 
>> Version 1.6 (i.e. prior to 1.6.1)
>> 
>> -Original Message-
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>> Castain
>> Sent: Friday, June 06, 2014 3:03 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>> passes to OpenMPI
>> 
>> It's possible that you are hitting a bug - not sure how much the 
>> cpus-per-proc option has been exercised in 1.6. Is this 1.6.5, or some other 
>> member of that series?
>> 
>> I don't have a Torque machine handy any more, but should be able to 
>> test this scenario on my boxes
>> 
>> 
>> On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
>> <john1.sa...@ge.com> wrote:
>> 
>>> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
>>> --hostfile (i.e. the two are the same).  
>>> 
>>> To further debug this, I passed "--display-allocation --display-map" to 
>>> orterun, which resulted in:
>>> 
>>> ==   ALLOCATED NODES   ==
>>> 
>>> Data for node: node0001Num slots: 16   Max slots: 0
>>> Data for node: node0002  Num slots: 8Max slots: 0
>>> 
>>> =
>>> 
>>>    JOB MAP   
>>> 
>>> Data for node: node0001Num procs: 24
>>>  Process OMPI jobid: [24552,1] Process rank: 0
>>>  Process OMPI jobid: [24552,1] Process rank: 1
>>>  Process OMPI jobid: [24552,1] Process rank: 2
>>>  Process OMPI jobid: [24552,1] Process rank: 3
>>>  Process OMPI jobid: [24552,1] Process rank: 4
>>>  Process OMPI jobid: [24552,1] Process rank: 5
>>>  Process OMPI jobid: [24552,1] Process rank: 6
>>>  Process OMPI jobid: [24552,1] Process rank: 7
>>>  Process OMPI jobid: [24552,1] Process rank: 8
>>>  Process OMPI jobid: [24552,1] Process rank: 9
>>>  Process OMPI jobid: [24552,1] Process rank: 10
>>>  Process OMPI jobid: [24552,1] Process rank: 11
>>>  Process OMPI jobid: [24552,1] Process rank: 12
>>>  Process OMPI jobid: [24552,1] Process rank: 13
>>>  Process OMPI jobid: [24552,1] Process rank: 14
>>>  Process OMPI jobid: [24552,1] Process rank: 15
>>>  Process OMPI jobid: [24552,1] Process rank: 16
>>>  Process OMPI jobid: [24552,1] Process rank: 17
>>>  Process OMPI jobid: [24552,1] Process rank: 18
>>>  Process OMPI jobid: [24552,1] Process rank: 19
>>>  Process OMPI jobid: [24552,1] Process rank: 20
>>>  Process OMPI jobid: [24552,1] Process rank: 21
>>>  Process OMPI jobid: [24552,1] Process rank: 22
>>>  Process OMPI jobid: [24552,1] Process rank: 23
>>> 
>>> I have been going through the man page of mpirun as well as the OpenMPI 
>>> mailing list and website, and thus far have been unable to determine the 
>>> reason for the oversubscription of the head node (node0001) when even the 
>>> PBS scheduler is passing along the correct slot count #s (16 and 8, resp).
>>> 
>>> Am I running into a bug w/ OpenMPI 1.6?
>>> 
>>> --john
>>> 
>>> 
>>> 
>>> -Original Message-
>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>>> Castain
>>> Sent: Friday, June 06, 2014 1:30 PM
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>>> passes to OpenMPI
>>> 
>>> 
>>> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
>&g

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

Version 1.6 (i.e. prior to 1.6.1)

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 3:03 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

It's possible that you are hitting a bug - not sure how much the cpus-per-proc 
option has been exercised in 1.6. Is this 1.6.5, or some other member of that 
series?

I don't have a Torque machine handy any more, but should be able to test this 
scenario on my boxes


On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
<john1.sa...@ge.com> wrote:

> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
> --hostfile (i.e. the two are the same).  
> 
> To further debug this, I passed "--display-allocation --display-map" to 
> orterun, which resulted in:
> 
> ==   ALLOCATED NODES   ==
> 
> Data for node: node0001Num slots: 16   Max slots: 0
> Data for node: node0002  Num slots: 8Max slots: 0
> 
> =
> 
>    JOB MAP   
> 
> Data for node: node0001Num procs: 24
>Process OMPI jobid: [24552,1] Process rank: 0
>Process OMPI jobid: [24552,1] Process rank: 1
>Process OMPI jobid: [24552,1] Process rank: 2
>Process OMPI jobid: [24552,1] Process rank: 3
>Process OMPI jobid: [24552,1] Process rank: 4
>Process OMPI jobid: [24552,1] Process rank: 5
>Process OMPI jobid: [24552,1] Process rank: 6
>Process OMPI jobid: [24552,1] Process rank: 7
>Process OMPI jobid: [24552,1] Process rank: 8
>Process OMPI jobid: [24552,1] Process rank: 9
>Process OMPI jobid: [24552,1] Process rank: 10
>Process OMPI jobid: [24552,1] Process rank: 11
>Process OMPI jobid: [24552,1] Process rank: 12
>Process OMPI jobid: [24552,1] Process rank: 13
>Process OMPI jobid: [24552,1] Process rank: 14
>Process OMPI jobid: [24552,1] Process rank: 15
>Process OMPI jobid: [24552,1] Process rank: 16
>Process OMPI jobid: [24552,1] Process rank: 17
>Process OMPI jobid: [24552,1] Process rank: 18
>Process OMPI jobid: [24552,1] Process rank: 19
>Process OMPI jobid: [24552,1] Process rank: 20
>Process OMPI jobid: [24552,1] Process rank: 21
>Process OMPI jobid: [24552,1] Process rank: 22
>Process OMPI jobid: [24552,1] Process rank: 23
> 
> I have been going through the man page of mpirun as well as the OpenMPI 
> mailing list and website, and thus far have been unable to determine the 
> reason for the oversubscription of the head node (node0001) when even the PBS 
> scheduler is passing along the correct slot count #s (16 and 8, resp).
> 
> Am I running into a bug w/ OpenMPI 1.6?
> 
> --john
> 
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
> Castain
> Sent: Friday, June 06, 2014 1:30 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler 
> passes to OpenMPI
> 
> 
> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
> 
>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>> You can always add --display-allocation to the cmd line to see what 
>>> we thought we received.
>>> 
>>> If you configure OMPI with --enable-debug, you can set --mca 
>>> ras_base_verbose 10 to see the details
>>> 
>>> 
>> 
>> Hi John
>> 
>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>> script.  This will list the nodes (multiple times according to the number of 
>> cores requested).
>> I find this useful documentation,
>> along with job number, work directory, etc.
>> "man qsub" will show you all the PBS_* environment variables 
>> available to the job.
>> For instance, you can echo them using a Torque 'prolog' script, if 
>> the user didn't do it. That will appear in the Torque STDOUT file.
>> 
>> From outside the job script, "qstat -n" (and variants, say, with -u
>> username) will list the nodes allocated to each job, again multiple 
>> times as per the requested cores.
>> 
>> "tracejob job_number" will show similar information.
>> 
>> 
>> If you configured Torque --with-cpuset, there is more information 
>> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
>> (on the first node listed above, called "mo

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
--hostfile (i.e. the two are the same).  

To further debug this, I passed "--display-allocation --display-map" to 
orterun, which resulted in:

==   ALLOCATED NODES   ==

 Data for node: node0001Num slots: 16   Max slots: 0
 Data for node: node0002  Num slots: 8Max slots: 0

=

    JOB MAP   

 Data for node: node0001Num procs: 24
Process OMPI jobid: [24552,1] Process rank: 0
Process OMPI jobid: [24552,1] Process rank: 1
Process OMPI jobid: [24552,1] Process rank: 2
Process OMPI jobid: [24552,1] Process rank: 3
Process OMPI jobid: [24552,1] Process rank: 4
Process OMPI jobid: [24552,1] Process rank: 5
Process OMPI jobid: [24552,1] Process rank: 6
Process OMPI jobid: [24552,1] Process rank: 7
Process OMPI jobid: [24552,1] Process rank: 8
Process OMPI jobid: [24552,1] Process rank: 9
Process OMPI jobid: [24552,1] Process rank: 10
Process OMPI jobid: [24552,1] Process rank: 11
Process OMPI jobid: [24552,1] Process rank: 12
Process OMPI jobid: [24552,1] Process rank: 13
Process OMPI jobid: [24552,1] Process rank: 14
Process OMPI jobid: [24552,1] Process rank: 15
Process OMPI jobid: [24552,1] Process rank: 16
Process OMPI jobid: [24552,1] Process rank: 17
Process OMPI jobid: [24552,1] Process rank: 18
Process OMPI jobid: [24552,1] Process rank: 19
Process OMPI jobid: [24552,1] Process rank: 20
Process OMPI jobid: [24552,1] Process rank: 21
Process OMPI jobid: [24552,1] Process rank: 22
Process OMPI jobid: [24552,1] Process rank: 23

I have been going through the man page of mpirun as well as the OpenMPI mailing 
list and website, and thus far have been unable to determine the reason for the 
oversubscription of the head node (node0001) when even the PBS scheduler is 
passing along the correct slot count #s (16 and 8, resp).

Am I running into a bug w/ OpenMPI 1.6?

--john



-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 1:30 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI


On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:

> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>> You can always add --display-allocation to the cmd line to see what 
>> we thought we received.
>> 
>> If you configure OMPI with --enable-debug, you can set --mca 
>> ras_base_verbose 10 to see the details
>> 
>> 
> 
> Hi John
> 
> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job script. 
>  This will list the nodes (multiple times according to the number of cores 
> requested).
> I find this useful documentation,
> along with job number, work directory, etc.
> "man qsub" will show you all the PBS_* environment variables available 
> to the job.
> For instance, you can echo them using a Torque 'prolog' script, if the 
> user didn't do it. That will appear in the Torque STDOUT file.
> 
> From outside the job script, "qstat -n" (and variants, say, with -u 
> username) will list the nodes allocated to each job, again multiple 
> times as per the requested cores.
> 
> "tracejob job_number" will show similar information.
> 
> 
> If you configured Torque --with-cpuset, there is more information 
> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
> (on the first node listed above, called "mother superior" in Torque parlance).
> This mostly matter if there is more than one job running on a node.
> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
> whatever.  As Ralph said, Open MPI does that.
> I believe Open MPI doesn't use the cpuset info from Torque.
> (Ralph, please correct me if I am wrong.)

You are correct in that we don't use any per-process designations. We do, 
however, work inside any overall envelope that Torque may impose on us - e.g., 
if you tell Torque to limit the job to cores 0-4, we will honor that directive 
and keep all processes within that envelope.


> 
> My two cents,
> Gus Correa
> 
> 
>> On Jun 6, 2014, at 10:01 AM, Reuti <re...@staff.uni-marburg.de 
>> <mailto:re...@staff.uni-marburg.de>> wrote:
>> 
>>> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
>>> 
>>>> OK, so at the least, how can I get the node and slots/node info 
>>>> that is passed from PBS?

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

OK, so at the least, how can I get the node and slots/node info that is passed 
from PBS?

I ask because I'm trying to troubleshoot a problem w/ PBS and the build of 
OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using a 
script which has:

/usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
--mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
/home/sasso/TEST/simplempihello.exe


And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 16 
being host node0001 and the last 8 being node0002), it appears that 24 MPI 
tasks try to start on node0001 instead of getting distributed as 16 on node0001 
and 8 on node0002.   Hence, I am curious what is being passed by PBS.

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 12:31 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

We currently only get the node and slots/node info from PBS - we don't get any 
task placement info at all. We then use the mpirun cmd options and built-in 
mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't really seen 
a reason to do so - the OMPI mappers are generally more flexible than anything 
in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
<john1.sa...@ge.com<mailto:john1.sa...@ge.com>> wrote:


For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
include files + libs, is there a way to determine (perhaps via some debugging 
flags passed to mpirun) what job placement parameters are passed from the PBS 
scheduler to OpenMPI?  In particular, I am talking about task placement info 
such as nodes to place on, etc.   Thanks!

  --john
___
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
include files + libs, is there a way to determine (perhaps via some debugging 
flags passed to mpirun) what job placement parameters are passed from the PBS 
scheduler to OpenMPI?  In particular, I am talking about task placement info 
such as nodes to place on, etc.   Thanks!

  --john

Re: [OMPI users] IMB Sendrecv hangs with OpenMPI 1.6.5 and XRC

[Sasso, John (GE Power & Water, Non-GE)]

Thank-you Jeff.  I re-ran IMB (a 64-core run, distributed across a number of 
nodes) under different mca parameters.  Here are the results using OpenMPI 
1.6.5:

1. --mca btl openib,sm,self --mca btl_openib_receive_queues 
X,9216,256,128,32:X,65536,256,128,32
IMB did not hang.  Consumed 9263 sec (aggregate) CPU time and 8986 MB 
memory

2. --mca btl openib,sm,self --mca btl_openib_receive_queues 
X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32:X,65536,256,128,32
IMB hung on Bcast benchmark on a 64-process run, with message size of 
64 bytes

3. --mca btl openib,sm,self
IMB did not hang.  Consumed 9360 sec (aggregate) CPU time and 9360 MB 
memory

4. --mca btl openib,tcp,self
IMB did not hang.  Consumed 41911 sec (aggregate) CPU time and 9239 MB 
memory

I did not try OpenMPI 1.8.1 since I am restricted to 1.6.5 at this time, but 
I'm doing a build of 1.8.1 now to test out.  BTW, the release notes refer to 
1.8.2 but the site only has 1.8.1 available for download.

I am a bit concerned, however, with my prior runs hanging.  First, I was unable 
to discern why IMB was hanging so any advice/guidance would be greatly 
appreciated.  I tried doing an strace on an MPI process but no helpful info.   

Second, the motivation behind using XRC was to cut down on memory demands 
w.r.t. the RC QPs.   I'd like to get this working, unless someone can elaborate 
on the negative aspects of using XRC instead of RC QPs.  Thanks!

--john


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Jeff Squyres 
(jsquyres)
Sent: Wednesday, April 23, 2014 11:19 AM
To: Open MPI Users
Subject: Re: [OMPI users] IMB Sendrecv hangs with OpenMPI 1.6.5 and XRC

A few suggestions:

- Try using Open MPI 1.8.1.  It's the newest release, and has many improvements 
since the 1.6.x series.

- Try using "--mca btl openib,sm,self" (in both v1.6.x and v1.8.x).  This 
allows Open MPI to use shared memory to communicate between processes on the 
same server, which can be a significant performance improvement over TCP or 
even IB.



On Apr 23, 2014, at 11:10 AM, "Sasso, John (GE Power & Water, Non-GE)" 
<john1.sa...@ge.com> wrote:

> I am running IMB (Intel MPI Benchmarks), the MPI-1 benchmarks, which was 
> built with Intel 12.1 compiler suite and OpenMPI 1.6.5 (and running w/ OMPI 
> 1.6.5).  I decided to use the following for the mca parameters:
>  
> --mca btl openib,tcp,self --mca btl_openib_receive_queues 
> X,9216,256,128,32:X,65536,256,128,32
>  
> where before, I always used "--mca btl openib,tcp,self".  This is for 
> performance analysis.  On the SendRecv benchmark at 32 processes, IMB hangs.  
> I then tried:
>  
> --mca btl_openib_receive_queues 
> X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32:X,65536,256,128,32
>  
> and IMB also hangs on the SendRecv benchmark, though at 64 processes.
>  
> No errors have been recorded, not even in any system log files but 'top' 
> shows the MPI tasks running.  How can I go about troubleshooting this hang, 
> as well as figuring out what (If any) MCA XRC-related parameters in 
> btl_openib_receive_queues I have to specify to get IMB running properly?   I 
> did verify the IB cards are ConnectX.
>  
> --john
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] IMB Sendrecv hangs with OpenMPI 1.6.5 and XRC

[Sasso, John (GE Power & Water, Non-GE)]

I am running IMB (Intel MPI Benchmarks), the MPI-1 benchmarks, which was built 
with Intel 12.1 compiler suite and OpenMPI 1.6.5 (and running w/ OMPI 1.6.5).  
I decided to use the following for the mca parameters:

--mca btl openib,tcp,self --mca btl_openib_receive_queues 
X,9216,256,128,32:X,65536,256,128,32

where before, I always used "--mca btl openib,tcp,self".  This is for 
performance analysis.  On the SendRecv benchmark at 32 processes, IMB hangs.  I 
then tried:

--mca btl_openib_receive_queues 
X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32:X,65536,256,128,32

and IMB also hangs on the SendRecv benchmark, though at 64 processes.

No errors have been recorded, not even in any system log files but 'top' shows 
the MPI tasks running.  How can I go about troubleshooting this hang, as well 
as figuring out what (If any) MCA XRC-related parameters in 
btl_openib_receive_queues I have to specify to get IMB running properly?   I 
did verify the IB cards are ConnectX.

--john

Re: [OMPI users] File locking in ADIO, OpenMPI 1.6.4

[Sasso, John (GE Power & Water, Non-GE)]

Dan, 

On the hosts where the ADIOI lock error occurs, are there any NFS errors in 
/var/log/messages, dmesg, or similar that refer to lockd?

--john


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Daniel Milroy
Sent: Tuesday, April 15, 2014 10:55 AM
To: Open MPI Users
Subject: Re: [OMPI users] File locking in ADIO, OpenMPI 1.6.4

Hi Rob,

The applications of the two users in question are different; I haven¹t looked 
through much of either code.  I can respond to your highlighted situations in 
sequence:

>- everywhere in NFS.  If you have a Lustre file system exported to some 
>clients as NFS, you'll get NFS (er, that might not be true unless you 
>pick up a recent patch)
The compute nodes are Lustre clients mounting the file system via IB.

>- note: you don't need to disable data sieving for reads, though you 
>might want to if the data sieving algorithm is wasting a lot of data.
That¹s good to know, though given the applications I can¹t say whether data 
sieving is wasting data.

>- if atomic mode was set on the file (i.e. you called
>MPI_File_set_atomicity)
>- if you use any of the shared file pointer operations
>- if you use any of the ordered mode collective operations
I don¹t know but will pass these questions on to the users.



Thank you,

Dan Milroy




On 4/14/14, 2:23 PM, "Rob Latham"  wrote:

>
>
>On 04/08/2014 05:49 PM, Daniel Milroy wrote:
>> Hello,
>>
>> The file system in question is indeed Lustre, and mounting with flock 
>> isn¹t possible in our environment.  I recommended the following 
>> changes to the users¹ code:
>
>Hi.  I'm the ROMIO guy, though I do rely on the community to help me 
>keep the lustre driver up to snuff.
>
>> MPI_Info_set(info, "collective_buffering", "true"); 
>> MPI_Info_set(info, "romio_lustre_ds_in_coll", "disable"); 
>> MPI_Info_set(info, "romio_ds_read", "disable"); MPI_Info_set(info, 
>> "romio_ds_write", "disable");
>>
>> Which results in the same error as before.  Are there any other MPI 
>> options I can set?
>
>I'd like to hear more about the workload generating these lock 
>messages, but I can tell you the situations in which ADIOI_SetLock gets called:
>- everywhere in NFS.  If you have a Lustre file system exported to some 
>clients as NFS, you'll get NFS (er, that might not be true unless you 
>pick up a recent patch)
>- when writing a non-contiguous region in file, unless you disable data 
>sieving, as you did above.
>- note: you don't need to disable data sieving for reads, though you 
>might want to if the data sieving algorithm is wasting a lot of data.
>- if atomic mode was set on the file (i.e. you called
>MPI_File_set_atomicity)
>- if you use any of the shared file pointer operations
>- if you use any of the ordered mode collective operations
>
>you've turned off data sieving writes, which is what I would have first 
>guessed would trigger this lock message.  So I guess you are hitting 
>one of the other cases.
>
>==rob
>
>--
>Rob Latham
>Mathematics and Computer Science Division Argonne National Lab, IL USA 
>___
>users mailing list
>us...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

[Sasso, John (GE Power & Water, Non-GE)]

Thanks again!  I tried --display-devel-map and I think it provides a bit too 
much info for our needs.  However, it is nice to know.

BTW, some interesting behavior in using "--report-bindings --bind-to-core" vs 
"--display-map".  

*  If I use "--report-bindings --bind-to-core" but the MPI tasks on a host fail 
to start up, then nothing is reported.  For example, I had a problem where a 
job started across 4 hosts but the hosts could not communicate with one another 
via TCP/IP.

* If I use "--display-map" then the mapping is shown, even in the failure case 
I mentioned in the last bullet.

* What is nice about "--report-bindings --bind-to-core" over "--display-map" is 
that it will report the binding of each rank to CPU, whereas the latter will 
show you what  ranks are running on a given host.  For our needs, this may be 
sufficient, tho it would be nice to have the CPU bindings shown as well

* If using "--report-bindings --bind-to-core" with OpenMPI 1.4.1 then the 
bindings on just the head node are shown.  In 1.6.1, full bindings across all 
hosts are shown.  (I'd have to read release notes on this...)

--john


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Thursday, March 27, 2014 7:01 PM
To: Open MPI Users
Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

Oooh...it's Jeff's fault!

Fwiw you can get even more detailed mapping info with --display-devel-map

Sent from my iPhone

> On Mar 27, 2014, at 2:58 PM, "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> 
> wrote:
> 
>> On Mar 27, 2014, at 4:06 PM, "Sasso, John (GE Power & Water, Non-GE)" 
>> <john1.sa...@ge.com> wrote:
>> 
>> Yes, I noticed that I could not find --display-map in any of the man pages.  
>> Intentional?
> 
> Oops; nope.  I'll ask Ralph to add it...
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

[Sasso, John (GE Power & Water, Non-GE)]

Yes, I noticed that I could not find --display-map in any of the man pages.  
Intentional?


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
Sent: Thursday, March 27, 2014 3:26 PM
To: Open MPI Users
Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

On 03/27/2014 03:02 PM, Ralph Castain wrote:
> Or use --display-map to see the process to node assignments
>

Aha!
That one was not on my radar.
Maybe because somehow I can't find it in the OMPI 1.6.5 mpiexec man page.
However, it seems to work with that version also, which is great.
(--display-map goes to stdout, whereas -report-bindings goes to stderr,
right?)
Thanks, Ralph!

Gus Correa

> Sent from my iPhone
>
>> On Mar 27, 2014, at 11:47 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
>>
>> PS - The (OMPI 1.6.5) mpiexec default is -bind-to-none, in which case 
>> -report-bindings won't report anything.
>>
>> So, if you are using the default,
>> you can apply Joe Landman's suggestion (or alternatively use the 
>> MPI_Get_processor_name function, in lieu of uname(); cpu_name = 
>> uts.nodename; ).
>>
>> However, many MPI applications benefit from some type of hardware 
>> binding, maybe yours will do also, and as a bonus -report-bindings will tell 
>> you where each rank ran.
>> mpiexec's -tag-output is also helpful for debugging, but won't tell 
>> you the node name, just the MPI rank.
>>
>> You can setup a lot of these things as your preferred defaults, via 
>> mca parameters, and omit them from the mpiexec command line.
>> The trick is to match each mpiexec option to the appropriate mca 
>> parameter, as the names are not exactly the same.
>> "ompi-info --all" may help in that regard.
>> See this FAQ:
>> http://www.open-mpi.org/faq/?category=tuning#setting-mca-params
>>
>> Again, the OMPI FAQ page is your friend!  :) 
>> http://www.open-mpi.org/faq/
>>
>> I hope this helps,
>> Gus Correa
>>
>>> On 03/27/2014 02:06 PM, Gus Correa wrote:
>>> Hi John
>>>
>>> Take a look at the mpiexec/mpirun options:
>>>
>>> -report-bindings (this one should report what you want)
>>>
>>> and maybe also also:
>>>
>>> -bycore, -bysocket, -bind-to-core, -bind-to-socket, ...
>>>
>>> and similar, if you want more control on where your MPI processes run.
>>>
>>> "man mpiexec" is your friend!
>>>
>>> I hope this helps,
>>> Gus Correa
>>>
>>>> On 03/27/2014 01:53 PM, Sasso, John (GE Power & Water, Non-GE) wrote:
>>>> When a piece of software built against OpenMPI fails, I will see an 
>>>> error referring to the rank of the MPI task which incurred the failure.
>>>> For example:
>>>>
>>>> MPI_ABORT was invoked on rank 1236 in communicator MPI_COMM_WORLD
>>>>
>>>> with errorcode 1.
>>>>
>>>> Unfortunately, I do not have access to the software code, just the 
>>>> installation directory tree for OpenMPI.  My question is:  Is there 
>>>> a flag that can be passed to mpirun, or an environment variable 
>>>> set, which would reveal the mapping of ranks to the hosts they are on?
>>>>
>>>> I do understand that one could have multiple MPI ranks running on 
>>>> the same host, but finding a way to determine which rank ran on 
>>>> what host would go a long way in help troubleshooting problems 
>>>> which may be central to the host.  Thanks!
>>>>
>>>>--john
>>>>
>>>>
>>>>
>>>> ___
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

[Sasso, John (GE Power & Water, Non-GE)]

Thank you!  That also works and is very helpful.

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Thursday, March 27, 2014 3:03 PM
To: Open MPI Users
Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

Or use --display-map to see the process to node assignments

Sent from my iPhone

> On Mar 27, 2014, at 11:47 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
> 
> PS - The (OMPI 1.6.5) mpiexec default is -bind-to-none, in which case 
> -report-bindings won't report anything.
> 
> So, if you are using the default,
> you can apply Joe Landman's suggestion (or alternatively use the 
> MPI_Get_processor_name function, in lieu of uname(); cpu_name = 
> uts.nodename; ).
> 
> However, many MPI applications benefit from some type of hardware 
> binding, maybe yours will do also, and as a bonus -report-bindings will tell 
> you where each rank ran.
> mpiexec's -tag-output is also helpful for debugging, but won't tell 
> you the node name, just the MPI rank.
> 
> You can setup a lot of these things as your preferred defaults, via 
> mca parameters, and omit them from the mpiexec command line.
> The trick is to match each mpiexec option to the appropriate mca 
> parameter, as the names are not exactly the same.
> "ompi-info --all" may help in that regard.
> See this FAQ:
> http://www.open-mpi.org/faq/?category=tuning#setting-mca-params
> 
> Again, the OMPI FAQ page is your friend!  :) 
> http://www.open-mpi.org/faq/
> 
> I hope this helps,
> Gus Correa
> 
>> On 03/27/2014 02:06 PM, Gus Correa wrote:
>> Hi John
>> 
>> Take a look at the mpiexec/mpirun options:
>> 
>> -report-bindings (this one should report what you want)
>> 
>> and maybe also also:
>> 
>> -bycore, -bysocket, -bind-to-core, -bind-to-socket, ...
>> 
>> and similar, if you want more control on where your MPI processes run.
>> 
>> "man mpiexec" is your friend!
>> 
>> I hope this helps,
>> Gus Correa
>> 
>>> On 03/27/2014 01:53 PM, Sasso, John (GE Power & Water, Non-GE) wrote:
>>> When a piece of software built against OpenMPI fails, I will see an 
>>> error referring to the rank of the MPI task which incurred the failure.
>>> For example:
>>> 
>>> MPI_ABORT was invoked on rank 1236 in communicator MPI_COMM_WORLD
>>> 
>>> with errorcode 1.
>>> 
>>> Unfortunately, I do not have access to the software code, just the 
>>> installation directory tree for OpenMPI.  My question is:  Is there 
>>> a flag that can be passed to mpirun, or an environment variable set, 
>>> which would reveal the mapping of ranks to the hosts they are on?
>>> 
>>> I do understand that one could have multiple MPI ranks running on 
>>> the same host, but finding a way to determine which rank ran on what 
>>> host would go a long way in help troubleshooting problems which may 
>>> be central to the host.  Thanks!
>>> 
>>>   --john
>>> 
>>> 
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

[Sasso, John (GE Power & Water, Non-GE)]

Thank you, Gus!  I did go through the mpiexec/mpirun man pages but wasn't quite 
clear that -report-bindings was what I was looking for.   So what I did is 
rerun a program w/ --report-bindings but no bindings were reported.

Scratching my head, I decided to include --bind-to-core as well.  Voila, the 
bindings are reported!  

Awesome, but now here is my concern.  If we have OpenMPI-based applications 
launched as batch jobs via a batch scheduler like SLURM, PBS, LSF, etc. (which 
decides the placement of the app and dispatches it to the compute hosts), then 
will including "--report-bindings --bind-to-core" cause problems?   Certainly I 
can test this, but concerned there may be a case where inclusion of 
--bind-to-core would cause an unexpected problem I did not account for.

--john


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
Sent: Thursday, March 27, 2014 2:06 PM
To: Open MPI Users
Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

Hi John

Take a look at the mpiexec/mpirun options:

-report-bindings (this one should report what you want)

and maybe also also:

-bycore, -bysocket, -bind-to-core, -bind-to-socket, ...

and similar, if you want more control on where your MPI processes run.

"man mpiexec" is your friend!

I hope this helps,
Gus Correa

On 03/27/2014 01:53 PM, Sasso, John (GE Power & Water, Non-GE) wrote:
> When a piece of software built against OpenMPI fails, I will see an 
> error referring to the rank of the MPI task which incurred the failure.
> For example:
>
> MPI_ABORT was invoked on rank 1236 in communicator MPI_COMM_WORLD
>
> with errorcode 1.
>
> Unfortunately, I do not have access to the software code, just the 
> installation directory tree for OpenMPI.  My question is:  Is there a 
> flag that can be passed to mpirun, or an environment variable set, 
> which would reveal the mapping of ranks to the hosts they are on?
>
> I do understand that one could have multiple MPI ranks running on the 
> same host, but finding a way to determine which rank ran on what host 
> would go a long way in help troubleshooting problems which may be 
> central to the host.  Thanks!
>
>--john
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Mapping ranks to hosts (from MPI error messages)

[Sasso, John (GE Power & Water, Non-GE)]

When a piece of software built against OpenMPI fails, I will see an error 
referring to the rank of the MPI task which incurred the failure.  For example:

MPI_ABORT was invoked on rank 1236 in communicator MPI_COMM_WORLD
with errorcode 1.

Unfortunately, I do not have access to the software code, just the installation 
directory tree for OpenMPI.  My question is:  Is there a flag that can be 
passed to mpirun, or an environment variable set, which would reveal the 
mapping of ranks to the hosts they are on?

I do understand that one could have multiple MPI ranks running on the same 
host, but finding a way to determine which rank ran on what host would go a 
long way in help troubleshooting problems which may be central to the host.  
Thanks!

  --john