Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-09 Thread Sasso, John (GE Power & Water, Non-GE) 
I will be testing w/ 1.6.5 and update accordingly.  Thanks much!


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Sunday, June 08, 2014 7:40 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

Sorry about the comment re cpus-per-proc - confused this momentarily with 
another user also using Torque. I confirmed that this works fine with 1.6.5, 
and would guess you are hitting some bug in 1.6.0. Can you update?


On Jun 6, 2014, at 12:20 PM, Ralph Castain <r...@open-mpi.org> wrote:

> You might want to update to 1.6.5, if you can - I'll see what I can 
> find
> 
> On Jun 6, 2014, at 12:07 PM, Sasso, John (GE Power & Water, Non-GE) 
> <john1.sa...@ge.com> wrote:
> 
>> Version 1.6 (i.e. prior to 1.6.1)
>> 
>> -Original Message-
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>> Castain
>> Sent: Friday, June 06, 2014 3:03 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>> passes to OpenMPI
>> 
>> It's possible that you are hitting a bug - not sure how much the 
>> cpus-per-proc option has been exercised in 1.6. Is this 1.6.5, or some other 
>> member of that series?
>> 
>> I don't have a Torque machine handy any more, but should be able to 
>> test this scenario on my boxes
>> 
>> 
>> On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
>> <john1.sa...@ge.com> wrote:
>> 
>>> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
>>> --hostfile (i.e. the two are the same).  
>>> 
>>> To further debug this, I passed "--display-allocation --display-map" to 
>>> orterun, which resulted in:
>>> 
>>> ==   ALLOCATED NODES   ==
>>> 
>>> Data for node: node0001Num slots: 16   Max slots: 0
>>> Data for node: node0002  Num slots: 8Max slots: 0
>>> 
>>> =
>>> 
>>>    JOB MAP   
>>> 
>>> Data for node: node0001Num procs: 24
>>>  Process OMPI jobid: [24552,1] Process rank: 0
>>>  Process OMPI jobid: [24552,1] Process rank: 1
>>>  Process OMPI jobid: [24552,1] Process rank: 2
>>>  Process OMPI jobid: [24552,1] Process rank: 3
>>>  Process OMPI jobid: [24552,1] Process rank: 4
>>>  Process OMPI jobid: [24552,1] Process rank: 5
>>>  Process OMPI jobid: [24552,1] Process rank: 6
>>>  Process OMPI jobid: [24552,1] Process rank: 7
>>>  Process OMPI jobid: [24552,1] Process rank: 8
>>>  Process OMPI jobid: [24552,1] Process rank: 9
>>>  Process OMPI jobid: [24552,1] Process rank: 10
>>>  Process OMPI jobid: [24552,1] Process rank: 11
>>>  Process OMPI jobid: [24552,1] Process rank: 12
>>>  Process OMPI jobid: [24552,1] Process rank: 13
>>>  Process OMPI jobid: [24552,1] Process rank: 14
>>>  Process OMPI jobid: [24552,1] Process rank: 15
>>>  Process OMPI jobid: [24552,1] Process rank: 16
>>>  Process OMPI jobid: [24552,1] Process rank: 17
>>>  Process OMPI jobid: [24552,1] Process rank: 18
>>>  Process OMPI jobid: [24552,1] Process rank: 19
>>>  Process OMPI jobid: [24552,1] Process rank: 20
>>>  Process OMPI jobid: [24552,1] Process rank: 21
>>>  Process OMPI jobid: [24552,1] Process rank: 22
>>>  Process OMPI jobid: [24552,1] Process rank: 23
>>> 
>>> I have been going through the man page of mpirun as well as the OpenMPI 
>>> mailing list and website, and thus far have been unable to determine the 
>>> reason for the oversubscription of the head node (node0001) when even the 
>>> PBS scheduler is passing along the correct slot count #s (16 and 8, resp).
>>> 
>>> Am I running into a bug w/ OpenMPI 1.6?
>>> 
>>> --john
>>> 
>>> 
>>> 
>>> -Original Message-
>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>>> Castain
>>> Sent: Friday, June 06, 2014 1:30 PM
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>>> passes to OpenMPI
>>> 
>>> 
>>> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
>&g

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-08 Thread Ralph Castain 
Sorry about the comment re cpus-per-proc - confused this momentarily with 
another user also using Torque. I confirmed that this works fine with 1.6.5, 
and would guess you are hitting some bug in 1.6.0. Can you update?


On Jun 6, 2014, at 12:20 PM, Ralph Castain <r...@open-mpi.org> wrote:

> You might want to update to 1.6.5, if you can - I'll see what I can find
> 
> On Jun 6, 2014, at 12:07 PM, Sasso, John (GE Power & Water, Non-GE) 
> <john1.sa...@ge.com> wrote:
> 
>> Version 1.6 (i.e. prior to 1.6.1)
>> 
>> -Original Message-
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
>> Sent: Friday, June 06, 2014 3:03 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
>> OpenMPI
>> 
>> It's possible that you are hitting a bug - not sure how much the 
>> cpus-per-proc option has been exercised in 1.6. Is this 1.6.5, or some other 
>> member of that series?
>> 
>> I don't have a Torque machine handy any more, but should be able to test 
>> this scenario on my boxes
>> 
>> 
>> On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
>> <john1.sa...@ge.com> wrote:
>> 
>>> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
>>> --hostfile (i.e. the two are the same).  
>>> 
>>> To further debug this, I passed "--display-allocation --display-map" to 
>>> orterun, which resulted in:
>>> 
>>> ==   ALLOCATED NODES   ==
>>> 
>>> Data for node: node0001Num slots: 16   Max slots: 0
>>> Data for node: node0002  Num slots: 8Max slots: 0
>>> 
>>> =
>>> 
>>>    JOB MAP   
>>> 
>>> Data for node: node0001Num procs: 24
>>>  Process OMPI jobid: [24552,1] Process rank: 0
>>>  Process OMPI jobid: [24552,1] Process rank: 1
>>>  Process OMPI jobid: [24552,1] Process rank: 2
>>>  Process OMPI jobid: [24552,1] Process rank: 3
>>>  Process OMPI jobid: [24552,1] Process rank: 4
>>>  Process OMPI jobid: [24552,1] Process rank: 5
>>>  Process OMPI jobid: [24552,1] Process rank: 6
>>>  Process OMPI jobid: [24552,1] Process rank: 7
>>>  Process OMPI jobid: [24552,1] Process rank: 8
>>>  Process OMPI jobid: [24552,1] Process rank: 9
>>>  Process OMPI jobid: [24552,1] Process rank: 10
>>>  Process OMPI jobid: [24552,1] Process rank: 11
>>>  Process OMPI jobid: [24552,1] Process rank: 12
>>>  Process OMPI jobid: [24552,1] Process rank: 13
>>>  Process OMPI jobid: [24552,1] Process rank: 14
>>>  Process OMPI jobid: [24552,1] Process rank: 15
>>>  Process OMPI jobid: [24552,1] Process rank: 16
>>>  Process OMPI jobid: [24552,1] Process rank: 17
>>>  Process OMPI jobid: [24552,1] Process rank: 18
>>>  Process OMPI jobid: [24552,1] Process rank: 19
>>>  Process OMPI jobid: [24552,1] Process rank: 20
>>>  Process OMPI jobid: [24552,1] Process rank: 21
>>>  Process OMPI jobid: [24552,1] Process rank: 22
>>>  Process OMPI jobid: [24552,1] Process rank: 23
>>> 
>>> I have been going through the man page of mpirun as well as the OpenMPI 
>>> mailing list and website, and thus far have been unable to determine the 
>>> reason for the oversubscription of the head node (node0001) when even the 
>>> PBS scheduler is passing along the correct slot count #s (16 and 8, resp).
>>> 
>>> Am I running into a bug w/ OpenMPI 1.6?
>>> 
>>> --john
>>> 
>>> 
>>> 
>>> -Original Message-
>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>>> Castain
>>> Sent: Friday, June 06, 2014 1:30 PM
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>>> passes to OpenMPI
>>> 
>>> 
>>> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
>>> 
>>>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>>>> You can always add --display-allocation to the cmd line to see what 
>>>>> we thought we received.
>>>>> 
>>>>> If you configure OMPI with --enable-debug, you can set --mca 
>>>>> ras_base_v

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Ralph Castain 
You might want to update to 1.6.5, if you can - I'll see what I can find

On Jun 6, 2014, at 12:07 PM, Sasso, John (GE Power & Water, Non-GE) 
<john1.sa...@ge.com> wrote:

> Version 1.6 (i.e. prior to 1.6.1)
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 3:03 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
> OpenMPI
> 
> It's possible that you are hitting a bug - not sure how much the 
> cpus-per-proc option has been exercised in 1.6. Is this 1.6.5, or some other 
> member of that series?
> 
> I don't have a Torque machine handy any more, but should be able to test this 
> scenario on my boxes
> 
> 
> On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
> <john1.sa...@ge.com> wrote:
> 
>> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
>> --hostfile (i.e. the two are the same).  
>> 
>> To further debug this, I passed "--display-allocation --display-map" to 
>> orterun, which resulted in:
>> 
>> ==   ALLOCATED NODES   ==
>> 
>> Data for node: node0001Num slots: 16   Max slots: 0
>> Data for node: node0002  Num slots: 8Max slots: 0
>> 
>> =
>> 
>>    JOB MAP   
>> 
>> Data for node: node0001Num procs: 24
>>   Process OMPI jobid: [24552,1] Process rank: 0
>>   Process OMPI jobid: [24552,1] Process rank: 1
>>   Process OMPI jobid: [24552,1] Process rank: 2
>>   Process OMPI jobid: [24552,1] Process rank: 3
>>   Process OMPI jobid: [24552,1] Process rank: 4
>>   Process OMPI jobid: [24552,1] Process rank: 5
>>   Process OMPI jobid: [24552,1] Process rank: 6
>>   Process OMPI jobid: [24552,1] Process rank: 7
>>   Process OMPI jobid: [24552,1] Process rank: 8
>>   Process OMPI jobid: [24552,1] Process rank: 9
>>   Process OMPI jobid: [24552,1] Process rank: 10
>>   Process OMPI jobid: [24552,1] Process rank: 11
>>   Process OMPI jobid: [24552,1] Process rank: 12
>>   Process OMPI jobid: [24552,1] Process rank: 13
>>   Process OMPI jobid: [24552,1] Process rank: 14
>>   Process OMPI jobid: [24552,1] Process rank: 15
>>   Process OMPI jobid: [24552,1] Process rank: 16
>>   Process OMPI jobid: [24552,1] Process rank: 17
>>   Process OMPI jobid: [24552,1] Process rank: 18
>>   Process OMPI jobid: [24552,1] Process rank: 19
>>   Process OMPI jobid: [24552,1] Process rank: 20
>>   Process OMPI jobid: [24552,1] Process rank: 21
>>   Process OMPI jobid: [24552,1] Process rank: 22
>>   Process OMPI jobid: [24552,1] Process rank: 23
>> 
>> I have been going through the man page of mpirun as well as the OpenMPI 
>> mailing list and website, and thus far have been unable to determine the 
>> reason for the oversubscription of the head node (node0001) when even the 
>> PBS scheduler is passing along the correct slot count #s (16 and 8, resp).
>> 
>> Am I running into a bug w/ OpenMPI 1.6?
>> 
>> --john
>> 
>> 
>> 
>> -Original Message-
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>> Castain
>> Sent: Friday, June 06, 2014 1:30 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>> passes to OpenMPI
>> 
>> 
>> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
>> 
>>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>>> You can always add --display-allocation to the cmd line to see what 
>>>> we thought we received.
>>>> 
>>>> If you configure OMPI with --enable-debug, you can set --mca 
>>>> ras_base_verbose 10 to see the details
>>>> 
>>>> 
>>> 
>>> Hi John
>>> 
>>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>>> script.  This will list the nodes (multiple times according to the number 
>>> of cores requested).
>>> I find this useful documentation,
>>> along with job number, work directory, etc.
>>> "man qsub" will show you all the PBS_* environment variables 
>>> available to the job.
>>> For instance, you can echo them using a Torque 'prolog' script, if 
>>> the user d

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Sasso, John (GE Power & Water, Non-GE) 
Version 1.6 (i.e. prior to 1.6.1)

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 3:03 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

It's possible that you are hitting a bug - not sure how much the cpus-per-proc 
option has been exercised in 1.6. Is this 1.6.5, or some other member of that 
series?

I don't have a Torque machine handy any more, but should be able to test this 
scenario on my boxes


On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
<john1.sa...@ge.com> wrote:

> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
> --hostfile (i.e. the two are the same).  
> 
> To further debug this, I passed "--display-allocation --display-map" to 
> orterun, which resulted in:
> 
> ==   ALLOCATED NODES   ==
> 
> Data for node: node0001Num slots: 16   Max slots: 0
> Data for node: node0002  Num slots: 8Max slots: 0
> 
> =
> 
>    JOB MAP   
> 
> Data for node: node0001Num procs: 24
>Process OMPI jobid: [24552,1] Process rank: 0
>Process OMPI jobid: [24552,1] Process rank: 1
>Process OMPI jobid: [24552,1] Process rank: 2
>Process OMPI jobid: [24552,1] Process rank: 3
>Process OMPI jobid: [24552,1] Process rank: 4
>Process OMPI jobid: [24552,1] Process rank: 5
>Process OMPI jobid: [24552,1] Process rank: 6
>Process OMPI jobid: [24552,1] Process rank: 7
>Process OMPI jobid: [24552,1] Process rank: 8
>Process OMPI jobid: [24552,1] Process rank: 9
>Process OMPI jobid: [24552,1] Process rank: 10
>Process OMPI jobid: [24552,1] Process rank: 11
>Process OMPI jobid: [24552,1] Process rank: 12
>Process OMPI jobid: [24552,1] Process rank: 13
>Process OMPI jobid: [24552,1] Process rank: 14
>Process OMPI jobid: [24552,1] Process rank: 15
>Process OMPI jobid: [24552,1] Process rank: 16
>Process OMPI jobid: [24552,1] Process rank: 17
>Process OMPI jobid: [24552,1] Process rank: 18
>Process OMPI jobid: [24552,1] Process rank: 19
>Process OMPI jobid: [24552,1] Process rank: 20
>Process OMPI jobid: [24552,1] Process rank: 21
>Process OMPI jobid: [24552,1] Process rank: 22
>Process OMPI jobid: [24552,1] Process rank: 23
> 
> I have been going through the man page of mpirun as well as the OpenMPI 
> mailing list and website, and thus far have been unable to determine the 
> reason for the oversubscription of the head node (node0001) when even the PBS 
> scheduler is passing along the correct slot count #s (16 and 8, resp).
> 
> Am I running into a bug w/ OpenMPI 1.6?
> 
> --john
> 
> 
> 
> -----Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
> Castain
> Sent: Friday, June 06, 2014 1:30 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler 
> passes to OpenMPI
> 
> 
> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
> 
>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>> You can always add --display-allocation to the cmd line to see what 
>>> we thought we received.
>>> 
>>> If you configure OMPI with --enable-debug, you can set --mca 
>>> ras_base_verbose 10 to see the details
>>> 
>>> 
>> 
>> Hi John
>> 
>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>> script.  This will list the nodes (multiple times according to the number of 
>> cores requested).
>> I find this useful documentation,
>> along with job number, work directory, etc.
>> "man qsub" will show you all the PBS_* environment variables 
>> available to the job.
>> For instance, you can echo them using a Torque 'prolog' script, if 
>> the user didn't do it. That will appear in the Torque STDOUT file.
>> 
>> From outside the job script, "qstat -n" (and variants, say, with -u
>> username) will list the nodes allocated to each job, again multiple 
>> times as per the requested cores.
>> 
>> "tracejob job_number" will show similar information.
>> 
>> 
>> If you configured Torque --with-cpuset, there is more information 
>> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
>> (on the first node listed above, called "mo

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Ralph Castain 
It's possible that you are hitting a bug - not sure how much the cpus-per-proc 
option has been exercised in 1.6. Is this 1.6.5, or some other member of that 
series?

I don't have a Torque machine handy any more, but should be able to test this 
scenario on my boxes


On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
<john1.sa...@ge.com> wrote:

> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
> --hostfile (i.e. the two are the same).  
> 
> To further debug this, I passed "--display-allocation --display-map" to 
> orterun, which resulted in:
> 
> ==   ALLOCATED NODES   ==
> 
> Data for node: node0001Num slots: 16   Max slots: 0
> Data for node: node0002  Num slots: 8Max slots: 0
> 
> =
> 
>    JOB MAP   
> 
> Data for node: node0001Num procs: 24
>Process OMPI jobid: [24552,1] Process rank: 0
>Process OMPI jobid: [24552,1] Process rank: 1
>Process OMPI jobid: [24552,1] Process rank: 2
>Process OMPI jobid: [24552,1] Process rank: 3
>Process OMPI jobid: [24552,1] Process rank: 4
>Process OMPI jobid: [24552,1] Process rank: 5
>Process OMPI jobid: [24552,1] Process rank: 6
>Process OMPI jobid: [24552,1] Process rank: 7
>Process OMPI jobid: [24552,1] Process rank: 8
>Process OMPI jobid: [24552,1] Process rank: 9
>Process OMPI jobid: [24552,1] Process rank: 10
>Process OMPI jobid: [24552,1] Process rank: 11
>Process OMPI jobid: [24552,1] Process rank: 12
>Process OMPI jobid: [24552,1] Process rank: 13
>Process OMPI jobid: [24552,1] Process rank: 14
>Process OMPI jobid: [24552,1] Process rank: 15
>Process OMPI jobid: [24552,1] Process rank: 16
>Process OMPI jobid: [24552,1] Process rank: 17
>Process OMPI jobid: [24552,1] Process rank: 18
>Process OMPI jobid: [24552,1] Process rank: 19
>Process OMPI jobid: [24552,1] Process rank: 20
>Process OMPI jobid: [24552,1] Process rank: 21
>Process OMPI jobid: [24552,1] Process rank: 22
>Process OMPI jobid: [24552,1] Process rank: 23
> 
> I have been going through the man page of mpirun as well as the OpenMPI 
> mailing list and website, and thus far have been unable to determine the 
> reason for the oversubscription of the head node (node0001) when even the PBS 
> scheduler is passing along the correct slot count #s (16 and 8, resp).
> 
> Am I running into a bug w/ OpenMPI 1.6?
> 
> --john
> 
> 
> 
> -Original Message-----
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 1:30 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
> OpenMPI
> 
> 
> On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:
> 
>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>> You can always add --display-allocation to the cmd line to see what 
>>> we thought we received.
>>> 
>>> If you configure OMPI with --enable-debug, you can set --mca 
>>> ras_base_verbose 10 to see the details
>>> 
>>> 
>> 
>> Hi John
>> 
>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>> script.  This will list the nodes (multiple times according to the number of 
>> cores requested).
>> I find this useful documentation,
>> along with job number, work directory, etc.
>> "man qsub" will show you all the PBS_* environment variables available 
>> to the job.
>> For instance, you can echo them using a Torque 'prolog' script, if the 
>> user didn't do it. That will appear in the Torque STDOUT file.
>> 
>> From outside the job script, "qstat -n" (and variants, say, with -u 
>> username) will list the nodes allocated to each job, again multiple 
>> times as per the requested cores.
>> 
>> "tracejob job_number" will show similar information.
>> 
>> 
>> If you configured Torque --with-cpuset, there is more information 
>> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
>> (on the first node listed above, called "mother superior" in Torque 
>> parlance).
>> This mostly matter if there is more than one job running on a node.
>> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
>> whatever.  As Ralph said, Open MPI does that.
>> I

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Sasso, John (GE Power & Water, Non-GE) 
Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
--hostfile (i.e. the two are the same).  

To further debug this, I passed "--display-allocation --display-map" to 
orterun, which resulted in:

==   ALLOCATED NODES   ==

 Data for node: node0001Num slots: 16   Max slots: 0
 Data for node: node0002  Num slots: 8Max slots: 0

=

    JOB MAP   

 Data for node: node0001Num procs: 24
Process OMPI jobid: [24552,1] Process rank: 0
Process OMPI jobid: [24552,1] Process rank: 1
Process OMPI jobid: [24552,1] Process rank: 2
Process OMPI jobid: [24552,1] Process rank: 3
Process OMPI jobid: [24552,1] Process rank: 4
Process OMPI jobid: [24552,1] Process rank: 5
Process OMPI jobid: [24552,1] Process rank: 6
Process OMPI jobid: [24552,1] Process rank: 7
Process OMPI jobid: [24552,1] Process rank: 8
Process OMPI jobid: [24552,1] Process rank: 9
Process OMPI jobid: [24552,1] Process rank: 10
Process OMPI jobid: [24552,1] Process rank: 11
Process OMPI jobid: [24552,1] Process rank: 12
Process OMPI jobid: [24552,1] Process rank: 13
Process OMPI jobid: [24552,1] Process rank: 14
Process OMPI jobid: [24552,1] Process rank: 15
Process OMPI jobid: [24552,1] Process rank: 16
Process OMPI jobid: [24552,1] Process rank: 17
Process OMPI jobid: [24552,1] Process rank: 18
Process OMPI jobid: [24552,1] Process rank: 19
Process OMPI jobid: [24552,1] Process rank: 20
Process OMPI jobid: [24552,1] Process rank: 21
Process OMPI jobid: [24552,1] Process rank: 22
Process OMPI jobid: [24552,1] Process rank: 23

I have been going through the man page of mpirun as well as the OpenMPI mailing 
list and website, and thus far have been unable to determine the reason for the 
oversubscription of the head node (node0001) when even the PBS scheduler is 
passing along the correct slot count #s (16 and 8, resp).

Am I running into a bug w/ OpenMPI 1.6?

--john



-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 1:30 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI


On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:

> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>> You can always add --display-allocation to the cmd line to see what 
>> we thought we received.
>> 
>> If you configure OMPI with --enable-debug, you can set --mca 
>> ras_base_verbose 10 to see the details
>> 
>> 
> 
> Hi John
> 
> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job script. 
>  This will list the nodes (multiple times according to the number of cores 
> requested).
> I find this useful documentation,
> along with job number, work directory, etc.
> "man qsub" will show you all the PBS_* environment variables available 
> to the job.
> For instance, you can echo them using a Torque 'prolog' script, if the 
> user didn't do it. That will appear in the Torque STDOUT file.
> 
> From outside the job script, "qstat -n" (and variants, say, with -u 
> username) will list the nodes allocated to each job, again multiple 
> times as per the requested cores.
> 
> "tracejob job_number" will show similar information.
> 
> 
> If you configured Torque --with-cpuset, there is more information 
> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
> (on the first node listed above, called "mother superior" in Torque parlance).
> This mostly matter if there is more than one job running on a node.
> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
> whatever.  As Ralph said, Open MPI does that.
> I believe Open MPI doesn't use the cpuset info from Torque.
> (Ralph, please correct me if I am wrong.)

You are correct in that we don't use any per-process designations. We do, 
however, work inside any overall envelope that Torque may impose on us - e.g., 
if you tell Torque to limit the job to cores 0-4, we will honor that directive 
and keep all processes within that envelope.


> 
> My two cents,
> Gus Correa
> 
> 
>> On Jun 6, 2014, at 10:01 AM, Reuti <re...@staff.uni-marburg.de 
>> <mailto:re...@staff.uni-marburg.de>> wrote:
>> 
>>> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
>>> 
>>>> OK, so at the least, how can I get the node and slots/node info 
>>>> that is passed from PBS?

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Ralph Castain 

On Jun 6, 2014, at 10:24 AM, Gus Correa <g...@ldeo.columbia.edu> wrote:

> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>> You can always add --display-allocation to the cmd line to see what we
>> thought we received.
>> 
>> If you configure OMPI with --enable-debug, you can set --mca
>> ras_base_verbose 10 to see the details
>> 
>> 
> 
> Hi John
> 
> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job script. 
>  This will list the nodes (multiple times according to the number of cores 
> requested).
> I find this useful documentation,
> along with job number, work directory, etc.
> "man qsub" will show you all the PBS_* environment variables
> available to the job.
> For instance, you can echo them using a Torque
> 'prolog' script, if the user
> didn't do it. That will appear in the Torque STDOUT file.
> 
> From outside the job script, "qstat -n" (and variants, say, with -u username)
> will list the nodes allocated to each job,
> again multiple times as per the requested cores.
> 
> "tracejob job_number" will show similar information.
> 
> 
> If you configured Torque --with-cpuset,
> there is more information about the cpuset allocated to the job
> in /dev/cpuset/torque/jobnumber (on the first node listed above, called 
> "mother superior" in Torque parlance).
> This mostly matter if there is more than one job running on a node.
> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
> whatever.  As Ralph said, Open MPI does that.
> I believe Open MPI doesn't use the cpuset info from Torque.
> (Ralph, please correct me if I am wrong.)

You are correct in that we don't use any per-process designations. We do, 
however, work inside any overall envelope that Torque may impose on us - e.g., 
if you tell Torque to limit the job to cores 0-4, we will honor that directive 
and keep all processes within that envelope.


> 
> My two cents,
> Gus Correa
> 
> 
>> On Jun 6, 2014, at 10:01 AM, Reuti <re...@staff.uni-marburg.de
>> <mailto:re...@staff.uni-marburg.de>> wrote:
>> 
>>> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
>>> 
>>>> OK, so at the least, how can I get the node and slots/node info that
>>>> is passed from PBS?
>>>> 
>>>> I ask because I’m trying to troubleshoot a problem w/ PBS and the
>>>> build of OpenMPI 1.6 I noted.  If I submit a 24-process simple job
>>>> through PBS using a script which has:
>>>> 
>>>> /usr/local/openmpi/bin/orterun -n 24 --hostfile
>>>> /home/sasso/TEST/hosts.file --mca orte_rsh_agent rsh --mca btl
>>>> openib,tcp,self --mca orte_base_help_aggregate 0 -x PATH -x
>>>> LD_LIBRARY_PATH /home/sasso/TEST/simplempihello.exe
>>> 
>>> Using the --hostfile on your own would mean to violate the granted
>>> slot allocation by PBS. Just leave this option out. How do you submit
>>> your job?
>>> 
>>> -- Reuti
>>> 
>>> 
>>>> And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the
>>>> first 16 being host node0001 and the last 8 being node0002), it
>>>> appears that 24 MPI tasks try to start on node0001 instead of getting
>>>> distributed as 16 on node0001 and 8 on node0002.   Hence, I am
>>>> curious what is being passed by PBS.
>>>> 
>>>> --john
>>>> 
>>>> 
>>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph
>>>> Castain
>>>> Sent: Friday, June 06, 2014 12:31 PM
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] Determining what parameters a scheduler
>>>> passes to OpenMPI
>>>> 
>>>> We currently only get the node and slots/node info from PBS - we
>>>> don't get any task placement info at all. We then use the mpirun cmd
>>>> options and built-in mappers to map the tasks to the nodes.
>>>> 
>>>> I suppose we could do more integration in that regard, but haven't
>>>> really seen a reason to do so - the OMPI mappers are generally more
>>>> flexible than anything in the schedulers.
>>>> 
>>>> 
>>>> On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE)
>>>> <john1.sa...@ge.com <mailto:john1.sa...@ge.com>> wrote:
>>>> 
>>>> 
>>>> For the PBS scheduler and using a build of OpenMPI 1.6 built against
>>>> PBS include

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Gus Correa 

On 06/06/2014 01:05 PM, Ralph Castain wrote:

You can always add --display-allocation to the cmd line to see what we
thought we received.

If you configure OMPI with --enable-debug, you can set --mca
ras_base_verbose 10 to see the details




Hi John

On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
script.  This will list the nodes (multiple times according to the 
number of cores requested).

I find this useful documentation,
along with job number, work directory, etc.
"man qsub" will show you all the PBS_* environment variables
available to the job.
For instance, you can echo them using a Torque
'prolog' script, if the user
didn't do it. That will appear in the Torque STDOUT file.

From outside the job script, "qstat -n" (and variants, say, with -u 
username)

will list the nodes allocated to each job,
again multiple times as per the requested cores.

"tracejob job_number" will show similar information.


If you configured Torque --with-cpuset,
there is more information about the cpuset allocated to the job
in /dev/cpuset/torque/jobnumber (on the first node listed above, called 
"mother superior" in Torque parlance).

This mostly matter if there is more than one job running on a node.
However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
whatever.  As Ralph said, Open MPI does that.

I believe Open MPI doesn't use the cpuset info from Torque.
(Ralph, please correct me if I am wrong.)

My two cents,
Gus Correa



On Jun 6, 2014, at 10:01 AM, Reuti <re...@staff.uni-marburg.de
<mailto:re...@staff.uni-marburg.de>> wrote:


Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):


OK, so at the least, how can I get the node and slots/node info that
is passed from PBS?

I ask because I’m trying to troubleshoot a problem w/ PBS and the
build of OpenMPI 1.6 I noted.  If I submit a 24-process simple job
through PBS using a script which has:

/usr/local/openmpi/bin/orterun -n 24 --hostfile
/home/sasso/TEST/hosts.file --mca orte_rsh_agent rsh --mca btl
openib,tcp,self --mca orte_base_help_aggregate 0 -x PATH -x
LD_LIBRARY_PATH /home/sasso/TEST/simplempihello.exe


Using the --hostfile on your own would mean to violate the granted
slot allocation by PBS. Just leave this option out. How do you submit
your job?

-- Reuti



And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the
first 16 being host node0001 and the last 8 being node0002), it
appears that 24 MPI tasks try to start on node0001 instead of getting
distributed as 16 on node0001 and 8 on node0002.   Hence, I am
curious what is being passed by PBS.

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph
Castain
Sent: Friday, June 06, 2014 12:31 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler
passes to OpenMPI

We currently only get the node and slots/node info from PBS - we
don't get any task placement info at all. We then use the mpirun cmd
options and built-in mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't
really seen a reason to do so - the OMPI mappers are generally more
flexible than anything in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE)
<john1.sa...@ge.com <mailto:john1.sa...@ge.com>> wrote:


For the PBS scheduler and using a build of OpenMPI 1.6 built against
PBS include files + libs, is there a way to determine (perhaps via
some debugging flags passed to mpirun) what job placement parameters
are passed from the PBS scheduler to OpenMPI?  In particular, I am
talking about task placement info such as nodes to place on, etc.
  Thanks!

 --john
___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Ralph Castain 
You can always add --display-allocation to the cmd line to see what we thought 
we received.

If you configure OMPI with --enable-debug, you can set --mca ras_base_verbose 
10 to see the details


On Jun 6, 2014, at 10:01 AM, Reuti <re...@staff.uni-marburg.de> wrote:

> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
> 
>> OK, so at the least, how can I get the node and slots/node info that is 
>> passed from PBS?
>> 
>> I ask because I’m trying to troubleshoot a problem w/ PBS and the build of 
>> OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using 
>> a script which has:
>> 
>> /usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
>> --mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
>> orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
>> /home/sasso/TEST/simplempihello.exe
> 
> Using the --hostfile on your own would mean to violate the granted slot 
> allocation by PBS. Just leave this option out. How do you submit your job?
> 
> -- Reuti
> 
> 
>> And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 
>> 16 being host node0001 and the last 8 being node0002), it appears that 24 
>> MPI tasks try to start on node0001 instead of getting distributed as 16 on 
>> node0001 and 8 on node0002.   Hence, I am curious what is being passed by 
>> PBS.
>> 
>> --john
>> 
>> 
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
>> Sent: Friday, June 06, 2014 12:31 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
>> OpenMPI
>> 
>> We currently only get the node and slots/node info from PBS - we don't get 
>> any task placement info at all. We then use the mpirun cmd options and 
>> built-in mappers to map the tasks to the nodes.
>> 
>> I suppose we could do more integration in that regard, but haven't really 
>> seen a reason to do so - the OMPI mappers are generally more flexible than 
>> anything in the schedulers.
>> 
>> 
>> On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
>> <john1.sa...@ge.com> wrote:
>> 
>> 
>> For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
>> include files + libs, is there a way to determine (perhaps via some 
>> debugging flags passed to mpirun) what job placement parameters are passed 
>> from the PBS scheduler to OpenMPI?  In particular, I am talking about task 
>> placement info such as nodes to place on, etc.   Thanks!
>> 
>>  --john
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Reuti 
Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):

> OK, so at the least, how can I get the node and slots/node info that is 
> passed from PBS?
>  
> I ask because I’m trying to troubleshoot a problem w/ PBS and the build of 
> OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using a 
> script which has:
>  
> /usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
> --mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
> orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
> /home/sasso/TEST/simplempihello.exe

Using the --hostfile on your own would mean to violate the granted slot 
allocation by PBS. Just leave this option out. How do you submit your job?

-- Reuti


> And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 
> 16 being host node0001 and the last 8 being node0002), it appears that 24 MPI 
> tasks try to start on node0001 instead of getting distributed as 16 on 
> node0001 and 8 on node0002.   Hence, I am curious what is being passed by PBS.
>  
> --john
>  
>  
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 12:31 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
> OpenMPI
>  
> We currently only get the node and slots/node info from PBS - we don't get 
> any task placement info at all. We then use the mpirun cmd options and 
> built-in mappers to map the tasks to the nodes.
>  
> I suppose we could do more integration in that regard, but haven't really 
> seen a reason to do so - the OMPI mappers are generally more flexible than 
> anything in the schedulers.
>  
>  
> On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
> <john1.sa...@ge.com> wrote:
> 
> 
> For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
> include files + libs, is there a way to determine (perhaps via some debugging 
> flags passed to mpirun) what job placement parameters are passed from the PBS 
> scheduler to OpenMPI?  In particular, I am talking about task placement info 
> such as nodes to place on, etc.   Thanks!
>  
>   --john
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>  
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Sasso, John (GE Power & Water, Non-GE) 
OK, so at the least, how can I get the node and slots/node info that is passed 
from PBS?

I ask because I'm trying to troubleshoot a problem w/ PBS and the build of 
OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using a 
script which has:

/usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
--mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
/home/sasso/TEST/simplempihello.exe


And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 16 
being host node0001 and the last 8 being node0002), it appears that 24 MPI 
tasks try to start on node0001 instead of getting distributed as 16 on node0001 
and 8 on node0002.   Hence, I am curious what is being passed by PBS.

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 12:31 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

We currently only get the node and slots/node info from PBS - we don't get any 
task placement info at all. We then use the mpirun cmd options and built-in 
mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't really seen 
a reason to do so - the OMPI mappers are generally more flexible than anything 
in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
<john1.sa...@ge.com<mailto:john1.sa...@ge.com>> wrote:


For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
include files + libs, is there a way to determine (perhaps via some debugging 
flags passed to mpirun) what job placement parameters are passed from the PBS 
scheduler to OpenMPI?  In particular, I am talking about task placement info 
such as nodes to place on, etc.   Thanks!

  --john
___
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Ralph Castain 
We currently only get the node and slots/node info from PBS - we don't get any 
task placement info at all. We then use the mpirun cmd options and built-in 
mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't really seen 
a reason to do so - the OMPI mappers are generally more flexible than anything 
in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
 wrote:

> For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
> include files + libs, is there a way to determine (perhaps via some debugging 
> flags passed to mpirun) what job placement parameters are passed from the PBS 
> scheduler to OpenMPI?  In particular, I am talking about task placement info 
> such as nodes to place on, etc.   Thanks!
>  
>   --john
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Determining what parameters a scheduler passes to OpenMPI

2014-06-06 Thread Sasso, John (GE Power & Water, Non-GE) 
For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
include files + libs, is there a way to determine (perhaps via some debugging 
flags passed to mpirun) what job placement parameters are passed from the PBS 
scheduler to OpenMPI?  In particular, I am talking about task placement info 
such as nodes to place on, etc.   Thanks!

  --john