Re: [OMPI users] Bind multiple cores to rank - OpenMPI 1.8.1

[Ralph Castain]

HmmmTetsuya is quite correct. Afraid I got distracted by the segfault 
(still investigating that one). Our default policy for 2 processes is to map-by 
core, and that would indeed fail when cpus-per-proc > 1. However, that seems 
like a non-intuitive requirement, so let me see if I can make this be a little 
more user-friendly.


On Jun 6, 2014, at 2:25 PM, tmish...@jcity.maeda.co.jp wrote:

> 
> 
> Hi Dan,
> 
> Please try:
> mpirun -np 2 --map-by socket:pe=8 ./hello
> or
> mpirun -np 2 --map-by slot:pe=8 ./hello
> 
> You can not bind 8 cpus to the object "core" which has
> only one cpu. This limitation started from 1.8 series.
> 
> The objcet "socket" has 8 cores in your case. So you
> can do it. And, the object "slot" is almost same as the
> "core" but it can exceed the limitation, because it's a
> fictitious object which has no size.
> 
> Regards,
> Tetsuya Mishima
> 
> 
>> No problem -
>> 
>> These are model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz chips.
>> 2 per node, 8 cores each. No threading enabled.
>> 
>> $ lstopo
>> Machine (64GB)
>> NUMANode L#0 (P#0 32GB)
>> Socket L#0 + L3 L#0 (20MB)
>> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0
> (P#0)
>> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1
> (P#1)
>> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2
> (P#2)
>> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3
> (P#3)
>> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4
> (P#4)
>> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5
> (P#5)
>> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6
> (P#6)
>> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7
> (P#7)
>> HostBridge L#0
>> PCIBridge
>> PCI 1000:0087
>> Block L#0 "sda"
>> PCIBridge
>> PCI 8086:2250
>> PCIBridge
>> PCI 8086:1521
>> Net L#1 "eth0"
>> PCI 8086:1521
>> Net L#2 "eth1"
>> PCIBridge
>> PCI 102b:0533
>> PCI 8086:1d02
>> NUMANode L#1 (P#1 32GB)
>> Socket L#1 + L3 L#1 (20MB)
>> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8
> (P#8)
>> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9
> (P#9)
>> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
>> + PU L#10 (P#10)
>> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
>> + PU L#11 (P#11)
>> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
>> + PU L#12 (P#12)
>> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
>> + PU L#13 (P#13)
>> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
>> + PU L#14 (P#14)
>> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
>> + PU L#15 (P#15)
>> HostBridge L#5
>> PCIBridge
>> PCI 15b3:1011
>> Net L#3 "ib0"
>> OpenFabrics L#4 "mlx5_0"
>> PCIBridge
>> PCI 8086:2250
>> 
>> From the segfault below. I tried reproducing the crash on less than an
>> 4 node allocation but wasn't able to.
>> 
>> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ mpirun -np 2
>> -machinefile ./nodes -mca plm_base_verbose 10 ./hello
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> registering plm components
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> found loaded component isolated
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> component isolated has no register or open function
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> found loaded component slurm
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> component slurm register function successful
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> found loaded component rsh
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> component rsh register function successful
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> found loaded component tm
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>> component tm register function successful
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: opening
>> plm components
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>> loaded component isolated
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>> component isolated open function successful
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>> loaded component slurm
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>> component slurm open function successful
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>> loaded component rsh
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>> component rsh open function successful
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>> loaded component tm
>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>> component tm open function 

Re: [OMPI users] --preload-binary does not work

[Ralph Castain]

Looks like there is some strange interaction there,  but I doubt I'll get 
around to fixing it soon unless someone has a burning reason to not use tree 
spawn when preloading binaries. I'll mark it down as something to look at as 
time permits.


On Jun 6, 2014, at 4:28 PM, Ralph Castain  wrote:

> Don't know - I haven't seen someone use that option in awhile. Is there some 
> reason to do so?
> 
> On Jun 6, 2014, at 3:44 PM, E.O.  wrote:
> 
>> Thank you!
>> With the patch, --preload-binary option is working fine.
>> However, if I add "--gmca plm_rsh_no_tree_spawn 1" as a mpirun command line 
>> option, it hangs.
>> 
>> # /mpi/bin/mpirun --allow-run-as-root --gmca plm_rsh_no_tree_spawn 1 
>> --preload-binary --hostfile /root/.hosts --prefix /mpi --np 120 a.out
>> 
>> If I ran the command without --preload-binary, it works fine (have to copy 
>> the binary to each node beforehand of course). I guess this is a different 
>> issue?
>> 
>> Eiichi
>> 
>> 
>> eiichi
>> 
>> 
>> 
>> On Fri, Jun 6, 2014 at 5:35 PM, Ralph Castain  wrote:
>> Okay, I found the problem and think I have a fix that I posted (copied EO on 
>> it). You are welcome to download the patch and try it. Scheduled for release 
>> in 1.8.2
>> 
>> Thanks
>> Ralph
>> 
>> 
>> On Jun 6, 2014, at 1:01 PM, Ralph Castain  wrote:
>> 
>>> Yeah, it doesn't require ssh any more - but I haven't tested it in a bit, 
>>> and so it's possible something crept in there.
>>> 
>>> On Jun 6, 2014, at 12:27 PM, Reuti  wrote:
>>> 
 Am 06.06.2014 um 21:04 schrieb Ralph Castain:
 
> Supposed to, yes - but I don't know how much testing it has seen. I can 
> try to take a look
 
 Wasn't it on the list recently, that 1.8.1 should do it even without 
 passphraseless SSH between the nodes?
 
 -- Reuti
 
 
> On Jun 6, 2014, at 12:02 PM, E.O.  wrote:
> 
>> Hello
>> I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
>> One Redhat 6.2 and three busybox machine. They are all 64bit environment.
>> 
>> I want to use --preload-binary option to send the binary file to hosts 
>> but it's not working.
>> 
>> # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out 
>> --allow-run-as-root --np 4 --host box0101,box0103 --preload-binary 
>> ./a.out
>> --
>> mpirun was unable to launch the specified application as it could not 
>> access
>> or execute an executable:
>> 
>> Executable: ./a.out
>> Node: box0101
>> 
>> while attempting to start process rank 17.
>> --
>> 17 total processes failed to start
>> #
>> 
>> If I sent the binary by SCP beforehand, the command works fine. SCP is 
>> working fine without password between the hosts.
>> Is the option supposed to work?
>> Thank you,
>> 
>> Eiichi
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
 
 ___
 users mailing list
 us...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 

Re: [OMPI users] --preload-binary does not work

[Ralph Castain]

Don't know - I haven't seen someone use that option in awhile. Is there some 
reason to do so?

On Jun 6, 2014, at 3:44 PM, E.O.  wrote:

> Thank you!
> With the patch, --preload-binary option is working fine.
> However, if I add "--gmca plm_rsh_no_tree_spawn 1" as a mpirun command line 
> option, it hangs.
> 
> # /mpi/bin/mpirun --allow-run-as-root --gmca plm_rsh_no_tree_spawn 1 
> --preload-binary --hostfile /root/.hosts --prefix /mpi --np 120 a.out
> 
> If I ran the command without --preload-binary, it works fine (have to copy 
> the binary to each node beforehand of course). I guess this is a different 
> issue?
> 
> Eiichi
> 
> 
> eiichi
> 
> 
> 
> On Fri, Jun 6, 2014 at 5:35 PM, Ralph Castain  wrote:
> Okay, I found the problem and think I have a fix that I posted (copied EO on 
> it). You are welcome to download the patch and try it. Scheduled for release 
> in 1.8.2
> 
> Thanks
> Ralph
> 
> 
> On Jun 6, 2014, at 1:01 PM, Ralph Castain  wrote:
> 
>> Yeah, it doesn't require ssh any more - but I haven't tested it in a bit, 
>> and so it's possible something crept in there.
>> 
>> On Jun 6, 2014, at 12:27 PM, Reuti  wrote:
>> 
>>> Am 06.06.2014 um 21:04 schrieb Ralph Castain:
>>> 
 Supposed to, yes - but I don't know how much testing it has seen. I can 
 try to take a look
>>> 
>>> Wasn't it on the list recently, that 1.8.1 should do it even without 
>>> passphraseless SSH between the nodes?
>>> 
>>> -- Reuti
>>> 
>>> 
 On Jun 6, 2014, at 12:02 PM, E.O.  wrote:
 
> Hello
> I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
> One Redhat 6.2 and three busybox machine. They are all 64bit environment.
> 
> I want to use --preload-binary option to send the binary file to hosts 
> but it's not working.
> 
> # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out 
> --allow-run-as-root --np 4 --host box0101,box0103 --preload-binary ./a.out
> --
> mpirun was unable to launch the specified application as it could not 
> access
> or execute an executable:
> 
> Executable: ./a.out
> Node: box0101
> 
> while attempting to start process rank 17.
> --
> 17 total processes failed to start
> #
> 
> If I sent the binary by SCP beforehand, the command works fine. SCP is 
> working fine without password between the hosts.
> Is the option supposed to work?
> Thank you,
> 
> Eiichi
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
 
 ___
 users mailing list
 us...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] --preload-binary does not work

[E.O.]

Thank you!
With the patch, --preload-binary option is working fine.
However, if I add "--gmca plm_rsh_no_tree_spawn 1" as a mpirun command line
option, it hangs.

# /mpi/bin/mpirun --allow-run-as-root --gmca plm_rsh_no_tree_spawn 1
--preload-binary --hostfile /root/.hosts --prefix /mpi --np 120 a.out

If I ran the command without --preload-binary, it works fine (have to copy
the binary to each node beforehand of course). I guess this is a different
issue?

Eiichi


eiichi



On Fri, Jun 6, 2014 at 5:35 PM, Ralph Castain  wrote:

> Okay, I found the problem and think I have a fix that I posted (copied EO
> on it). You are welcome to download the patch and try it. Scheduled for
> release in 1.8.2
>
> Thanks
> Ralph
>
>
> On Jun 6, 2014, at 1:01 PM, Ralph Castain  wrote:
>
> Yeah, it doesn't require ssh any more - but I haven't tested it in a bit,
> and so it's possible something crept in there.
>
> On Jun 6, 2014, at 12:27 PM, Reuti  wrote:
>
> Am 06.06.2014 um 21:04 schrieb Ralph Castain:
>
> Supposed to, yes - but I don't know how much testing it has seen. I can
> try to take a look
>
>
> Wasn't it on the list recently, that 1.8.1 should do it even without
> passphraseless SSH between the nodes?
>
> -- Reuti
>
>
> On Jun 6, 2014, at 12:02 PM, E.O.  wrote:
>
> Hello
> I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
> One Redhat 6.2 and three busybox machine. They are all 64bit environment.
>
> I want to use --preload-binary option to send the binary file to hosts but
> it's not working.
>
> # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out
> --allow-run-as-root --np 4 --host box0101,box0103 --preload-binary ./a.out
> --
> mpirun was unable to launch the specified application as it could not
> access
> or execute an executable:
>
> Executable: ./a.out
> Node: box0101
>
> while attempting to start process rank 17.
> --
> 17 total processes failed to start
> #
>
> If I sent the binary by SCP beforehand, the command works fine. SCP is
> working fine without password between the hosts.
> Is the option supposed to work?
> Thank you,
>
> Eiichi
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] OPENIB unknown transport errors

[Tim Miller]

Hi Josh,

I asked one of our more advanced users to add the "-mca btl_openib_if_include
mlx4_0:1" argument to his job script. Unfortunately, the same error
occurred as before.

We'll keep digging on our end; if you have any other suggestions, please
let us know.

Tim


On Thu, Jun 5, 2014 at 7:32 PM, Tim Miller  wrote:

> Hi Josh,
>
> Thanks for attempting to sort this out. In answer to your questions:
>
> 1. Node allocation is done by TORQUE, however we don't use the TM API to
> launch jobs (long story). Instead, we just pass a hostfile to mpirun, and
> mpirun uses the ssh launcher to actually communicate and launch the
> processes on remote nodes.
> 2. We have only one port per HCA (the HCA silicon is integrated with the
> motherboard on most of our nodes, including all that have this issue). They
> are all configured to use InfiniBand (no IPoIB or other protocols).
> 3. No, we don't explicitly ask for a device port pair. We will try your
> suggestion and report back.
>
> Thanks again!
>
> Tim
>
>
> On Thu, Jun 5, 2014 at 2:22 PM, Joshua Ladd  wrote:
>
>> Strange indeed. This info (remote adapter info) is passed around in the
>> modex and the struct is locally populated during add procs.
>>
>> 1. How do you launch jobs? Mpirun, srun, or something else?
>> 2. How many active ports do you have on each HCA? Are they all configured
>> to use IB?
>> 3. Do you explicitly ask for a device:port pair with the "if include" mca
>> param? If not, can you please add "-mca btl_openib_if_include mlx4_0:1"
>> (assuming you have a ConnectX-3 HCA and port 1 is configured to run over
>> IB.)
>>
>> Josh
>>
>>
>> On Wed, Jun 4, 2014 at 12:47 PM, Tim Miller  wrote:
>>
>>> Hi,
>>>
>>> I'd like to revive this thread, since I am still periodically getting
>>> errors of this type. I have built 1.8.1 with --enable-debug and run with
>>> -mca btl_openib_verbose 10. Unfortunately, this doesn't seem to provide any
>>> additional information that I can find useful. I've gone ahead and attached
>>> a dump of the output under 1.8.1. The key lines are:
>>>
>>>
>>> --
>>> Open MPI detected two different OpenFabrics transport types in the same
>>> Infiniband network.
>>> Such mixed network trasport configuration is not supported by Open MPI.
>>>
>>>   Local host:w1
>>>   Local adapter: mlx4_0 (vendor 0x2c9, part ID 26428)
>>>   Local transport type:  MCA_BTL_OPENIB_TRANSPORT_IB
>>>
>>>   Remote host:   w16
>>>   Remote Adapter:(vendor 0x2c9, part ID 26428)
>>>   Remote transport type: MCA_BTL_OPENIB_TRANSPORT_UNKNOWN
>>> -
>>>
>>> Note that the vendor and part IDs are the same. If I immediately run on
>>> the same two nodes using MVAPICH2, everything is fine.
>>>
>>> I'm really very befuddled by this. OpenMPI sees that the two cards are
>>> the same and made by the same vendor, yet it thinks the transport types are
>>> different (and one is unknown). I'm hoping someone with some experience
>>> with how the OpenIB BTL works can shed some light on this problem...
>>>
>>> Tim
>>>
>>>
>>> On Fri, May 9, 2014 at 7:39 PM, Joshua Ladd 
>>> wrote:
>>>

 Just wondering if you've tried with the latest stable OMPI, 1.8.1? I'm
 wondering if this is an issue with the OOB. If you have a debug build, you
 can run -mca btl_openib_verbose 10

 Josh


 On Fri, May 9, 2014 at 6:26 PM, Joshua Ladd 
 wrote:

> Hi, Tim
>
> Run "ibstat" on each host:
>
> 1. Make sure the adapters are alive and active.
>
> 2. Look at the Link Layer settings for host w34. Does it match host
> w4's?
>
>
> Josh
>
>
> On Fri, May 9, 2014 at 1:18 PM, Tim Miller 
> wrote:
>
>> Hi All,
>>
>> We're using OpenMPI 1.7.3 with Mellanox ConnectX InfiniBand adapters,
>> and periodically our jobs abort at start-up with the following error:
>>
>> ===
>> Open MPI detected two different OpenFabrics transport types in the
>> same Infiniband network.
>> Such mixed network trasport configuration is not supported by Open
>> MPI.
>>
>>   Local host:w4
>>   Local adapter: mlx4_0 (vendor 0x2c9, part ID 26428)
>>   Local transport type:  MCA_BTL_OPENIB_TRANSPORT_IB
>>
>>   Remote host:   w34
>>   Remote Adapter:(vendor 0x2c9, part ID 26428)
>>   Remote transport type: MCA_BTL_OPENIB_TRANSPORT_UNKNOWN
>> ===
>>
>> I've done a bit of googling and not found very much. We do not see
>> this issue when we run with MVAPICH2 on the same sets of nodes.
>>
>> Any advice or thoughts would be very welcome, as I am stumped by what
>> causes this. The nodes are all running 

Re: [OMPI users] --preload-binary does not work

[Ralph Castain]

Okay, I found the problem and think I have a fix that I posted (copied EO on 
it). You are welcome to download the patch and try it. Scheduled for release in 
1.8.2

Thanks
Ralph


On Jun 6, 2014, at 1:01 PM, Ralph Castain  wrote:

> Yeah, it doesn't require ssh any more - but I haven't tested it in a bit, and 
> so it's possible something crept in there.
> 
> On Jun 6, 2014, at 12:27 PM, Reuti  wrote:
> 
>> Am 06.06.2014 um 21:04 schrieb Ralph Castain:
>> 
>>> Supposed to, yes - but I don't know how much testing it has seen. I can try 
>>> to take a look
>> 
>> Wasn't it on the list recently, that 1.8.1 should do it even without 
>> passphraseless SSH between the nodes?
>> 
>> -- Reuti
>> 
>> 
>>> On Jun 6, 2014, at 12:02 PM, E.O.  wrote:
>>> 
 Hello
 I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
 One Redhat 6.2 and three busybox machine. They are all 64bit environment.
 
 I want to use --preload-binary option to send the binary file to hosts but 
 it's not working.
 
 # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out 
 --allow-run-as-root --np 4 --host box0101,box0103 --preload-binary ./a.out
 --
 mpirun was unable to launch the specified application as it could not 
 access
 or execute an executable:
 
 Executable: ./a.out
 Node: box0101
 
 while attempting to start process rank 17.
 --
 17 total processes failed to start
 #
 
 If I sent the binary by SCP beforehand, the command works fine. SCP is 
 working fine without password between the hosts.
 Is the option supposed to work?
 Thank you,
 
 Eiichi
 
 ___
 users mailing list
 us...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 

Re: [OMPI users] Bind multiple cores to rank - OpenMPI 1.8.1

[tmishima]

Hi Dan,

Please try:
mpirun -np 2 --map-by socket:pe=8 ./hello
or
mpirun -np 2 --map-by slot:pe=8 ./hello

You can not bind 8 cpus to the object "core" which has
only one cpu. This limitation started from 1.8 series.

The objcet "socket" has 8 cores in your case. So you
can do it. And, the object "slot" is almost same as the
"core" but it can exceed the limitation, because it's a
fictitious object which has no size.

Regards,
Tetsuya Mishima


> No problem -
>
> These are model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz chips.
> 2 per node, 8 cores each. No threading enabled.
>
> $ lstopo
> Machine (64GB)
> NUMANode L#0 (P#0 32GB)
> Socket L#0 + L3 L#0 (20MB)
> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0
(P#0)
> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1
(P#1)
> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2
(P#2)
> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3
(P#3)
> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4
(P#4)
> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5
(P#5)
> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6
(P#6)
> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7
(P#7)
> HostBridge L#0
> PCIBridge
> PCI 1000:0087
> Block L#0 "sda"
> PCIBridge
> PCI 8086:2250
> PCIBridge
> PCI 8086:1521
> Net L#1 "eth0"
> PCI 8086:1521
> Net L#2 "eth1"
> PCIBridge
> PCI 102b:0533
> PCI 8086:1d02
> NUMANode L#1 (P#1 32GB)
> Socket L#1 + L3 L#1 (20MB)
> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8
(P#8)
> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9
(P#9)
> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
> + PU L#10 (P#10)
> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
> + PU L#11 (P#11)
> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
> + PU L#12 (P#12)
> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
> + PU L#13 (P#13)
> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
> + PU L#14 (P#14)
> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
> + PU L#15 (P#15)
> HostBridge L#5
> PCIBridge
> PCI 15b3:1011
> Net L#3 "ib0"
> OpenFabrics L#4 "mlx5_0"
> PCIBridge
> PCI 8086:2250
>
> From the segfault below. I tried reproducing the crash on less than an
> 4 node allocation but wasn't able to.
>
> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ mpirun -np 2
> -machinefile ./nodes -mca plm_base_verbose 10 ./hello
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> registering plm components
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component isolated
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component isolated has no register or open function
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component slurm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component slurm register function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component rsh
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component rsh register function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component tm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component tm register function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: opening
> plm components
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component isolated
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component isolated open function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component slurm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component slurm open function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component rsh
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component rsh open function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component tm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component tm open function successful
> [conte-a009.rcac.purdue.edu:55685] mca:base:select: Auto-selecting plm
> components
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
> component [isolated]
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Query of
> component [isolated] set priority to 0
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
> component [slurm]
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Skipping
> component [slurm]. Query failed to return a module
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  

Re: [OMPI users] --preload-binary does not work

[Ralph Castain]

Yeah, it doesn't require ssh any more - but I haven't tested it in a bit, and 
so it's possible something crept in there.

On Jun 6, 2014, at 12:27 PM, Reuti  wrote:

> Am 06.06.2014 um 21:04 schrieb Ralph Castain:
> 
>> Supposed to, yes - but I don't know how much testing it has seen. I can try 
>> to take a look
> 
> Wasn't it on the list recently, that 1.8.1 should do it even without 
> passphraseless SSH between the nodes?
> 
> -- Reuti
> 
> 
>> On Jun 6, 2014, at 12:02 PM, E.O.  wrote:
>> 
>>> Hello
>>> I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
>>> One Redhat 6.2 and three busybox machine. They are all 64bit environment.
>>> 
>>> I want to use --preload-binary option to send the binary file to hosts but 
>>> it's not working.
>>> 
>>> # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out --allow-run-as-root 
>>> --np 4 --host box0101,box0103 --preload-binary ./a.out
>>> --
>>> mpirun was unable to launch the specified application as it could not access
>>> or execute an executable:
>>> 
>>> Executable: ./a.out
>>> Node: box0101
>>> 
>>> while attempting to start process rank 17.
>>> --
>>> 17 total processes failed to start
>>> #
>>> 
>>> If I sent the binary by SCP beforehand, the command works fine. SCP is 
>>> working fine without password between the hosts.
>>> Is the option supposed to work?
>>> Thank you,
>>> 
>>> Eiichi
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Bind multiple cores to rank - OpenMPI 1.8.1

[Ralph Castain]

Okay, I'll poke into this - thanks!

On Jun 6, 2014, at 12:48 PM, Dan Dietz  wrote:

> No problem -
> 
> These are model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz chips.
> 2 per node, 8 cores each. No threading enabled.
> 
> $ lstopo
> Machine (64GB)
>  NUMANode L#0 (P#0 32GB)
>Socket L#0 + L3 L#0 (20MB)
>  L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 
> (P#0)
>  L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 
> (P#1)
>  L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 
> (P#2)
>  L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 
> (P#3)
>  L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 
> (P#4)
>  L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 
> (P#5)
>  L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 
> (P#6)
>  L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 
> (P#7)
>HostBridge L#0
>  PCIBridge
>PCI 1000:0087
>  Block L#0 "sda"
>  PCIBridge
>PCI 8086:2250
>  PCIBridge
>PCI 8086:1521
>  Net L#1 "eth0"
>PCI 8086:1521
>  Net L#2 "eth1"
>  PCIBridge
>PCI 102b:0533
>  PCI 8086:1d02
>  NUMANode L#1 (P#1 32GB)
>Socket L#1 + L3 L#1 (20MB)
>  L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 
> (P#8)
>  L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 
> (P#9)
>  L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
> + PU L#10 (P#10)
>  L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
> + PU L#11 (P#11)
>  L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
> + PU L#12 (P#12)
>  L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
> + PU L#13 (P#13)
>  L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
> + PU L#14 (P#14)
>  L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
> + PU L#15 (P#15)
>HostBridge L#5
>  PCIBridge
>PCI 15b3:1011
>  Net L#3 "ib0"
>  OpenFabrics L#4 "mlx5_0"
>  PCIBridge
>PCI 8086:2250
> 
> From the segfault below. I tried reproducing the crash on less than an
> 4 node allocation but wasn't able to.
> 
> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ mpirun -np 2
> -machinefile ./nodes -mca plm_base_verbose 10 ./hello
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> registering plm components
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component isolated
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component isolated has no register or open function
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component slurm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component slurm register function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component rsh
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component rsh register function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> found loaded component tm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
> component tm register function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: opening
> plm components
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component isolated
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component isolated open function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component slurm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component slurm open function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component rsh
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component rsh open function successful
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
> loaded component tm
> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
> component tm open function successful
> [conte-a009.rcac.purdue.edu:55685] mca:base:select: Auto-selecting plm
> components
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
> component [isolated]
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Query of
> component [isolated] set priority to 0
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
> component [slurm]
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Skipping
> component [slurm]. Query failed to return a module
> [conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
> component [rsh]
> [conte-a009.rcac.purdue.edu:55685] [[INVALID],INVALID] 

Re: [OMPI users] Bind multiple cores to rank - OpenMPI 1.8.1

[Dan Dietz]

No problem -

These are model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz chips.
2 per node, 8 cores each. No threading enabled.

$ lstopo
Machine (64GB)
  NUMANode L#0 (P#0 32GB)
Socket L#0 + L3 L#0 (20MB)
  L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
  L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#1)
  L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#2)
  L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#3)
  L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#4)
  L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#5)
  L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#6)
  L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#7)
HostBridge L#0
  PCIBridge
PCI 1000:0087
  Block L#0 "sda"
  PCIBridge
PCI 8086:2250
  PCIBridge
PCI 8086:1521
  Net L#1 "eth0"
PCI 8086:1521
  Net L#2 "eth1"
  PCIBridge
PCI 102b:0533
  PCI 8086:1d02
  NUMANode L#1 (P#1 32GB)
Socket L#1 + L3 L#1 (20MB)
  L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#8)
  L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#9)
  L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
+ PU L#10 (P#10)
  L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
+ PU L#11 (P#11)
  L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
+ PU L#12 (P#12)
  L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
+ PU L#13 (P#13)
  L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
+ PU L#14 (P#14)
  L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
+ PU L#15 (P#15)
HostBridge L#5
  PCIBridge
PCI 15b3:1011
  Net L#3 "ib0"
  OpenFabrics L#4 "mlx5_0"
  PCIBridge
PCI 8086:2250

>From the segfault below. I tried reproducing the crash on less than an
4 node allocation but wasn't able to.

ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ mpirun -np 2
-machinefile ./nodes -mca plm_base_verbose 10 ./hello
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
registering plm components
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
found loaded component isolated
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
component isolated has no register or open function
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
found loaded component slurm
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
component slurm register function successful
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
found loaded component rsh
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
component rsh register function successful
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
found loaded component tm
[conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
component tm register function successful
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open: opening
plm components
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
loaded component isolated
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
component isolated open function successful
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
loaded component slurm
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
component slurm open function successful
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
loaded component rsh
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
component rsh open function successful
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
loaded component tm
[conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
component tm open function successful
[conte-a009.rcac.purdue.edu:55685] mca:base:select: Auto-selecting plm
components
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
component [isolated]
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Query of
component [isolated] set priority to 0
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
component [slurm]
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Skipping
component [slurm]. Query failed to return a module
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
component [rsh]
[conte-a009.rcac.purdue.edu:55685] [[INVALID],INVALID] plm:rsh_lookup
on agent ssh : rsh path NULL
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Query of
component [rsh] set priority to 10
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Querying
component [tm]
[conte-a009.rcac.purdue.edu:55685] mca:base:select:(  plm) Query of
component 

Re: [OMPI users] --preload-binary does not work

[Reuti]

Am 06.06.2014 um 21:04 schrieb Ralph Castain:

> Supposed to, yes - but I don't know how much testing it has seen. I can try 
> to take a look

Wasn't it on the list recently, that 1.8.1 should do it even without 
passphraseless SSH between the nodes?

-- Reuti


> On Jun 6, 2014, at 12:02 PM, E.O.  wrote:
> 
>> Hello
>> I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
>> One Redhat 6.2 and three busybox machine. They are all 64bit environment.
>> 
>> I want to use --preload-binary option to send the binary file to hosts but 
>> it's not working.
>> 
>> # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out --allow-run-as-root 
>> --np 4 --host box0101,box0103 --preload-binary ./a.out
>> --
>> mpirun was unable to launch the specified application as it could not access
>> or execute an executable:
>> 
>> Executable: ./a.out
>> Node: box0101
>> 
>> while attempting to start process rank 17.
>> --
>> 17 total processes failed to start
>> #
>> 
>> If I sent the binary by SCP beforehand, the command works fine. SCP is 
>> working fine without password between the hosts.
>> Is the option supposed to work?
>> Thank you,
>> 
>> Eiichi
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Ralph Castain]

You might want to update to 1.6.5, if you can - I'll see what I can find

On Jun 6, 2014, at 12:07 PM, Sasso, John (GE Power & Water, Non-GE) 
 wrote:

> Version 1.6 (i.e. prior to 1.6.1)
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 3:03 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
> OpenMPI
> 
> It's possible that you are hitting a bug - not sure how much the 
> cpus-per-proc option has been exercised in 1.6. Is this 1.6.5, or some other 
> member of that series?
> 
> I don't have a Torque machine handy any more, but should be able to test this 
> scenario on my boxes
> 
> 
> On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
>  wrote:
> 
>> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
>> --hostfile (i.e. the two are the same).  
>> 
>> To further debug this, I passed "--display-allocation --display-map" to 
>> orterun, which resulted in:
>> 
>> ==   ALLOCATED NODES   ==
>> 
>> Data for node: node0001Num slots: 16   Max slots: 0
>> Data for node: node0002  Num slots: 8Max slots: 0
>> 
>> =
>> 
>>    JOB MAP   
>> 
>> Data for node: node0001Num procs: 24
>>   Process OMPI jobid: [24552,1] Process rank: 0
>>   Process OMPI jobid: [24552,1] Process rank: 1
>>   Process OMPI jobid: [24552,1] Process rank: 2
>>   Process OMPI jobid: [24552,1] Process rank: 3
>>   Process OMPI jobid: [24552,1] Process rank: 4
>>   Process OMPI jobid: [24552,1] Process rank: 5
>>   Process OMPI jobid: [24552,1] Process rank: 6
>>   Process OMPI jobid: [24552,1] Process rank: 7
>>   Process OMPI jobid: [24552,1] Process rank: 8
>>   Process OMPI jobid: [24552,1] Process rank: 9
>>   Process OMPI jobid: [24552,1] Process rank: 10
>>   Process OMPI jobid: [24552,1] Process rank: 11
>>   Process OMPI jobid: [24552,1] Process rank: 12
>>   Process OMPI jobid: [24552,1] Process rank: 13
>>   Process OMPI jobid: [24552,1] Process rank: 14
>>   Process OMPI jobid: [24552,1] Process rank: 15
>>   Process OMPI jobid: [24552,1] Process rank: 16
>>   Process OMPI jobid: [24552,1] Process rank: 17
>>   Process OMPI jobid: [24552,1] Process rank: 18
>>   Process OMPI jobid: [24552,1] Process rank: 19
>>   Process OMPI jobid: [24552,1] Process rank: 20
>>   Process OMPI jobid: [24552,1] Process rank: 21
>>   Process OMPI jobid: [24552,1] Process rank: 22
>>   Process OMPI jobid: [24552,1] Process rank: 23
>> 
>> I have been going through the man page of mpirun as well as the OpenMPI 
>> mailing list and website, and thus far have been unable to determine the 
>> reason for the oversubscription of the head node (node0001) when even the 
>> PBS scheduler is passing along the correct slot count #s (16 and 8, resp).
>> 
>> Am I running into a bug w/ OpenMPI 1.6?
>> 
>> --john
>> 
>> 
>> 
>> -Original Message-
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
>> Castain
>> Sent: Friday, June 06, 2014 1:30 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler 
>> passes to OpenMPI
>> 
>> 
>> On Jun 6, 2014, at 10:24 AM, Gus Correa  wrote:
>> 
>>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
 You can always add --display-allocation to the cmd line to see what 
 we thought we received.
 
 If you configure OMPI with --enable-debug, you can set --mca 
 ras_base_verbose 10 to see the details
 
 
>>> 
>>> Hi John
>>> 
>>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>>> script.  This will list the nodes (multiple times according to the number 
>>> of cores requested).
>>> I find this useful documentation,
>>> along with job number, work directory, etc.
>>> "man qsub" will show you all the PBS_* environment variables 
>>> available to the job.
>>> For instance, you can echo them using a Torque 'prolog' script, if 
>>> the user didn't do it. That will appear in the Torque STDOUT file.
>>> 
>>> From outside the job script, "qstat -n" (and variants, say, with -u
>>> username) will list the nodes allocated to each job, again multiple 
>>> times as per the requested cores.
>>> 
>>> "tracejob job_number" will show similar information.
>>> 
>>> 
>>> If you configured Torque --with-cpuset, there is more information 
>>> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
>>> (on the first node listed above, called "mother superior" in Torque 
>>> parlance).
>>> This mostly matter if there is more than one job running on a node.
>>> However, Torque doesn't bind processes/MPI_ranks to cores or sockets 

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

Version 1.6 (i.e. prior to 1.6.1)

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 3:03 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

It's possible that you are hitting a bug - not sure how much the cpus-per-proc 
option has been exercised in 1.6. Is this 1.6.5, or some other member of that 
series?

I don't have a Torque machine handy any more, but should be able to test this 
scenario on my boxes


On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
 wrote:

> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
> --hostfile (i.e. the two are the same).  
> 
> To further debug this, I passed "--display-allocation --display-map" to 
> orterun, which resulted in:
> 
> ==   ALLOCATED NODES   ==
> 
> Data for node: node0001Num slots: 16   Max slots: 0
> Data for node: node0002  Num slots: 8Max slots: 0
> 
> =
> 
>    JOB MAP   
> 
> Data for node: node0001Num procs: 24
>Process OMPI jobid: [24552,1] Process rank: 0
>Process OMPI jobid: [24552,1] Process rank: 1
>Process OMPI jobid: [24552,1] Process rank: 2
>Process OMPI jobid: [24552,1] Process rank: 3
>Process OMPI jobid: [24552,1] Process rank: 4
>Process OMPI jobid: [24552,1] Process rank: 5
>Process OMPI jobid: [24552,1] Process rank: 6
>Process OMPI jobid: [24552,1] Process rank: 7
>Process OMPI jobid: [24552,1] Process rank: 8
>Process OMPI jobid: [24552,1] Process rank: 9
>Process OMPI jobid: [24552,1] Process rank: 10
>Process OMPI jobid: [24552,1] Process rank: 11
>Process OMPI jobid: [24552,1] Process rank: 12
>Process OMPI jobid: [24552,1] Process rank: 13
>Process OMPI jobid: [24552,1] Process rank: 14
>Process OMPI jobid: [24552,1] Process rank: 15
>Process OMPI jobid: [24552,1] Process rank: 16
>Process OMPI jobid: [24552,1] Process rank: 17
>Process OMPI jobid: [24552,1] Process rank: 18
>Process OMPI jobid: [24552,1] Process rank: 19
>Process OMPI jobid: [24552,1] Process rank: 20
>Process OMPI jobid: [24552,1] Process rank: 21
>Process OMPI jobid: [24552,1] Process rank: 22
>Process OMPI jobid: [24552,1] Process rank: 23
> 
> I have been going through the man page of mpirun as well as the OpenMPI 
> mailing list and website, and thus far have been unable to determine the 
> reason for the oversubscription of the head node (node0001) when even the PBS 
> scheduler is passing along the correct slot count #s (16 and 8, resp).
> 
> Am I running into a bug w/ OpenMPI 1.6?
> 
> --john
> 
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph 
> Castain
> Sent: Friday, June 06, 2014 1:30 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler 
> passes to OpenMPI
> 
> 
> On Jun 6, 2014, at 10:24 AM, Gus Correa  wrote:
> 
>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>> You can always add --display-allocation to the cmd line to see what 
>>> we thought we received.
>>> 
>>> If you configure OMPI with --enable-debug, you can set --mca 
>>> ras_base_verbose 10 to see the details
>>> 
>>> 
>> 
>> Hi John
>> 
>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>> script.  This will list the nodes (multiple times according to the number of 
>> cores requested).
>> I find this useful documentation,
>> along with job number, work directory, etc.
>> "man qsub" will show you all the PBS_* environment variables 
>> available to the job.
>> For instance, you can echo them using a Torque 'prolog' script, if 
>> the user didn't do it. That will appear in the Torque STDOUT file.
>> 
>> From outside the job script, "qstat -n" (and variants, say, with -u
>> username) will list the nodes allocated to each job, again multiple 
>> times as per the requested cores.
>> 
>> "tracejob job_number" will show similar information.
>> 
>> 
>> If you configured Torque --with-cpuset, there is more information 
>> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
>> (on the first node listed above, called "mother superior" in Torque 
>> parlance).
>> This mostly matter if there is more than one job running on a node.
>> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
>> whatever.  As Ralph said, Open MPI does that.
>> I believe Open MPI doesn't use the cpuset info from Torque.
>> (Ralph, please correct me if I am wrong.)
> 
> You are correct in that we don't use any per-process designations. We do, 
> however, work inside any overall envelope 

Re: [OMPI users] --preload-binary does not work

[Ralph Castain]

Supposed to, yes - but I don't know how much testing it has seen. I can try to 
take a look

On Jun 6, 2014, at 12:02 PM, E.O.  wrote:

> Hello
> I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
> One Redhat 6.2 and three busybox machine. They are all 64bit environment.
> 
> I want to use --preload-binary option to send the binary file to hosts but 
> it's not working.
> 
> # /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out --allow-run-as-root 
> --np 4 --host box0101,box0103 --preload-binary ./a.out
> --
> mpirun was unable to launch the specified application as it could not access
> or execute an executable:
> 
> Executable: ./a.out
> Node: box0101
> 
> while attempting to start process rank 17.
> --
> 17 total processes failed to start
> #
> 
> If I sent the binary by SCP beforehand, the command works fine. SCP is 
> working fine without password between the hosts.
> Is the option supposed to work?
> Thank you,
> 
> Eiichi
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Ralph Castain]

It's possible that you are hitting a bug - not sure how much the cpus-per-proc 
option has been exercised in 1.6. Is this 1.6.5, or some other member of that 
series?

I don't have a Torque machine handy any more, but should be able to test this 
scenario on my boxes


On Jun 6, 2014, at 10:51 AM, Sasso, John (GE Power & Water, Non-GE) 
 wrote:

> Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
> --hostfile (i.e. the two are the same).  
> 
> To further debug this, I passed "--display-allocation --display-map" to 
> orterun, which resulted in:
> 
> ==   ALLOCATED NODES   ==
> 
> Data for node: node0001Num slots: 16   Max slots: 0
> Data for node: node0002  Num slots: 8Max slots: 0
> 
> =
> 
>    JOB MAP   
> 
> Data for node: node0001Num procs: 24
>Process OMPI jobid: [24552,1] Process rank: 0
>Process OMPI jobid: [24552,1] Process rank: 1
>Process OMPI jobid: [24552,1] Process rank: 2
>Process OMPI jobid: [24552,1] Process rank: 3
>Process OMPI jobid: [24552,1] Process rank: 4
>Process OMPI jobid: [24552,1] Process rank: 5
>Process OMPI jobid: [24552,1] Process rank: 6
>Process OMPI jobid: [24552,1] Process rank: 7
>Process OMPI jobid: [24552,1] Process rank: 8
>Process OMPI jobid: [24552,1] Process rank: 9
>Process OMPI jobid: [24552,1] Process rank: 10
>Process OMPI jobid: [24552,1] Process rank: 11
>Process OMPI jobid: [24552,1] Process rank: 12
>Process OMPI jobid: [24552,1] Process rank: 13
>Process OMPI jobid: [24552,1] Process rank: 14
>Process OMPI jobid: [24552,1] Process rank: 15
>Process OMPI jobid: [24552,1] Process rank: 16
>Process OMPI jobid: [24552,1] Process rank: 17
>Process OMPI jobid: [24552,1] Process rank: 18
>Process OMPI jobid: [24552,1] Process rank: 19
>Process OMPI jobid: [24552,1] Process rank: 20
>Process OMPI jobid: [24552,1] Process rank: 21
>Process OMPI jobid: [24552,1] Process rank: 22
>Process OMPI jobid: [24552,1] Process rank: 23
> 
> I have been going through the man page of mpirun as well as the OpenMPI 
> mailing list and website, and thus far have been unable to determine the 
> reason for the oversubscription of the head node (node0001) when even the PBS 
> scheduler is passing along the correct slot count #s (16 and 8, resp).
> 
> Am I running into a bug w/ OpenMPI 1.6?
> 
> --john
> 
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 1:30 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
> OpenMPI
> 
> 
> On Jun 6, 2014, at 10:24 AM, Gus Correa  wrote:
> 
>> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>>> You can always add --display-allocation to the cmd line to see what 
>>> we thought we received.
>>> 
>>> If you configure OMPI with --enable-debug, you can set --mca 
>>> ras_base_verbose 10 to see the details
>>> 
>>> 
>> 
>> Hi John
>> 
>> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
>> script.  This will list the nodes (multiple times according to the number of 
>> cores requested).
>> I find this useful documentation,
>> along with job number, work directory, etc.
>> "man qsub" will show you all the PBS_* environment variables available 
>> to the job.
>> For instance, you can echo them using a Torque 'prolog' script, if the 
>> user didn't do it. That will appear in the Torque STDOUT file.
>> 
>> From outside the job script, "qstat -n" (and variants, say, with -u 
>> username) will list the nodes allocated to each job, again multiple 
>> times as per the requested cores.
>> 
>> "tracejob job_number" will show similar information.
>> 
>> 
>> If you configured Torque --with-cpuset, there is more information 
>> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
>> (on the first node listed above, called "mother superior" in Torque 
>> parlance).
>> This mostly matter if there is more than one job running on a node.
>> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
>> whatever.  As Ralph said, Open MPI does that.
>> I believe Open MPI doesn't use the cpuset info from Torque.
>> (Ralph, please correct me if I am wrong.)
> 
> You are correct in that we don't use any per-process designations. We do, 
> however, work inside any overall envelope that Torque may impose on us - 
> e.g., if you tell Torque to limit the job to cores 0-4, we will honor that 
> directive and keep all processes within that envelope.
> 
> 
>> 
>> My two cents,
>> Gus Correa
>> 
>> 
>>> On Jun 6, 2014, at 10:01 AM, Reuti 

[OMPI users] --preload-binary does not work

[E.O.]

Hello
I am using OpenMPI ver 1.8.1 on a cluster of 4 machines.
One Redhat 6.2 and three busybox machine. They are all 64bit environment.

I want to use --preload-binary option to send the binary file to hosts but
it's not working.

# /mpi/bin/mpirun --prefix /mpi --preload-files ./a.out --allow-run-as-root
--np 4 --host box0101,box0103 --preload-binary ./a.out
--
mpirun was unable to launch the specified application as it could not access
or execute an executable:

Executable: ./a.out
Node: box0101

while attempting to start process rank 17.
--
17 total processes failed to start
#

If I sent the binary by SCP beforehand, the command works fine. SCP is
working fine without password between the hosts.
Is the option supposed to work?
Thank you,

Eiichi

Re: [OMPI users] Bind multiple cores to rank - OpenMPI 1.8.1

[Ralph Castain]

Sorry to pester with questions, but I'm trying to narrow down the issue.

* What kind of chips are on these machines?

* If they have h/w threads, are they enabled?

* you might have lstopo on one of those machines - could you pass along its 
output? Otherwise, you can run a simple "mpirun -n 1 -mca ess_base_verbose 20 
hostname" and it will print out. Only need one node in your allocation as we 
don't need a fountain of output.

I'll look into the segfault - hard to understand offhand, but could be an 
uninitialized variable. If you have a chance, could you rerun that test with 
"-mca plm_base_verbose 10" on the cmd line?

Thanks again
Ralph

On Jun 6, 2014, at 10:31 AM, Dan Dietz  wrote:

> Thanks for the reply. I tried out the --display-allocation option with
> several different combinations and have attached the output. I see
> this behavior on both RHEL6.4, RHEL6.5, and RHEL5.10 clusters.
> 
> 
> Here's debugging info on the segfault. Does that help? FWIW this does
> not seem to crash on the RHEL5 cluster or RHEL6.5 cluster. Just
> crashes on RHEL6.4.
> 
> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ gdb -c core.22623
> `which mpirun`
> No symbol table is loaded.  Use the "file" command.
> GNU gdb (GDB) 7.5-1.3.187
> Copyright (C) 2012 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu".
> For bug reporting instructions, please see:
> ...
> Reading symbols from
> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/bin/mpirun...done.
> [New LWP 22623]
> [New LWP 22624]
> 
> warning: Can't read pathname for load map: Input/output error.
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `mpirun -np 2 -machinefile ./nodes ./hello'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x2acc602920e1 in orte_plm_base_complete_setup (fd=-1,
> args=-1, cbdata=0x20c0840) at base/plm_base_launch_support.c:422
> 422node->hostid = node->daemon->name.vpid;
> (gdb) bt
> #0  0x2acc602920e1 in orte_plm_base_complete_setup (fd=-1,
> args=-1, cbdata=0x20c0840) at base/plm_base_launch_support.c:422
> #1  0x2acc60eec145 in opal_libevent2021_event_base_loop () from
> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib/libopen-pal.so.6
> #2  0x004073b5 in orterun (argc=6, argv=0x7fff5bb2a3a8) at
> orterun.c:1077
> #3  0x004048f4 in main (argc=6, argv=0x7fff5bb2a3a8) at main.c:13
> 
> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ cat nodes
> conte-a009
> conte-a009
> conte-a055
> conte-a055
> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ uname -r
> 2.6.32-358.14.1.el6.x86_64
> 
> On Thu, Jun 5, 2014 at 7:54 PM, Ralph Castain  wrote:
>> 
>> On Jun 5, 2014, at 2:13 PM, Dan Dietz  wrote:
>> 
>>> Hello all,
>>> 
>>> I'd like to bind 8 cores to a single MPI rank for hybrid MPI/OpenMP
>>> codes. In OMPI 1.6.3, I can do:
>>> 
>>> $ mpirun -np 2 -cpus-per-rank 8  -machinefile ./nodes ./hello
>>> 
>>> I get one rank bound to procs 0-7 and the other bound to 8-15. Great!
>>> 
>>> But I'm having some difficulties doing this with openmpi 1.8.1:
>>> 
>>> $ mpirun -np 2 -cpus-per-rank 8  -machinefile ./nodes ./hello
>>> --
>>> The following command line options and corresponding MCA parameter have
>>> been deprecated and replaced as follows:
>>> 
>>> Command line options:
>>>   Deprecated:  --cpus-per-proc, -cpus-per-proc, --cpus-per-rank,
>>> -cpus-per-rank
>>>   Replacement: --map-by :PE=N
>>> 
>>> Equivalent MCA parameter:
>>>   Deprecated:  rmaps_base_cpus_per_proc
>>>   Replacement: rmaps_base_mapping_policy=:PE=N
>>> 
>>> The deprecated forms *will* disappear in a future version of Open MPI.
>>> Please update to the new syntax.
>>> --
>>> --
>>> There are not enough slots available in the system to satisfy the 2 slots
>>> that were requested by the application:
>>> ./hello
>>> 
>>> Either request fewer slots for your application, or make more slots 
>>> available
>>> for use.
>>> --
>>> 
>>> OK, let me try the new syntax...
>>> 
>>> $ mpirun -np 2 --map-by core:pe=8 -machinefile ./nodes ./hello
>>> --
>>> There are not enough slots available in the system to satisfy the 2 slots
>>> that were requested by the application:

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

Re: $PBS_NODEFILE, we use that to create the hostfile that is passed via 
--hostfile (i.e. the two are the same).  

To further debug this, I passed "--display-allocation --display-map" to 
orterun, which resulted in:

==   ALLOCATED NODES   ==

 Data for node: node0001Num slots: 16   Max slots: 0
 Data for node: node0002  Num slots: 8Max slots: 0

=

    JOB MAP   

 Data for node: node0001Num procs: 24
Process OMPI jobid: [24552,1] Process rank: 0
Process OMPI jobid: [24552,1] Process rank: 1
Process OMPI jobid: [24552,1] Process rank: 2
Process OMPI jobid: [24552,1] Process rank: 3
Process OMPI jobid: [24552,1] Process rank: 4
Process OMPI jobid: [24552,1] Process rank: 5
Process OMPI jobid: [24552,1] Process rank: 6
Process OMPI jobid: [24552,1] Process rank: 7
Process OMPI jobid: [24552,1] Process rank: 8
Process OMPI jobid: [24552,1] Process rank: 9
Process OMPI jobid: [24552,1] Process rank: 10
Process OMPI jobid: [24552,1] Process rank: 11
Process OMPI jobid: [24552,1] Process rank: 12
Process OMPI jobid: [24552,1] Process rank: 13
Process OMPI jobid: [24552,1] Process rank: 14
Process OMPI jobid: [24552,1] Process rank: 15
Process OMPI jobid: [24552,1] Process rank: 16
Process OMPI jobid: [24552,1] Process rank: 17
Process OMPI jobid: [24552,1] Process rank: 18
Process OMPI jobid: [24552,1] Process rank: 19
Process OMPI jobid: [24552,1] Process rank: 20
Process OMPI jobid: [24552,1] Process rank: 21
Process OMPI jobid: [24552,1] Process rank: 22
Process OMPI jobid: [24552,1] Process rank: 23

I have been going through the man page of mpirun as well as the OpenMPI mailing 
list and website, and thus far have been unable to determine the reason for the 
oversubscription of the head node (node0001) when even the PBS scheduler is 
passing along the correct slot count #s (16 and 8, resp).

Am I running into a bug w/ OpenMPI 1.6?

--john



-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 1:30 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI


On Jun 6, 2014, at 10:24 AM, Gus Correa  wrote:

> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>> You can always add --display-allocation to the cmd line to see what 
>> we thought we received.
>> 
>> If you configure OMPI with --enable-debug, you can set --mca 
>> ras_base_verbose 10 to see the details
>> 
>> 
> 
> Hi John
> 
> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job script. 
>  This will list the nodes (multiple times according to the number of cores 
> requested).
> I find this useful documentation,
> along with job number, work directory, etc.
> "man qsub" will show you all the PBS_* environment variables available 
> to the job.
> For instance, you can echo them using a Torque 'prolog' script, if the 
> user didn't do it. That will appear in the Torque STDOUT file.
> 
> From outside the job script, "qstat -n" (and variants, say, with -u 
> username) will list the nodes allocated to each job, again multiple 
> times as per the requested cores.
> 
> "tracejob job_number" will show similar information.
> 
> 
> If you configured Torque --with-cpuset, there is more information 
> about the cpuset allocated to the job in /dev/cpuset/torque/jobnumber 
> (on the first node listed above, called "mother superior" in Torque parlance).
> This mostly matter if there is more than one job running on a node.
> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
> whatever.  As Ralph said, Open MPI does that.
> I believe Open MPI doesn't use the cpuset info from Torque.
> (Ralph, please correct me if I am wrong.)

You are correct in that we don't use any per-process designations. We do, 
however, work inside any overall envelope that Torque may impose on us - e.g., 
if you tell Torque to limit the job to cores 0-4, we will honor that directive 
and keep all processes within that envelope.


> 
> My two cents,
> Gus Correa
> 
> 
>> On Jun 6, 2014, at 10:01 AM, Reuti > > wrote:
>> 
>>> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
>>> 
 OK, so at the least, how can I get the node and slots/node info 
 that is passed from PBS?
 
 I ask because I'm trying to troubleshoot a problem w/ PBS and the 
 build of OpenMPI 1.6 I noted.  If I submit a 24-process simple job 
 through PBS using a script which has:
 
 /usr/local/openmpi/bin/orterun -n 24 --hostfile 
 /home/sasso/TEST/hosts.file 

Re: [OMPI users] spml_ikrit_np random values

[Mike Dubman]

fixed here: https://svn.open-mpi.org/trac/ompi/changeset/31962

Thanks for report.


On Thu, Jun 5, 2014 at 7:45 PM, Mike Dubman 
wrote:

> seems oshmem_info uses uninitialized value.
> we will check it, thanks for report.
>
>
> On Thu, Jun 5, 2014 at 6:56 PM, Timur Ismagilov 
> wrote:
>
>> Hello!
>>
>> I am using Open MPI v1.8.1.
>>
>> $oshmem_info -a --parsable | grep spml_ikrit_np
>>
>> mca:spml:ikrit:param:spml_ikrit_np:value:1620524368  (alwase new value)
>> mca:spml:ikrit:param:spml_ikrit_np:source:default
>> mca:spml:ikrit:param:spml_ikrit_np:status:writeable
>> mca:spml:ikrit:param:spml_ikrit_np:level:9
>> mca:spml:ikrit:param:spml_ikrit_np:help:[integer] Minimal allowed job's
>> NP to activate ikrit
>> mca:spml:ikrit:param:spml_ikrit_np:deprecated:no
>> mca:spml:ikrit:param:spml_ikrit_np:type:int
>> mca:spml:ikrit:param:spml_ikrit_np:disabled:false
>>
>> why spml_ikrit_np gets a new value each time?
>> Regards,
>> Timur
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>

Re: [OMPI users] Bind multiple cores to rank - OpenMPI 1.8.1

[Dan Dietz]

Thanks for the reply. I tried out the --display-allocation option with
several different combinations and have attached the output. I see
this behavior on both RHEL6.4, RHEL6.5, and RHEL5.10 clusters.


Here's debugging info on the segfault. Does that help? FWIW this does
not seem to crash on the RHEL5 cluster or RHEL6.5 cluster. Just
crashes on RHEL6.4.

ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ gdb -c core.22623
`which mpirun`
No symbol table is loaded.  Use the "file" command.
GNU gdb (GDB) 7.5-1.3.187
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from
/scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/bin/mpirun...done.
[New LWP 22623]
[New LWP 22624]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `mpirun -np 2 -machinefile ./nodes ./hello'.
Program terminated with signal 11, Segmentation fault.
#0  0x2acc602920e1 in orte_plm_base_complete_setup (fd=-1,
args=-1, cbdata=0x20c0840) at base/plm_base_launch_support.c:422
422node->hostid = node->daemon->name.vpid;
(gdb) bt
#0  0x2acc602920e1 in orte_plm_base_complete_setup (fd=-1,
args=-1, cbdata=0x20c0840) at base/plm_base_launch_support.c:422
#1  0x2acc60eec145 in opal_libevent2021_event_base_loop () from
/scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib/libopen-pal.so.6
#2  0x004073b5 in orterun (argc=6, argv=0x7fff5bb2a3a8) at
orterun.c:1077
#3  0x004048f4 in main (argc=6, argv=0x7fff5bb2a3a8) at main.c:13

ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ cat nodes
conte-a009
conte-a009
conte-a055
conte-a055
ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ uname -r
2.6.32-358.14.1.el6.x86_64

On Thu, Jun 5, 2014 at 7:54 PM, Ralph Castain  wrote:
>
> On Jun 5, 2014, at 2:13 PM, Dan Dietz  wrote:
>
>> Hello all,
>>
>> I'd like to bind 8 cores to a single MPI rank for hybrid MPI/OpenMP
>> codes. In OMPI 1.6.3, I can do:
>>
>> $ mpirun -np 2 -cpus-per-rank 8  -machinefile ./nodes ./hello
>>
>> I get one rank bound to procs 0-7 and the other bound to 8-15. Great!
>>
>> But I'm having some difficulties doing this with openmpi 1.8.1:
>>
>> $ mpirun -np 2 -cpus-per-rank 8  -machinefile ./nodes ./hello
>> --
>> The following command line options and corresponding MCA parameter have
>> been deprecated and replaced as follows:
>>
>>  Command line options:
>>Deprecated:  --cpus-per-proc, -cpus-per-proc, --cpus-per-rank,
>> -cpus-per-rank
>>Replacement: --map-by :PE=N
>>
>>  Equivalent MCA parameter:
>>Deprecated:  rmaps_base_cpus_per_proc
>>Replacement: rmaps_base_mapping_policy=:PE=N
>>
>> The deprecated forms *will* disappear in a future version of Open MPI.
>> Please update to the new syntax.
>> --
>> --
>> There are not enough slots available in the system to satisfy the 2 slots
>> that were requested by the application:
>>  ./hello
>>
>> Either request fewer slots for your application, or make more slots available
>> for use.
>> --
>>
>> OK, let me try the new syntax...
>>
>> $ mpirun -np 2 --map-by core:pe=8 -machinefile ./nodes ./hello
>> --
>> There are not enough slots available in the system to satisfy the 2 slots
>> that were requested by the application:
>>  ./hello
>>
>> Either request fewer slots for your application, or make more slots available
>> for use.
>> --
>>
>> What am I doing wrong? The documentation on these new options is
>> somewhat poor and confusing so I'm probably doing something wrong. If
>> anyone could provide some pointers here it'd be much appreciated! If
>> it's not something simple and you need config logs and such please let
>> me know.
>
> Looks like we think there are less than 16 slots allocated on that node. What 
> is in this "nodes" file? Without it, OMPI should read the Torque allocation 
> directly. You might check what we think the allocation is by adding 
> --display-allocation to the cmd line
>
>>
>> As a side note -
>>
>> If I try this using the PBS nodefile with the above, I get a confusing 
>> message:
>>
>> 

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Ralph Castain]

On Jun 6, 2014, at 10:24 AM, Gus Correa  wrote:

> On 06/06/2014 01:05 PM, Ralph Castain wrote:
>> You can always add --display-allocation to the cmd line to see what we
>> thought we received.
>> 
>> If you configure OMPI with --enable-debug, you can set --mca
>> ras_base_verbose 10 to see the details
>> 
>> 
> 
> Hi John
> 
> On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job script. 
>  This will list the nodes (multiple times according to the number of cores 
> requested).
> I find this useful documentation,
> along with job number, work directory, etc.
> "man qsub" will show you all the PBS_* environment variables
> available to the job.
> For instance, you can echo them using a Torque
> 'prolog' script, if the user
> didn't do it. That will appear in the Torque STDOUT file.
> 
> From outside the job script, "qstat -n" (and variants, say, with -u username)
> will list the nodes allocated to each job,
> again multiple times as per the requested cores.
> 
> "tracejob job_number" will show similar information.
> 
> 
> If you configured Torque --with-cpuset,
> there is more information about the cpuset allocated to the job
> in /dev/cpuset/torque/jobnumber (on the first node listed above, called 
> "mother superior" in Torque parlance).
> This mostly matter if there is more than one job running on a node.
> However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
> whatever.  As Ralph said, Open MPI does that.
> I believe Open MPI doesn't use the cpuset info from Torque.
> (Ralph, please correct me if I am wrong.)

You are correct in that we don't use any per-process designations. We do, 
however, work inside any overall envelope that Torque may impose on us - e.g., 
if you tell Torque to limit the job to cores 0-4, we will honor that directive 
and keep all processes within that envelope.


> 
> My two cents,
> Gus Correa
> 
> 
>> On Jun 6, 2014, at 10:01 AM, Reuti > > wrote:
>> 
>>> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
>>> 
 OK, so at the least, how can I get the node and slots/node info that
 is passed from PBS?
 
 I ask because I’m trying to troubleshoot a problem w/ PBS and the
 build of OpenMPI 1.6 I noted.  If I submit a 24-process simple job
 through PBS using a script which has:
 
 /usr/local/openmpi/bin/orterun -n 24 --hostfile
 /home/sasso/TEST/hosts.file --mca orte_rsh_agent rsh --mca btl
 openib,tcp,self --mca orte_base_help_aggregate 0 -x PATH -x
 LD_LIBRARY_PATH /home/sasso/TEST/simplempihello.exe
>>> 
>>> Using the --hostfile on your own would mean to violate the granted
>>> slot allocation by PBS. Just leave this option out. How do you submit
>>> your job?
>>> 
>>> -- Reuti
>>> 
>>> 
 And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the
 first 16 being host node0001 and the last 8 being node0002), it
 appears that 24 MPI tasks try to start on node0001 instead of getting
 distributed as 16 on node0001 and 8 on node0002.   Hence, I am
 curious what is being passed by PBS.
 
 --john
 
 
 From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph
 Castain
 Sent: Friday, June 06, 2014 12:31 PM
 To: Open MPI Users
 Subject: Re: [OMPI users] Determining what parameters a scheduler
 passes to OpenMPI
 
 We currently only get the node and slots/node info from PBS - we
 don't get any task placement info at all. We then use the mpirun cmd
 options and built-in mappers to map the tasks to the nodes.
 
 I suppose we could do more integration in that regard, but haven't
 really seen a reason to do so - the OMPI mappers are generally more
 flexible than anything in the schedulers.
 
 
 On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE)
 > wrote:
 
 
 For the PBS scheduler and using a build of OpenMPI 1.6 built against
 PBS include files + libs, is there a way to determine (perhaps via
 some debugging flags passed to mpirun) what job placement parameters
 are passed from the PBS scheduler to OpenMPI?  In particular, I am
 talking about task placement info such as nodes to place on, etc.
  Thanks!
 
 --john
 ___
 users mailing list
 us...@open-mpi.org 
 http://www.open-mpi.org/mailman/listinfo.cgi/users
 
 ___
 users mailing list
 us...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org 
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> 
>> 

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Gus Correa]

On 06/06/2014 01:05 PM, Ralph Castain wrote:

You can always add --display-allocation to the cmd line to see what we
thought we received.

If you configure OMPI with --enable-debug, you can set --mca
ras_base_verbose 10 to see the details




Hi John

On the Torque side, you can put a line "cat $PBS_NODEFILE" on the job 
script.  This will list the nodes (multiple times according to the 
number of cores requested).

I find this useful documentation,
along with job number, work directory, etc.
"man qsub" will show you all the PBS_* environment variables
available to the job.
For instance, you can echo them using a Torque
'prolog' script, if the user
didn't do it. That will appear in the Torque STDOUT file.

From outside the job script, "qstat -n" (and variants, say, with -u 
username)

will list the nodes allocated to each job,
again multiple times as per the requested cores.

"tracejob job_number" will show similar information.


If you configured Torque --with-cpuset,
there is more information about the cpuset allocated to the job
in /dev/cpuset/torque/jobnumber (on the first node listed above, called 
"mother superior" in Torque parlance).

This mostly matter if there is more than one job running on a node.
However, Torque doesn't bind processes/MPI_ranks to cores or sockets or 
whatever.  As Ralph said, Open MPI does that.

I believe Open MPI doesn't use the cpuset info from Torque.
(Ralph, please correct me if I am wrong.)

My two cents,
Gus Correa



On Jun 6, 2014, at 10:01 AM, Reuti > wrote:


Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):


OK, so at the least, how can I get the node and slots/node info that
is passed from PBS?

I ask because I’m trying to troubleshoot a problem w/ PBS and the
build of OpenMPI 1.6 I noted.  If I submit a 24-process simple job
through PBS using a script which has:

/usr/local/openmpi/bin/orterun -n 24 --hostfile
/home/sasso/TEST/hosts.file --mca orte_rsh_agent rsh --mca btl
openib,tcp,self --mca orte_base_help_aggregate 0 -x PATH -x
LD_LIBRARY_PATH /home/sasso/TEST/simplempihello.exe


Using the --hostfile on your own would mean to violate the granted
slot allocation by PBS. Just leave this option out. How do you submit
your job?

-- Reuti



And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the
first 16 being host node0001 and the last 8 being node0002), it
appears that 24 MPI tasks try to start on node0001 instead of getting
distributed as 16 on node0001 and 8 on node0002.   Hence, I am
curious what is being passed by PBS.

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph
Castain
Sent: Friday, June 06, 2014 12:31 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler
passes to OpenMPI

We currently only get the node and slots/node info from PBS - we
don't get any task placement info at all. We then use the mpirun cmd
options and built-in mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't
really seen a reason to do so - the OMPI mappers are generally more
flexible than anything in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE)
> wrote:


For the PBS scheduler and using a build of OpenMPI 1.6 built against
PBS include files + libs, is there a way to determine (perhaps via
some debugging flags passed to mpirun) what job placement parameters
are passed from the PBS scheduler to OpenMPI?  In particular, I am
talking about task placement info such as nodes to place on, etc.
  Thanks!

 --john
___
users mailing list
us...@open-mpi.org 
http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org 
http://www.open-mpi.org/mailman/listinfo.cgi/users




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Ralph Castain]

You can always add --display-allocation to the cmd line to see what we thought 
we received.

If you configure OMPI with --enable-debug, you can set --mca ras_base_verbose 
10 to see the details


On Jun 6, 2014, at 10:01 AM, Reuti  wrote:

> Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):
> 
>> OK, so at the least, how can I get the node and slots/node info that is 
>> passed from PBS?
>> 
>> I ask because I’m trying to troubleshoot a problem w/ PBS and the build of 
>> OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using 
>> a script which has:
>> 
>> /usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
>> --mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
>> orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
>> /home/sasso/TEST/simplempihello.exe
> 
> Using the --hostfile on your own would mean to violate the granted slot 
> allocation by PBS. Just leave this option out. How do you submit your job?
> 
> -- Reuti
> 
> 
>> And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 
>> 16 being host node0001 and the last 8 being node0002), it appears that 24 
>> MPI tasks try to start on node0001 instead of getting distributed as 16 on 
>> node0001 and 8 on node0002.   Hence, I am curious what is being passed by 
>> PBS.
>> 
>> --john
>> 
>> 
>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
>> Sent: Friday, June 06, 2014 12:31 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
>> OpenMPI
>> 
>> We currently only get the node and slots/node info from PBS - we don't get 
>> any task placement info at all. We then use the mpirun cmd options and 
>> built-in mappers to map the tasks to the nodes.
>> 
>> I suppose we could do more integration in that regard, but haven't really 
>> seen a reason to do so - the OMPI mappers are generally more flexible than 
>> anything in the schedulers.
>> 
>> 
>> On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
>>  wrote:
>> 
>> 
>> For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
>> include files + libs, is there a way to determine (perhaps via some 
>> debugging flags passed to mpirun) what job placement parameters are passed 
>> from the PBS scheduler to OpenMPI?  In particular, I am talking about task 
>> placement info such as nodes to place on, etc.   Thanks!
>> 
>>  --john
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Reuti]

Am 06.06.2014 um 18:58 schrieb Sasso, John (GE Power & Water, Non-GE):

> OK, so at the least, how can I get the node and slots/node info that is 
> passed from PBS?
>  
> I ask because I’m trying to troubleshoot a problem w/ PBS and the build of 
> OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using a 
> script which has:
>  
> /usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
> --mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
> orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
> /home/sasso/TEST/simplempihello.exe

Using the --hostfile on your own would mean to violate the granted slot 
allocation by PBS. Just leave this option out. How do you submit your job?

-- Reuti


> And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 
> 16 being host node0001 and the last 8 being node0002), it appears that 24 MPI 
> tasks try to start on node0001 instead of getting distributed as 16 on 
> node0001 and 8 on node0002.   Hence, I am curious what is being passed by PBS.
>  
> --john
>  
>  
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 12:31 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
> OpenMPI
>  
> We currently only get the node and slots/node info from PBS - we don't get 
> any task placement info at all. We then use the mpirun cmd options and 
> built-in mappers to map the tasks to the nodes.
>  
> I suppose we could do more integration in that regard, but haven't really 
> seen a reason to do so - the OMPI mappers are generally more flexible than 
> anything in the schedulers.
>  
>  
> On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
>  wrote:
> 
> 
> For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
> include files + libs, is there a way to determine (perhaps via some debugging 
> flags passed to mpirun) what job placement parameters are passed from the PBS 
> scheduler to OpenMPI?  In particular, I am talking about task placement info 
> such as nodes to place on, etc.   Thanks!
>  
>   --john
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>  
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

OK, so at the least, how can I get the node and slots/node info that is passed 
from PBS?

I ask because I'm trying to troubleshoot a problem w/ PBS and the build of 
OpenMPI 1.6 I noted.  If I submit a 24-process simple job through PBS using a 
script which has:

/usr/local/openmpi/bin/orterun -n 24 --hostfile /home/sasso/TEST/hosts.file 
--mca orte_rsh_agent rsh --mca btl openib,tcp,self --mca 
orte_base_help_aggregate 0 -x PATH -x LD_LIBRARY_PATH 
/home/sasso/TEST/simplempihello.exe


And the hostfile /home/sasso/TEST/hosts.file contains 24 entries (the first 16 
being host node0001 and the last 8 being node0002), it appears that 24 MPI 
tasks try to start on node0001 instead of getting distributed as 16 on node0001 
and 8 on node0002.   Hence, I am curious what is being passed by PBS.

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 12:31 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determining what parameters a scheduler passes to 
OpenMPI

We currently only get the node and slots/node info from PBS - we don't get any 
task placement info at all. We then use the mpirun cmd options and built-in 
mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't really seen 
a reason to do so - the OMPI mappers are generally more flexible than anything 
in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
> wrote:


For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
include files + libs, is there a way to determine (perhaps via some debugging 
flags passed to mpirun) what job placement parameters are passed from the PBS 
scheduler to OpenMPI?  In particular, I am talking about task placement info 
such as nodes to place on, etc.   Thanks!

  --john
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openib segfaults with Torque

[Ralph Castain]

Fascinating - I can only assume that Torque is setting something in the 
environment that is creating the confusion. Sadly, Nathan is at the MPI Forum 
this week, so we may have to wait until Mon to get his input on the problem as 
he wrote the udcm code.


On Jun 6, 2014, at 8:51 AM, Fischer, Greg A.  wrote:

> Yep, TCP works fine when launched via Torque/qsub:
>  
> [binf315:fischega] $ mpirun -np 2 -mca btl tcp,sm,self ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> Process 0 decremented value: 8
> Process 0 decremented value: 7
> Process 0 decremented value: 6
> Process 0 decremented value: 5
> Process 0 decremented value: 4
> Process 0 decremented value: 3
> Process 0 decremented value: 2
> Process 0 decremented value: 1
> Process 0 decremented value: 0
> Process 0 exiting
> Process 1 exiting
>  
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Friday, June 06, 2014 10:34 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] openib segfaults with Torque
>  
> Huh - how strange. I can't imagine what it has to do with Torque vs rsh - 
> this is failing when the openib BTL is trying to create the connection, which 
> comes way after the launch is complete.
>  
> Are you able to run this with btl tcp,sm,self? If so, that would confirm that 
> everything else is correct, and the problem truly is limited to the udcm 
> itself...which shouldn't have anything to do with how the proc was launched.
>  
>  
> On Jun 6, 2014, at 6:47 AM, Fischer, Greg A.  
> wrote:
> 
> 
> Here are the results when logging in to the compute node via ssh and running 
> as you suggest:
>  
> [binf102:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> Process 0 decremented value: 8
> Process 0 decremented value: 7
> Process 0 decremented value: 6
> Process 0 decremented value: 5
> Process 0 decremented value: 4
> Process 0 decremented value: 3
> Process 0 decremented value: 2
> Process 0 decremented value: 1
> Process 0 decremented value: 0
> Process 0 exiting
> Process 1 exiting
>  
> Here are the results when executing over Torque (launch the shell with “qsub 
> -l nodes=2 –I”):
>  
> [binf316:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
> ring_c: 
> ../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
>  udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
> ((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
> [binf316:21584] *** Process received signal ***
> [binf316:21584] Signal: Aborted (6)
> [binf316:21584] Signal code:  (-6)
> ring_c: 
> ../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
>  udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
> ((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
> [binf316:21583] *** Process received signal ***
> [binf316:21583] Signal: Aborted (6)
> [binf316:21583] Signal code:  (-6)
> [binf316:21584] [ 0] /lib64/libpthread.so.0(+0xf7c0)[0x7fe33a2637c0]
> [binf316:21584] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x7fe339f0fb55]
> [binf316:21584] [ 2] /lib64/libc.so.6(abort+0x181)[0x7fe339f11131]
> [binf316:21584] [ 3] /lib64/libc.so.6(__assert_fail+0xf0)[0x7fe339f08a10]
> [binf316:21584] [ 4] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7fe3355a984b]
> [binf316:21584] [ 5] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7fe3355a8474]
> [binf316:21584] [ 6] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7fe3355a1316]
> [binf316:21584] [ 7] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7fe33558a817]
> [binf316:21584] [ 8] [binf316:21583] [ 0] 
> /lib64/libpthread.so.0(+0xf7c0)[0x7f3b586697c0]
> [binf316:21583] [ 1] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_btl_base_select+0x1b2)[0x7fe33a532a5e]
> [binf316:21584] [ 9] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x20)[0x7fe3357ccd42]
> [binf316:21584] [10] /lib64/libc.so.6(gsignal+0x35)[0x7f3b58315b55]
> [binf316:21583] [ 2] /lib64/libc.so.6(abort+0x181)[0x7f3b58317131]
> [binf316:21583] [ 3] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_bml_base_init+0xd6)[0x7fe33a531d1b]
> [binf316:21584] [11] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_pml_ob1.so(+0x7739)[0x7fe3344e7739]
> [binf316:21584] [12] /lib64/libc.so.6(__assert_fail+0xf0)[0x7f3b5830ea10]
> [binf316:21583] [ 4] 
> 

Re: [OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Ralph Castain]

We currently only get the node and slots/node info from PBS - we don't get any 
task placement info at all. We then use the mpirun cmd options and built-in 
mappers to map the tasks to the nodes.

I suppose we could do more integration in that regard, but haven't really seen 
a reason to do so - the OMPI mappers are generally more flexible than anything 
in the schedulers.


On Jun 6, 2014, at 9:08 AM, Sasso, John (GE Power & Water, Non-GE) 
 wrote:

> For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
> include files + libs, is there a way to determine (perhaps via some debugging 
> flags passed to mpirun) what job placement parameters are passed from the PBS 
> scheduler to OpenMPI?  In particular, I am talking about task placement info 
> such as nodes to place on, etc.   Thanks!
>  
>   --john
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Determining what parameters a scheduler passes to OpenMPI

[Sasso, John (GE Power & Water, Non-GE)]

For the PBS scheduler and using a build of OpenMPI 1.6 built against PBS 
include files + libs, is there a way to determine (perhaps via some debugging 
flags passed to mpirun) what job placement parameters are passed from the PBS 
scheduler to OpenMPI?  In particular, I am talking about task placement info 
such as nodes to place on, etc.   Thanks!

  --john

Re: [OMPI users] openib segfaults with Torque

[Fischer, Greg A.]

Yep, TCP works fine when launched via Torque/qsub:

[binf315:fischega] $ mpirun -np 2 -mca btl tcp,sm,self ring_c
Process 0 sending 10 to 1, tag 201 (2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Friday, June 06, 2014 10:34 AM
To: Open MPI Users
Subject: Re: [OMPI users] openib segfaults with Torque

Huh - how strange. I can't imagine what it has to do with Torque vs rsh - this 
is failing when the openib BTL is trying to create the connection, which comes 
way after the launch is complete.

Are you able to run this with btl tcp,sm,self? If so, that would confirm that 
everything else is correct, and the problem truly is limited to the udcm 
itself...which shouldn't have anything to do with how the proc was launched.


On Jun 6, 2014, at 6:47 AM, Fischer, Greg A. 
> wrote:


Here are the results when logging in to the compute node via ssh and running as 
you suggest:

[binf102:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
Process 0 sending 10 to 1, tag 201 (2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting

Here are the results when executing over Torque (launch the shell with "qsub -l 
nodes=2 -I"):

[binf316:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
ring_c: 
../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
 udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
[binf316:21584] *** Process received signal ***
[binf316:21584] Signal: Aborted (6)
[binf316:21584] Signal code:  (-6)
ring_c: 
../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
 udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
[binf316:21583] *** Process received signal ***
[binf316:21583] Signal: Aborted (6)
[binf316:21583] Signal code:  (-6)
[binf316:21584] [ 0] /lib64/libpthread.so.0(+0xf7c0)[0x7fe33a2637c0]
[binf316:21584] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x7fe339f0fb55]
[binf316:21584] [ 2] /lib64/libc.so.6(abort+0x181)[0x7fe339f11131]
[binf316:21584] [ 3] /lib64/libc.so.6(__assert_fail+0xf0)[0x7fe339f08a10]
[binf316:21584] [ 4] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7fe3355a984b]
[binf316:21584] [ 5] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7fe3355a8474]
[binf316:21584] [ 6] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7fe3355a1316]
[binf316:21584] [ 7] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7fe33558a817]
[binf316:21584] [ 8] [binf316:21583] [ 0] 
/lib64/libpthread.so.0(+0xf7c0)[0x7f3b586697c0]
[binf316:21583] [ 1] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_btl_base_select+0x1b2)[0x7fe33a532a5e]
[binf316:21584] [ 9] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x20)[0x7fe3357ccd42]
[binf316:21584] [10] /lib64/libc.so.6(gsignal+0x35)[0x7f3b58315b55]
[binf316:21583] [ 2] /lib64/libc.so.6(abort+0x181)[0x7f3b58317131]
[binf316:21583] [ 3] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_bml_base_init+0xd6)[0x7fe33a531d1b]
[binf316:21584] [11] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_pml_ob1.so(+0x7739)[0x7fe3344e7739]
[binf316:21584] [12] /lib64/libc.so.6(__assert_fail+0xf0)[0x7f3b5830ea10]
[binf316:21583] [ 4] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7f3b539af84b]
[binf316:21583] [ 5] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7f3b539ae474]
[binf316:21583] [ 6] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7f3b539a7316]
[binf316:21583] [ 7] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7f3b53990817]
[binf316:21583] [ 

Re: [OMPI users] openib segfaults with Torque

[Ralph Castain]

Huh - how strange. I can't imagine what it has to do with Torque vs rsh - this 
is failing when the openib BTL is trying to create the connection, which comes 
way after the launch is complete.

Are you able to run this with btl tcp,sm,self? If so, that would confirm that 
everything else is correct, and the problem truly is limited to the udcm 
itself...which shouldn't have anything to do with how the proc was launched.


On Jun 6, 2014, at 6:47 AM, Fischer, Greg A.  wrote:

> Here are the results when logging in to the compute node via ssh and running 
> as you suggest:
>  
> [binf102:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> Process 0 decremented value: 8
> Process 0 decremented value: 7
> Process 0 decremented value: 6
> Process 0 decremented value: 5
> Process 0 decremented value: 4
> Process 0 decremented value: 3
> Process 0 decremented value: 2
> Process 0 decremented value: 1
> Process 0 decremented value: 0
> Process 0 exiting
> Process 1 exiting
>  
> Here are the results when executing over Torque (launch the shell with “qsub 
> -l nodes=2 –I”):
>  
> [binf316:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
> ring_c: 
> ../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
>  udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
> ((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
> [binf316:21584] *** Process received signal ***
> [binf316:21584] Signal: Aborted (6)
> [binf316:21584] Signal code:  (-6)
> ring_c: 
> ../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
>  udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
> ((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
> [binf316:21583] *** Process received signal ***
> [binf316:21583] Signal: Aborted (6)
> [binf316:21583] Signal code:  (-6)
> [binf316:21584] [ 0] /lib64/libpthread.so.0(+0xf7c0)[0x7fe33a2637c0]
> [binf316:21584] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x7fe339f0fb55]
> [binf316:21584] [ 2] /lib64/libc.so.6(abort+0x181)[0x7fe339f11131]
> [binf316:21584] [ 3] /lib64/libc.so.6(__assert_fail+0xf0)[0x7fe339f08a10]
> [binf316:21584] [ 4] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7fe3355a984b]
> [binf316:21584] [ 5] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7fe3355a8474]
> [binf316:21584] [ 6] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7fe3355a1316]
> [binf316:21584] [ 7] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7fe33558a817]
> [binf316:21584] [ 8] [binf316:21583] [ 0] 
> /lib64/libpthread.so.0(+0xf7c0)[0x7f3b586697c0]
> [binf316:21583] [ 1] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_btl_base_select+0x1b2)[0x7fe33a532a5e]
> [binf316:21584] [ 9] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x20)[0x7fe3357ccd42]
> [binf316:21584] [10] /lib64/libc.so.6(gsignal+0x35)[0x7f3b58315b55]
> [binf316:21583] [ 2] /lib64/libc.so.6(abort+0x181)[0x7f3b58317131]
> [binf316:21583] [ 3] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_bml_base_init+0xd6)[0x7fe33a531d1b]
> [binf316:21584] [11] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_pml_ob1.so(+0x7739)[0x7fe3344e7739]
> [binf316:21584] [12] /lib64/libc.so.6(__assert_fail+0xf0)[0x7f3b5830ea10]
> [binf316:21583] [ 4] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7f3b539af84b]
> [binf316:21583] [ 5] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7f3b539ae474]
> [binf316:21583] [ 6] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7f3b539a7316]
> [binf316:21583] [ 7] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7f3b53990817]
> [binf316:21583] [ 8] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_pml_base_select+0x26e)[0x7fe33a5589b2]
> [binf316:21584] [13] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_btl_base_select+0x1b2)[0x7f3b58938a5e]
> [binf316:21583] [ 9] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x20)[0x7f3b53bd2d42]
> [binf316:21583] [10] 
> //_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(ompi_mpi_init+0x5f6)[0x7fe33a4c533c]
> [binf316:21584] [14] 
> 

Re: [OMPI users] openib segfaults with Torque

[Fischer, Greg A.]

Here are the results when logging in to the compute node via ssh and running as 
you suggest:

[binf102:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
Process 0 sending 10 to 1, tag 201 (2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting

Here are the results when executing over Torque (launch the shell with "qsub -l 
nodes=2 -I"):

[binf316:fischega] $ mpirun -np 2 -mca btl openib,sm,self ring_c
ring_c: 
../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
 udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
[binf316:21584] *** Process received signal ***
[binf316:21584] Signal: Aborted (6)
[binf316:21584] Signal code:  (-6)
ring_c: 
../../../../../openmpi-1.8.1/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:734:
 udcm_module_finalize: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == 
((opal_object_t *) (>cm_recv_msg_queue))->obj_magic_id' failed.
[binf316:21583] *** Process received signal ***
[binf316:21583] Signal: Aborted (6)
[binf316:21583] Signal code:  (-6)
[binf316:21584] [ 0] /lib64/libpthread.so.0(+0xf7c0)[0x7fe33a2637c0]
[binf316:21584] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x7fe339f0fb55]
[binf316:21584] [ 2] /lib64/libc.so.6(abort+0x181)[0x7fe339f11131]
[binf316:21584] [ 3] /lib64/libc.so.6(__assert_fail+0xf0)[0x7fe339f08a10]
[binf316:21584] [ 4] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7fe3355a984b]
[binf316:21584] [ 5] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7fe3355a8474]
[binf316:21584] [ 6] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7fe3355a1316]
[binf316:21584] [ 7] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7fe33558a817]
[binf316:21584] [ 8] [binf316:21583] [ 0] 
/lib64/libpthread.so.0(+0xf7c0)[0x7f3b586697c0]
[binf316:21583] [ 1] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_btl_base_select+0x1b2)[0x7fe33a532a5e]
[binf316:21584] [ 9] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x20)[0x7fe3357ccd42]
[binf316:21584] [10] /lib64/libc.so.6(gsignal+0x35)[0x7f3b58315b55]
[binf316:21583] [ 2] /lib64/libc.so.6(abort+0x181)[0x7f3b58317131]
[binf316:21583] [ 3] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_bml_base_init+0xd6)[0x7fe33a531d1b]
[binf316:21584] [11] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_pml_ob1.so(+0x7739)[0x7fe3344e7739]
[binf316:21584] [12] /lib64/libc.so.6(__assert_fail+0xf0)[0x7f3b5830ea10]
[binf316:21583] [ 4] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x3784b)[0x7f3b539af84b]
[binf316:21583] [ 5] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x36474)[0x7f3b539ae474]
[binf316:21583] [ 6] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0x15b)[0x7f3b539a7316]
[binf316:21583] [ 7] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_btl_openib.so(+0x18817)[0x7f3b53990817]
[binf316:21583] [ 8] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_pml_base_select+0x26e)[0x7fe33a5589b2]
[binf316:21584] [13] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_btl_base_select+0x1b2)[0x7f3b58938a5e]
[binf316:21583] [ 9] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x20)[0x7f3b53bd2d42]
[binf316:21583] [10] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(ompi_mpi_init+0x5f6)[0x7fe33a4c533c]
[binf316:21584] [14] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_bml_base_init+0xd6)[0x7f3b58937d1b]
[binf316:21583] [11] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/openmpi/mca_pml_ob1.so(+0x7739)[0x7f3b528ed739]
[binf316:21583] [12] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(mca_pml_base_select+0x26e)[0x7f3b5895e9b2]
[binf316:21583] [13] 
//_ib/gcc-4.8.3/toolset/openmpi-1.8.1_debug/lib/libmpi.so.1(MPI_Init+0x17e)[0x7fe33a4fa386]
[binf316:21584] [15] ring_c[0x40096f]
[binf316:21584] [16] /lib64/libc.so.6(__libc_start_main+0xe6)[0x7fe339efbc36]
[binf316:21584] [17] ring_c[0x400889]
[binf316:21584] *** End of error message ***

Re: [OMPI users] [warn] Epoll ADD(1) on fd 0 failed

[Ralph Castain]

Possible - honestly don't know

On Jun 6, 2014, at 12:16 AM, Timur Ismagilov  wrote:

> Sometimes,  after termination of the program, launched with the command 
> "sbatch ... -o myprogram.out .",  no file "myprogram.out"  is being 
> produced. Could this be due to the above mentioned problem?
> 
> 
> Thu, 5 Jun 2014 07:45:01 -0700 от Ralph Castain :
> FWIW: support for the --resv-ports option was deprecated and removed on the 
> OMPI side a long time ago.
> 
> I'm not familiar enough with "oshrun" to know if it is doing anything unusual 
> - I believe it is just a renaming of our usual "mpirun". I suspect this is 
> some interaction with sbatch, but I'll take a look. I haven't see that 
> warning. Mike indicated he thought it is due to both slurm and OMPI trying to 
> control stdin/stdout, in which case it shouldn't be happening but you can 
> safely ignore it
> 
> 
> On Jun 5, 2014, at 3:04 AM, Timur Ismagilov  wrote:
> 
>> I use cmd line
>> 
>> $sbatch -p test --exclusive -N 2 -o hello_oshmem.out -e hello_oshmem.err 
>> shrun_mxm3.0 ./hello_oshmem
>> 
>> where script shrun_mxm3.0:
>> 
>> $cat shrun_mxm3.0
>> 
>>   #!/bin/sh
>> 
>>   #srun --resv-ports "$@"
>>   #exit $?
>> 
>>   [ x"$TMPDIR" == x"" ] && TMPDIR=/tmp
>>   HOSTFILE=${TMPDIR}/hostfile.${SLURM_JOB_ID}
>>   srun hostname -s|sort|uniq -c|awk '{print $2" slots="$1}' > $HOSTFILE || { 
>> rm -f $HOSTFILE; exit 255; }
>>   
>> LD_PRELOAD=/mnt/data/users/dm2/vol3/semenov/_scratch/mxm/mxm-3.0/lib/libmxm.so
>>  oshrun -x LD_PRELOAD -x MXM_SHM_KCOPY_MODE=off --hostfile $HOSTFILE "$@"
>> 
>>   rc=$?
>>   rm -f $HOSTFILE
>> 
>>   exit $rc
>> 
>> I configured openmpi using
>> 
>> ./configure CC=icc CXX=icpc F77=ifort FC=ifort 
>> --prefix=/mnt/data/users/dm2/vol3/semenov/_scratch/openmpi-1.8.1_mxm-3.0 
>> --with-mxm=/mnt/data/users/dm2/vol3/semenov/_scratch/mxm/mxm-3.0/ --with-
>>   slurm --with-platform=contrib/platform/mellanox/optimized
>> 
>> 
>> Fri, 30 May 2014 07:09:54 -0700 от Ralph Castain :
>> 
>> Can you pass along the cmd line that generated that output, and how OMPI was 
>> configured?
>> 
>> On May 30, 2014, at 5:11 AM, Тимур Исмагилов  wrote:
>> 
>>> Hello!
>>> 
>>> I am using Open MPI v1.8.1 and slurm 2.5.6.
>>> 
>>> I got this messages when i try to run example (hello_oshmem.cpp) program:
>>> 
>>> [warn] Epoll ADD(1) on fd 0 failed. Old events were 0; read change was 1 
>>> (add); write change was 0 (none): Operation not permitted
>>> [warn] Epoll ADD(4) on fd 1 failed. Old events were 0; read change was 0 
>>> (none); write change was 1 (add): Operation not permitted
>>> Hello, world, I am 0 of 2
>>> Hello, world, I am 1 of 2
>>> 
>>> What does this warnings mean?
>>> 
>>> I lunch this job using sbatch and mpirun with hostfile (got it from :  
>>> $srun hostname -s|sort|uniq -c|awk '{print $2" slots="$1}' > $HOSTFILE)
>>> 
>>> Regards,
>>> Timur
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> 
>> 

> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> 

> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Problem with yoda component in oshmem.

[Mike Dubman]

could you please provide command line ?


On Fri, Jun 6, 2014 at 10:56 AM, Timur Ismagilov  wrote:

> Hello!
>
> I am using Open MPI v1.8.1 in
> example program hello_oshmem.cpp.
>
> When I put  spml_ikrit_np = 1000 (more than 4) and run task on 4 (2,1)
> nodes, I get an:
> in out file:
> No available spml components were found!
>
> This means that there are no components of this type installed on your
> system or all the components reported that they could not be used.
>
> This is a fatal error; your SHMEM process is likely to abort. Check the
> output of the "ompi_info" command and ensure that components of this
> type are available on your system. You may also wish to check the
> value of the "component_path" MCA parameter and ensure that it has at
> least one directory that contains valid MCA components
>
> in err file:
> [node1-128-31:05405] SPML ikrit cannot be selected
>
> Regards,
>
> Timur
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] Problem with yoda component in oshmem.

[Timur Ismagilov]

Hello!
I am using Open MPI v1.8.1 in
example program hello_oshmem.cpp.
When I put  spml_ikrit_np = 1000 (more than 4) and run task on 4 (2,1) nodes, I 
get an:
in out file: 
No available spml components were found!
This means that there are no components of this type installed on your
system or all the components reported that they could not be used.
This is a fatal error; your SHMEM process is likely to abort. Check the
output of the "ompi_info" command and ensure that components of this
type are available on your system. You may also wish to check the
value of the "component_path" MCA parameter and ensure that it has at
least one directory that contains valid MCA components
in err file:
[node1-128-31:05405] SPML ikrit cannot be selected
Regards,
Timur

Re: [OMPI users] [warn] Epoll ADD(1) on fd 0 failed

[Timur Ismagilov]

 Sometimes,  after termination of the program, launched with the command 
"sbatch ... -o myprogram.out .",  no file "myprogram.out"  is being 
produced. Could this be due to the above mentioned problem?


Thu, 5 Jun 2014 07:45:01 -0700 от Ralph Castain :
>FWIW: support for the --resv-ports option was deprecated and removed on the 
>OMPI side a long time ago.
>
>I'm not familiar enough with "oshrun" to know if it is doing anything unusual 
>- I believe it is just a renaming of our usual "mpirun". I suspect this is 
>some interaction with sbatch, but I'll take a look. I haven't see that 
>warning. Mike indicated he thought it is due to both slurm and OMPI trying to 
>control stdin/stdout, in which case it shouldn't be happening but you can 
>safely ignore it
>
>
>On Jun 5, 2014, at 3:04 AM, Timur Ismagilov < tismagi...@mail.ru > wrote:
>>I use cmd line
>>$sbatch -p test --exclusive -N 2 -o hello_oshmem.out -e hello_oshmem.err 
>>shrun_mxm3.0 ./hello_oshmem
>>
>>where script shrun_mxm3.0:
>>$cat shrun_mxm3.0
>>  #!/bin/sh
>>  #srun --resv-ports "$@"
>>  #exit $?
>>  [ x"$TMPDIR" == x"" ] && TMPDIR=/tmp
>>  HOSTFILE=${TMPDIR}/hostfile.${SLURM_JOB_ID}
>>  srun hostname -s|sort|uniq -c|awk '{print $2" slots="$1}' > $HOSTFILE || { 
>>rm -f $HOSTFILE; exit 255; }
>>  
>>LD_PRELOAD=/mnt/data/users/dm2/vol3/semenov/_scratch/mxm/mxm-3.0/lib/libmxm.so
>> oshrun -x LD_PRELOAD -x MXM_SHM_KCOPY_MODE=off --hostfile $HOSTFILE "$@"
>>
>>  rc=$?
>>  rm -f $HOSTFILE
>>  exit $rc
>>I configured openmpi using
>>./configure CC=icc CXX=icpc F77=ifort FC=ifort 
>>--prefix=/mnt/data/users/dm2/vol3/semenov/_scratch/openmpi-1.8.1_mxm-3.0 
>>--with-mxm=/mnt/data/users/dm2/vol3/semenov/_scratch/mxm/mxm-3.0/ --with-     
>>         slurm --with-platform=contrib/platform/mellanox/optimized
>>
>>
>>Fri, 30 May 2014 07:09:54 -0700 от Ralph Castain < r...@open-mpi.org >:
>>>Can you pass along the cmd line that generated that output, and how OMPI was 
>>>configured?
>>>
>>>On May 30, 2014, at 5:11 AM, Тимур Исмагилов < tismagi...@mail.ru > wrote:
Hello!
I am using Open MPI v1.8.1 and slurm 2.5.6.
I got this messages when i try to run example (hello_oshmem.cpp) program:
[warn] Epoll ADD(1) on fd 0 failed. Old events were 0; read change was 1 
(add); write change was 0 (none): Operation not permitted
[warn] Epoll ADD(4) on fd 1 failed. Old events were 0; read change was 0 
(none); write change was 1 (add): Operation not permitted
Hello, world, I am 0 of 2
Hello, world, I am 1 of 2
What does this warnings mean?
I lunch this job using sbatch and mpirun with hostfile (got it from :  
$srun hostname -s|sort|uniq -c|awk '{print $2" slots="$1}' > $HOSTFILE)
Regards,
Timur
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>

>___
>users mailing list
>us...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/users