Hi Ralph,
The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache coherent)NUMA, which seems to be a little bit different from the traditional numa defined in openmpi. I notice that ccNUMA object is almost same as L3cache object. So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want to do. Therefore, "do not touch it" is one of the solution, I think ... Anyway, mixing up these two types of numa is the problem. Regards, Tetsuya Mishima > I can wait it'll be fixed in 1.7.5 or later, because putting "-bind-to > numa" > and "-map-by numa" at the same time works as a workaround. > > Thanks, > Tetsuya Mishima > > > Yeah, it will impact everything that uses hwloc topology maps, I fear. > > > > One side note: you'll need to add --hetero-nodes to your cmd line. If we > don't see that, we assume that all the node topologies are identical - > which clearly isn't true here. > > > > I'll try to resolve the hier inversion over the holiday - won't be for > 1.7.4, but hopefully for 1.7.5 > > > > Thanks > > Ralph > > > > On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: > > > > > > > > > > > I think it's normal for AMD opteron having 8/16 cores such as > > > magny cours or interlagos. Because it usually has 2 numa nodes > > > in a cpu(socket), numa-node can not include a socket. This type > > > of hierarchy would be natural. > > > > > > (node03 is Dell PowerEdge R815 and maybe quite common, I guess) > > > > > > By the way, I think this inversion should affect rmaps_lama mapping. > > > > > > Tetsuya Mishima > > > > > >> Ick - yeah, that would be a problem. I haven't seen that type of > > > hierarchical inversion before - is node03 a different type of chip? > > >> > > >> Might take awhile for me to adjust the code to handle hier > > > inversion... :-( > > >> > > >> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: > > >> > > >>> > > >>> > > >>> Hi Ralph, > > >>> > > >>> I found the reason. I attached the main part of output with 32 > > >>> core node(node03) and 8 core node(node05) at the bottom. > > >>> > > >>> From this information, socket of node03 includes numa-node. > > >>> On the other hand, numa-node of node05 includes socket. > > >>> The direction of object tree is opposite. > > >>> > > >>> Since "-map-by socket" may be assumed as default, > > >>> for node05, "-bind-to numa and -map-by socket" means > > >>> upward search. For node03, this should be downward. > > >>> > > >>> I guess that openmpi-1.7.4rc1 will always assume numa-node > > >>> includes socket. Is it right? Then, upward search is assumed > > >>> in orte_rmaps_base_compute_bindings even for node03 when I > > >>> put "-bind-to numa and -map-by socket" option. > > >>> > > >>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage > > >>> [node03.cluster:15508] mca:rmaps: compute bindings for job [38286,1] > > > with > > >>> policy NUMA > > >>> [node03.cluster:15508] mca:rmaps: bind upwards for job [38286,1] with > > >>> bindings NUMA > > >>> [node03.cluster:15508] [[38286,0],0] bind:upward target NUMANode type > > >>> Machine > > >>> > > >>> That's the reason of this trouble. Therefore, adding "-map-by core" > > > works. > > >>> (mapping pattern seems to be strange ...) > > >>> > > >>> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by core > > >>> -report-bindings myprog > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > > Cache > > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode type > > >>> NUMANode > > >>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt 0]], > > > socket > > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > > >>> cket 0[core 3[hwt 0]]: > > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt 0]], > > > socket > > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > > >>> cket 0[core 3[hwt 0]]: > > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt 0]], > > > socket > > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > > >>> cket 0[core 7[hwt 0]]: > > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt 0]], > > > socket > > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > > >>> cket 0[core 7[hwt 0]]: > > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt 0]], > > > socket > > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > > >>> cket 0[core 7[hwt 0]]: > > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt 0]], > > > socket > > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > > >>> cket 0[core 7[hwt 0]]: > > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt 0]], > > > socket > > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > > >>> cket 0[core 3[hwt 0]]: > > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > > >>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt 0]], > > > socket > > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > > >>> cket 0[core 3[hwt 0]]: > > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > > >>> Hello world from process 6 of 8 > > >>> Hello world from process 5 of 8 > > >>> Hello world from process 0 of 8 > > >>> Hello world from process 7 of 8 > > >>> Hello world from process 3 of 8 > > >>> Hello world from process 4 of 8 > > >>> Hello world from process 2 of 8 > > >>> Hello world from process 1 of 8 > > >>> > > >>> Regards, > > >>> Tetsuya Mishima > > >>> > > >>> [node03.cluster:15508] Type: Machine Number of child objects: 4 > > >>> Name=NULL > > >>> total=132358820KB > > >>> Backend=Linux > > >>> OSName=Linux > > >>> OSRelease=2.6.18-308.16.1.el5 > > >>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > > >>> Architecture=x86_64 > > >>> Cpuset: 0xffffffff > > >>> Online: 0xffffffff > > >>> Allowed: 0xffffffff > > >>> Bind CPU proc: TRUE > > >>> Bind CPU thread: TRUE > > >>> Bind MEM proc: FALSE > > >>> Bind MEM thread: TRUE > > >>> Type: Socket Number of child objects: 2 > > >>> Name=NULL > > >>> total=33071780KB > > >>> CPUModel="AMD Opteron(tm) Processor 6136" > > >>> Cpuset: 0x000000ff > > >>> Online: 0x000000ff > > >>> Allowed: 0x000000ff > > >>> Type: NUMANode Number of child objects: 1 > > >>> > > >>> > > >>> [node05.cluster:21750] Type: Machine Number of child objects: 2 > > >>> Name=NULL > > >>> total=33080072KB > > >>> Backend=Linux > > >>> OSName=Linux > > >>> OSRelease=2.6.18-308.16.1.el5 > > >>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > > >>> Architecture=x86_64 > > >>> Cpuset: 0x000000ff > > >>> Online: 0x000000ff > > >>> Allowed: 0x000000ff > > >>> Bind CPU proc: TRUE > > >>> Bind CPU thread: TRUE > > >>> Bind MEM proc: FALSE > > >>> Bind MEM thread: TRUE > > >>> Type: NUMANode Number of child objects: 1 > > >>> Name=NULL > > >>> local=16532232KB > > >>> total=16532232KB > > >>> Cpuset: 0x0000000f > > >>> Online: 0x0000000f > > >>> Allowed: 0x0000000f > > >>> Type: Socket Number of child objects: 1 > > >>> > > >>> > > >>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca ess_base_verbose > 5" > > > to > > >>> your cmd line and let's see what it thinks it found. > > >>>> > > >>>> > > >>>> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp wrote: > > >>>> > > >>>>> > > >>>>> > > >>>>> Hi, I report one more problem with openmpi-1.7.4rc1, > > >>>>> which is more serious. > > >>>>> > > >>>>> For our 32 core nodes(AMD magny cours based) which has > > >>>>> 8 numa-nodes, "-bind-to numa" does not work. Without > > >>>>> this option, it works. For your infomation, at the > > >>>>> bottom of this mail, I added the lstopo information > > >>>>> of the node. > > >>>>> > > >>>>> Regards, > > >>>>> Tetsuya Mishima > > >>>>> > > >>>>> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32 > > >>>>> qsub: waiting for job 8352.manage.cluster to start > > >>>>> qsub: job 8352.manage.cluster ready > > >>>>> > > >>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings -bind-to numa > > >>> myprog > > >>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target NUMANode > type > > >>>>> Machine > > >>>>> > > >>> > > > > -------------------------------------------------------------------------- > > >>>>> A request was made to bind to NUMA, but an appropriate target could > > > not > > >>>>> be found on node node03. > > >>>>> > > >>> > > > > -------------------------------------------------------------------------- > > >>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ > > >>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog > > >>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8[hwt 0]]: > > >>>>> [./././././././.][B/././././././.][./././././././.][ > > >>>>> ./././././././.] > > >>>>> [node03.cluster:15282] MCW rank 3 bound to socket 1[core 9[hwt 0]]: > > >>>>> [./././././././.][./B/./././././.][./././././././.][ > > >>>>> ./././././././.] > > >>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16[hwt > 0]]: > > >>>>> [./././././././.][./././././././.][B/././././././.] > > >>>>> [./././././././.] > > >>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17[hwt > 0]]: > > >>>>> [./././././././.][./././././././.][./B/./././././.] > > >>>>> [./././././././.] > > >>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24[hwt > 0]]: > > >>>>> [./././././././.][./././././././.][./././././././.] > > >>>>> [B/././././././.] > > >>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25[hwt > 0]]: > > >>>>> [./././././././.][./././././././.][./././././././.] > > >>>>> [./B/./././././.] > > >>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > > >>>>> [B/././././././.][./././././././.][./././././././.][ > > >>>>> ./././././././.] > > >>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1[hwt 0]]: > > >>>>> [./B/./././././.][./././././././.][./././././././.][ > > >>>>> ./././././././.] > > >>>>> Hello world from process 2 of 8 > > >>>>> Hello world from process 5 of 8 > > >>>>> Hello world from process 4 of 8 > > >>>>> Hello world from process 3 of 8 > > >>>>> Hello world from process 1 of 8 > > >>>>> Hello world from process 7 of 8 > > >>>>> Hello world from process 6 of 8 > > >>>>> Hello world from process 0 of 8 > > >>>>> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics > > >>>>> Machine (126GB) > > >>>>> Socket L#0 (32GB) > > >>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) > > >>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU > > >>> L#0 > > >>>>> (P#0) > > >>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU > > >>> L#1 > > >>>>> (P#1) > > >>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU > > >>> L#2 > > >>>>> (P#2) > > >>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU > > >>> L#3 > > >>>>> (P#3) > > >>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) > > >>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU > > >>> L#4 > > >>>>> (P#4) > > >>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU > > >>> L#5 > > >>>>> (P#5) > > >>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU > > >>> L#6 > > >>>>> (P#6) > > >>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU > > >>> L#7 > > >>>>> (P#7) > > >>>>> Socket L#1 (32GB) > > >>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) > > >>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU > > >>> L#8 > > >>>>> (P#8) > > >>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU > > >>> L#9 > > >>>>> (P#9) > > >>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 > + > > >>> PU > > >>>>> L#10 (P#10) > > >>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 > + > > >>> PU > > >>>>> L#11 (P#11) > > >>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) > > >>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 > + > > >>> PU > > >>>>> L#12 (P#12) > > >>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 > + > > >>> PU > > >>>>> L#13 (P#13) > > >>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 > + > > >>> PU > > >>>>> L#14 (P#14) > > >>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 > + > > >>> PU > > >>>>> L#15 (P#15) > > >>>>> Socket L#2 (32GB) > > >>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) > > >>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core L#16 > + > > >>> PU > > >>>>> L#16 (P#16) > > >>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core L#17 > + > > >>> PU > > >>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + L1i L#18 (64KB) + Core L#18 > + > > >>> PU > > >>>>> L#18 (P#18) > > >>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core L#19 > + > > >>> PU > > >>>>> L#19 (P#19) > > >>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) > > >>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core L#20 > + > > >>> PU > > >>>>> L#20 (P#20) > > >>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core L#21 > + > > >>> PU > > >>>>> L#21 (P#21) > > >>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core L#22 > + > > >>> PU > > >>>>> L#22 (P#22) > > >>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core L#23 > + > > >>> PU > > >>>>> L#23 (P#23) > > >>>>> Socket L#3 (32GB) > > >>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) > > >>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core L#24 > + > > >>> PU > > >>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 > (64KB) + Core L#25 + > > >>> PU > > >>>>> L#25 (P#25) > > >>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core L#26 > + > > >>> PU > > >>>>> L#26 (P#26) > > >>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core L#27 > + > > >>> PU > > >>>>> L#27 (P#27) > > >>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) > > >>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core L#28 > + > > >>> PU > > >>>>> L#28 (P#28) > > >>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core L#29 > + > > >>> PU > > >>>>> L#29 (P#29) > > >>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core L#30 > + > > >>> PU > > >>>>> L#30 (P#30) > > >>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core L#31 > + > > >>> PU > > >>>>> L#31 (P#31) > > >>>>> HostBridge L#0 > > >>>>> PCIBridge > > >>>>> PCI 14e4:1639 > > >>>>> Net L#0 "eth0" > > >>>>> PCI 14e4:1639 > > >>>>> Net L#1 "eth1" > > >>>>> PCIBridge > > >>>>> PCI 14e4:1639 > > >>>>> Net L#2 "eth2" > > >>>>> PCI 14e4:1639 > > >>>>> Net L#3 "eth3" > > >>>>> PCIBridge > > >>>>> PCIBridge > > >>>>> PCIBridge > > >>>>> PCI 1000:0072 > > >>>>> Block L#4 "sdb" > > >>>>> Block L#5 "sda" > > >>>>> PCI 1002:4390 > > >>>>> Block L#6 "sr0" > > >>>>> PCIBridge > > >>>>> PCI 102b:0532 > > >>>>> HostBridge L#7 > > >>>>> PCIBridge > > >>>>> PCI 15b3:6274 > > >>>>> Net L#7 "ib0" > > >>>>> OpenFabrics L#8 "mthca0" > > >>>>> > > >>>>> _______________________________________________ > > >>>>> users mailing list > > >>>>> us...@open-mpi.org > > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > > >>>> > > >>>> _______________________________________________ > > >>>> users mailing list > > >>>> us...@open-mpi.org>> > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > >>> > > >>> _______________________________________________ > > >>> users mailing list > > >>> us...@open-mpi.org > > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > > >> > > >> _______________________________________________ > > >> users mailing list > > >> us...@open-mpi.org > > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
