from:"Devendar Bureddy"

Re: [OMPI devel] [OMPI users] simple mpi hello world segfaults when coll ml not disabled

2015-06-29 Thread Devendar Bureddy

Fixed this issue in HCOLL by renaming conflicting symbols.  Repro case is 
working fine after this.

also explored –Bsymbolic linker option, but it seems not safe to do.

-Devendar

From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Thursday, June 25, 2015 9:31 PM
To: Open MPI Developers
Subject: Re: [OMPI devel] [OMPI users] simple mpi hello world segfaults when 
coll ml not disabled

Crud - thanks Paul! Mellanox is working on a fix (renaming the symbols in their 
proprietary library so they don't conflict). If they can release that soon, I'm 
hoping to avoid having to release a quick 1.8.7 to fix the problem from inside 
OMPI (i.e., removing one of the conflicting plugins).

On Thu, Jun 25, 2015 at 8:31 PM, Paul Hargrove 
> wrote:

On Thu, Jun 25, 2015 at 5:05 PM, Paul Hargrove 
> wrote:

On Thu, Jun 25, 2015 at 4:59 PM, Gilles Gouaillardet 
> wrote:
In this case, mca_coll_hcoll module is linked with the proprietary libhcoll.so.
the ml symbols are defined in both mca_coll_ml.so and libhcoll.so
i am not sure (i blame my poor understanding of linkers) this is an error if
Open MPI is configure'd with --disable-dlopen

I will run the test now on a system w/ Mellanox's libhcoll and report what I 
find.

Gilles,

I had originally missed the fact that the conflicts were between Open MPI code 
and "vendor code".
Otherwise I don't think I'd have put forward the --disable-dlopen suggestion.
However, as promised I tried the experiment.

I find that having both coll:ml and coll:hcoll in a --disable-dlopen build this 
does NOT result in failures linking libmpi nor in linking an MPI application.  
So, having Jenkins (for instance) testing in this way would not have exposed 
this problem.

To sure I was testing what I thought I was:

I did confirm that I get a SEGV running hello_c (from the examples subdir) 
unless I use "-mca coll ^hcoll".

I tried using "-mca coll ^ml" but still get a SEGV that appears to show 
coll:hcoll invoking functions in coll_ml_module.c, just as I do with no mca 
options at all.

Note I did this all with the released 1.8.6 tarball.

-Paul

--
Paul H. Hargrove  
phhargr...@lbl.gov
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department   Tel: 
+1-510-495-2352
Lawrence Berkeley National Laboratory Fax: 
+1-510-486-6900

___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2015/06/17542.php

[OMPI devel] mpich test ini file

2015-06-24 Thread Devendar Bureddy

Jeff,

Attached mlnx internal mpich test ini file. please note that we don't run all 
the tests in the mpich tests - we remove some of them during the build stage in 
the mpich_tests.ini file

-Devendar


mpich_tests.ini
Description: mpich_tests.ini

Re: [OMPI devel] binding output error

2015-04-21 Thread Devendar Bureddy

I agree.   

-Original Message-
From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Jeff Squyres 
(jsquyres)
Sent: Tuesday, April 21, 2015 7:17 AM
To: Open MPI Developers List
Subject: Re: [OMPI devel] binding output error

+1

Devendar, you seem to be reporting a different issue than Elena...?  FWIW: Open 
MPI has always used logical CPU numbering.  As far as I can tell from your 
output, it looks like Open MPI did the Right Thing with your examples.

Elena's example seemed to show conflicting cpu numbering -- where OMPI said it 
would bind a process and then where it actually bound it.  Ralph mentioned to 
me that he would look at this as soon as he could; he thinks it might just be 
an error in the printf output (and that the binding is actually occurring in 
the right location).



> On Apr 20, 2015, at 9:48 PM, tmish...@jcity.maeda.co.jp wrote:
> 
> Hi Devendar,
> 
> As far as I know, the report-bindings option shows the logical cpu 
> order. On the other hand, you are talking about physical one, I guess.
> 
> Regards,
> Tetsuya Mishima
> 
> 2015/04/21 9:04:37、"devel"さんは「Re: [OMPI devel] binding output
> error」で書きました
>> HT is not enabled.  All node are same topo . This is reproducible 
>> even on
> single node.
>> 
>> 
>> 
>> I ran osu latency to see if it is really is mapped to other socket or 
>> not
> with –map-by socket.  It looks likes mapping is correct as per latency 
> test.
>> 
>> 
>> 
>> $mpirun -np 2 -report-bindings -map-by
> socket  
> /hpc/local/benchmarks/hpc-stack-icc/install/ompi-mellanox-v1.8/tests/o
> su-micro-benchmarks-4.4.1/osu_latency
> 
>> 
>> [clx-orion-001:10084] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> [B/././././././././././././.][./././././././././././././.]
>> 
>> [clx-orion-001:10084] MCW rank 1 bound to socket 1[core 14[hwt 0]]:
> [./././././././././././././.][B/././././././././././././.]
>> 
>> # OSU MPI Latency Test v4.4.1
>> 
>> # Size  Latency (us)
>> 
>> 0   0.50
>> 
>> 1   0.50
>> 
>> 2   0.50
>> 
>> 4   0.49
>> 
>> 
>> 
>> 
>> 
>> $mpirun -np 2 -report-bindings -cpu-set
> 1,7 
> /hpc/local/benchmarks/hpc-stack-icc/install/ompi-mellanox-v1.8/tests/o
> su-micro-benchmarks-4.4.1/osu_latency
> 
>> 
>> [clx-orion-001:10155] MCW rank 0 bound to socket 0[core 1[hwt 0]]:
> [./B/./././././././././././.][./././././././././././././.]
>> 
>> [clx-orion-001:10155] MCW rank 1 bound to socket 0[core 7[hwt 0]]:
> [./././././././B/./././././.][./././././././././././././.]
>> 
>> # OSU MPI Latency Test v4.4.1
>> 
>> # Size  Latency (us)
>> 
>> 0   0.23
>> 
>> 1   0.24
>> 
>> 2   0.23
>> 
>> 4   0.22
>> 
>> 8   0.23
>> 
>> 
>> 
>> Both hwloc and /proc/cpuinfo indicates following cpu numbering
>> 
>> socket 0 cpus: 0 1 2 3 4 5 6 14 15 16 17 18 19 20
>> 
>> socket 1 cpus: 7 8 9 10 11 12 13 21 22 23 24 25 26 27
>> 
>> 
>> 
>> $hwloc-info -f
>> 
>> Machine (256GB)
>> 
>>   NUMANode L#0 (P#0 128GB) + Socket L#0 + L3 L#0 (35MB)
>> 
>> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
>> 
>> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#1)
>> 
>> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2)
>> 
>> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3)
>> 
>> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 + PU L#4 (P#4)
>> 
>> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5)
>> 
>> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 + PU L#6 (P#6)
>> 
>> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 + PU L#7 (P#14)
>> 
>> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 + PU L#8 (P#15)
>> 
>> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 + PU L#9 (P#16)
>> 
>> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 + PU L#10 (P#17)
>> 
>> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 + PU L#11 (P#18)
>> 
>> L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 + PU L#12 (P#19)
>> 
>> L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 + PU L#13 (P#20)
>> 
>>   NUMANode L#1 (P#1 128GB) + Socket L#1 + L3 L#1 (35MB)
>> 
>> L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 + PU L#14 (P#7)
>> 
>> L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 + PU L#15 (P#8)
>> 
>> L2 L#16 (256KB) + L1 L#16 (32KB) + Core L#16 + PU L#16 (P#9)
>> 
>> L2 L#17 (256KB) + L1 L#17 (32KB) + Core L#17 + PU L#17 (P#10)
>> 
>> L2 L#18 (256KB) + L1 L#18 (32KB) + Core L#18 + PU L#18 (P#11)
>> 
>> L2 L#19 (256KB) + L1 L#19 (32KB) + Core L#19 + PU L#19 (P#12)
>> 
>> L2 L#20 (256KB) + L1 L#20 (32KB) + Core L#20 + PU L#20 (P#13)
>> 
>> L2 L#21 (256KB) + L1 L#21 (32KB) + Core L#21 + PU L#21 (P#21)
>> 
>> L2 L#22 (256KB) + L1 L#22 (32KB) + Core L#22 + PU L#22 (P#22)
>> 
>> L2 L#23 (256KB) + L1 L#23 (32KB) + Core L#23 + PU L#23 (P#23)
>> 
>> L2 L#24 (256KB) + L1 L#24 (32KB) + Core L#24 + PU L#24 (P#24)
>> 
>> L2 L#25 (256KB) + L1 L#25

Re: [OMPI devel] binding output error

2015-04-20 Thread Devendar Bureddy

HT is not enabled.  All node are same topo . This is reproducible even on 
single node.

I ran osu latency to see if it is really is mapped to other socket or not with 
–map-by socket.  It looks likes mapping is correct as per latency test.

$mpirun -np 2 -report-bindings -map-by socket  
/hpc/local/benchmarks/hpc-stack-icc/install/ompi-mellanox-v1.8/tests/osu-micro-benchmarks-4.4.1/osu_latency
[clx-orion-001:10084] MCW rank 0 bound to socket 0[core 0[hwt 0]]: 
[B/././././././././././././.][./././././././././././././.]
[clx-orion-001:10084] MCW rank 1 bound to socket 1[core 14[hwt 0]]: 
[./././././././././././././.][B/././././././././././././.]
# OSU MPI Latency Test v4.4.1
# Size  Latency (us)
0   0.50
1   0.50
2   0.50
4   0.49


$mpirun -np 2 -report-bindings -cpu-set 1,7 
/hpc/local/benchmarks/hpc-stack-icc/install/ompi-mellanox-v1.8/tests/osu-micro-benchmarks-4.4.1/osu_latency
[clx-orion-001:10155] MCW rank 0 bound to socket 0[core 1[hwt 0]]: 
[./B/./././././././././././.][./././././././././././././.]
[clx-orion-001:10155] MCW rank 1 bound to socket 0[core 7[hwt 0]]: 
[./././././././B/./././././.][./././././././././././././.]
# OSU MPI Latency Test v4.4.1
# Size  Latency (us)
0   0.23
1   0.24
2   0.23
4   0.22
8   0.23

Both hwloc and /proc/cpuinfo indicates following cpu numbering
socket 0 cpus: 0 1 2 3 4 5 6 14 15 16 17 18 19 20
socket 1 cpus: 7 8 9 10 11 12 13 21 22 23 24 25 26 27

$hwloc-info -f
Machine (256GB)
  NUMANode L#0 (P#0 128GB) + Socket L#0 + L3 L#0 (35MB)
L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#1)
L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2)
L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3)
L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 + PU L#4 (P#4)
L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5)
L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 + PU L#6 (P#6)
L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 + PU L#7 (P#14)
L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 + PU L#8 (P#15)
L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 + PU L#9 (P#16)
L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 + PU L#10 (P#17)
L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 + PU L#11 (P#18)
L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 + PU L#12 (P#19)
L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 + PU L#13 (P#20)
  NUMANode L#1 (P#1 128GB) + Socket L#1 + L3 L#1 (35MB)
L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 + PU L#14 (P#7)
L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 + PU L#15 (P#8)
L2 L#16 (256KB) + L1 L#16 (32KB) + Core L#16 + PU L#16 (P#9)
L2 L#17 (256KB) + L1 L#17 (32KB) + Core L#17 + PU L#17 (P#10)
L2 L#18 (256KB) + L1 L#18 (32KB) + Core L#18 + PU L#18 (P#11)
L2 L#19 (256KB) + L1 L#19 (32KB) + Core L#19 + PU L#19 (P#12)
L2 L#20 (256KB) + L1 L#20 (32KB) + Core L#20 + PU L#20 (P#13)
L2 L#21 (256KB) + L1 L#21 (32KB) + Core L#21 + PU L#21 (P#21)
L2 L#22 (256KB) + L1 L#22 (32KB) + Core L#22 + PU L#22 (P#22)
L2 L#23 (256KB) + L1 L#23 (32KB) + Core L#23 + PU L#23 (P#23)
L2 L#24 (256KB) + L1 L#24 (32KB) + Core L#24 + PU L#24 (P#24)
L2 L#25 (256KB) + L1 L#25 (32KB) + Core L#25 + PU L#25 (P#25)
L2 L#26 (256KB) + L1 L#26 (32KB) + Core L#26 + PU L#26 (P#26)
L2 L#27 (256KB) + L1 L#27 (32KB) + Core L#27 + PU L#27 (P#27)


So, Is --reporting-binding shows one more level of logical CPU numbering?


-Devendar


From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Monday, April 20, 2015 3:52 PM
To: Open MPI Developers
Subject: Re: [OMPI devel] binding output error

Also, was this with HT's enabled? I'm wondering if the print code is 
incorrectly computing the core because it isn't correctly accounting for HT 
cpus.


On Mon, Apr 20, 2015 at 3:49 PM, Jeff Squyres (jsquyres) 
> wrote:
Ralph's the authority on this one, but just to be sure: are all nodes the same 
topology? E.g., does adding "--hetero-nodes" to the mpirun command line fix the 
problem?


> On Apr 20, 2015, at 9:29 AM, Elena Elkina 
> > wrote:
>
> Hi guys,
>
> I faced with an issue on our cluster related to mapping & binding policies on 
> 1.8.5.
>
> The matter is that --report-bindings output doesn't correspond to the locale. 
> It looks like there is a mistake on the output itself, because it just puts 
> serial core number while that core can be on another socket. For example,
>
> mpirun -np 2 --display-devel-map --report-bindings --map-by socket hostname
>  Data for JOB [43064,1] offset 0
>
>  Mapper requested: NULL  Last mapper: round_robin  Mapping policy: BYSOCKET  
> Ranking policy: SOCKET
>  Binding policy: CORE  Cpu set: NULL  PPR: NULL

Re: [OMPI devel] mlx4 QP operation err

2015-01-28 Thread Devendar Bureddy

are you able to reproduce this error with ib verbs bw test?  I hope,  you are 
running on lossless Ethernet fabric setup and selecting correct VLAN .

-Devendar

From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Dave Turner
Sent: Wednesday, January 28, 2015 4:31 PM
To: de...@open-mpi.org
Subject: [OMPI devel] mlx4 QP operation err

I'm testing RoCE on 40 Gbps Mellanox ethernet cards and am getting a
mlx4 QP operation error every time it gets to testing 132 kB packets.  These
are aggregate tests in that 16 cores on one host are doing bi-directional
ping-pongs to 16 cores on another host across the Mellanox cards.

  I've found some old references to similar mlx4 errors dating back to
2009 that lead me to believe this may be a firmware error.  I believe we're
running the most up to date version of the firmware.

 Could someone comment on whether these are firmware issues, and
if so how to report them to Mellanox?  I've attached some files with more
detailed information on this problem.

 Dave Turner

--
Work: davetur...@ksu.edu (785) 532-7791
 118 Nichols Hall, Manhattan KS  66502
Home:drdavetur...@gmail.com
  cell: (785) 770-5929

Re: [OMPI devel] openib receive queue settings

2014-12-26 Thread Devendar Bureddy

Looks like a bug here. It is considering  source of var is  
MCA_BASE_VAR_SOURCE_FILE  for both variables reading from  mca-param.conf andv  
INI file(opal/mca/btl/openib/mca-btl-openib-device-params.ini).  
But, in function mca_btl_openib_tune_endpoint(),  where this error triggered is 
only checking values from INI file (opal_btl_openib_ini_query()) for receive 
queue value correctness.  

Not sure, if it should consider  MCA_BASE_VAR_SOURCE_ENV  as a source for 
variables read from .openmpi/mca-param.conf file.
Nathan - any idea?

-Devendar

-Original Message-
From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Edgar Gabriel
Sent: Friday, December 26, 2014 10:35 AM
To: Open MPI Developers
Subject: [OMPI devel] openib receive queue settings

I still see an issue with the openib receive queues settings. 
Interestingly, it seems to work if I pass the setting with the mpirun command, 
e.g.

mpirun  --mca btl_openib_receive_queues
S,12288,128,64,32:S,65536,128,64,32 --npernode 1 -np 2 ./lat

but if I add it to the ${HOME}/.openmpi/mca-param.conf file, e.g.

---snip---
  cat  ~/.openmpi/mca-params.conf
...
btl_openib_receive_queues = S,12288,128,64,32:S,65536,128,64,32
...
--snip---

I receive the following error message:
gabriel@crill:~> mpirun  --npernode 1 -np 2 ./lat
--
The Open MPI receive queue configuration for the OpenFabrics devices on two 
nodes are incompatible, meaning that MPI processes on two specific nodes were 
unable to communicate with each other.  This generally happens when you are 
using OpenFabrics devices from different vendors on the same network.  You 
should be able to use the mca_btl_openib_receive_queues MCA parameter to set a 
uniform receive queue configuration for all the devices in the MPI job, and 
therefore be able to run successfully.

   Local host:   crill-003
   Local adapter:mlx4_0 (vendor 0x2c9, part ID 26418)
   Local queues: S,12288,128,64,32:S,65536,128,64,32

   Remote host:  crill-004
   Remote adapter:   (vendor 0x2c9, part ID 26418)
   Remote queues: 
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64
--

Does anybody have an idea what I should be looking for to fix this? I can 
definitely confirm, that the home file system is mounted on all nodes correctly 
(i.e. all processes can access the same mca-params.conf file), and they have 
the identical IB hardware (in contrary to what the error message says).


Thanks
Edgar

--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16731.php

Re: [OMPI devel] --enable-visibility ( OPAL_C_HAVE_VISIBILITY) behavior in trunk

2014-09-29 Thread Devendar Bureddy

I see behavioral difference between 1.8.x and trunk for OPAL_C_HAVE_VISIBILITY 
definition on same build environment.  is this expected?

-Devendar

-Original Message-
From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Jeff Squyres 
(jsquyres)
Sent: Monday, September 29, 2014 4:25 PM
To: Open MPI Developers List
Subject: Re: [OMPI devel] --enable-visibility ( OPAL_C_HAVE_VISIBILITY) 
behavior in trunk

I can't quite parse what you are saying -- do you have a specific question?


On Sep 29, 2014, at 7:18 PM, Devendar Bureddy <deven...@mellanox.com> wrote:

> This is supposed to be enable by default.  In trunk, I see that  
> OPAL_C_HAVE_VISIBILITY is defined to 0 by default.   1.8.x looks fine
>  
> Configure : ./configure -prefix=$PWD/install 
> --enable-mpirun-prefix-by-default --disable-mpi-fortran --disable-vt 
> --enable-debug --enable-oshmem --with-pmi GCC : gcc version 4.4.7 
> 20120313 (Red Hat 4.4.7-3) (GCC)
>  
> -Devendar
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/09/15936.php


--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/09/15937.php

[OMPI devel] --enable-visibility ( OPAL_C_HAVE_VISIBILITY) behavior in trunk

2014-09-29 Thread Devendar Bureddy

This is supposed to be enable by default.  In trunk, I see that  
OPAL_C_HAVE_VISIBILITY is defined to 0 by default.   1.8.x looks fine

Configure : ./configure -prefix=$PWD/install --enable-mpirun-prefix-by-default 
--disable-mpi-fortran --disable-vt --enable-debug --enable-oshmem --with-pmi
GCC : gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)

-Devendar

Re: [OMPI devel] [OMPI users] simple mpi hello world segfaults when coll ml not disabled

[OMPI devel] mpich test ini file

Re: [OMPI devel] binding output error

Re: [OMPI devel] binding output error

Re: [OMPI devel] mlx4 QP operation err

Re: [OMPI devel] openib receive queue settings

Re: [OMPI devel] --enable-visibility ( OPAL_C_HAVE_VISIBILITY) behavior in trunk

[OMPI devel] --enable-visibility ( OPAL_C_HAVE_VISIBILITY) behavior in trunk

8 matches

Site Navigation

Mail list logo

Footer information