Re: [OMPI users] Error building openmpi on Raspberry pi 2

[Jeff Layton]

I could never get OpenMPI < 2.x to build on a Pi 2. I ended up using the binary 
from the repos. Pi 3 is a different matter - I got that to build after a little 
experimentation :)
Jeff
 

On Wednesday, September 27, 2017 8:03 AM, Nathan Hjelm  
wrote:
 

 Open MPI does not officially support ARM in the v2.1 series. Can you download 
a nightly tarball from https://www.open-mpi.org/nightly/master/ and see if it 
works for you?

-Nathan

> On Sep 26, 2017, at 7:32 PM, Faraz Hussain  wrote:
> 
> I am receiving the make errors below on my pi 2:
> 
> pi@pi001:~/openmpi-2.1.1 $ uname -a
> Linux pi001 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
> 
> pi@pi001:~/openmpi-2.1.1 $ make -j 4
> .
> .
> .
> .
> make[2]: Entering directory '/home/pi/openmpi-2.1.1/opal/asm'
>  CPPAS    atomic-asm.lo
> atomic-asm.S: Assembler messages:
> atomic-asm.S:7: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:15: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:23: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:55: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:70: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:86: Error: selected processor does not support ARM mode `ldrexd 
> r4,r5,[r0]'
> atomic-asm.S:91: Error: selected processor does not support ARM mode `strexd 
> r1,r6,r7,[r0]'
> atomic-asm.S:107: Error: selected processor does not support ARM mode `ldrexd 
> r4,r5,[r0]'
> atomic-asm.S:112: Error: selected processor does not support ARM mode `strexd 
> r1,r6,r7,[r0]'
> atomic-asm.S:115: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:130: Error: selected processor does not support ARM mode `ldrexd 
> r4,r5,[r0]'
> atomic-asm.S:135: Error: selected processor does not support ARM mode `dmb'
> atomic-asm.S:136: Error: selected processor does not support ARM mode `strexd 
> r1,r6,r7,[r0]'
> Makefile:1743: recipe for target 'atomic-asm.lo' failed
> make[2]: *** [atomic-asm.lo] Error 1
> make[2]: Leaving directory '/home/pi/openmpi-2.1.1/opal/asm'
> Makefile:2307: recipe for target 'all-recursive' failed
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory '/home/pi/openmpi-2.1.1/opal'
> Makefile:1806: recipe for target 'all-recursive' failed
> make: *** [all-recursive] Error 1
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users


   ___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] [Open MPI Announce] Follow-up to Open MPI SC'16 BOF

[Jeff Layton]

We're going to make Open MPI Great Again

(don't shoot me please)



Thanks to all who provided their opinion.

Based on the results, the Open MPI developer community will work on 
v3.0.0 after the v2.1.x series.



On Nov 28, 2016, at 5:18 PM, Jeff Squyres (jsquyres) 
> wrote:


If you have an opinion on the v2.2.x-vs-v3.x question, please submit 
your vote by COB this upcoming Friday, 2 Dec, 2016:


https://www.open-mpi.org/sc16/

Thanks!


On Nov 22, 2016, at 4:32 PM, Pritchard Jr., Howard > wrote:


Hello Folks,

This is a followup to the question posed at the SC’16 Open MPI BOF: 
 Would the community
prefer to have a v2.2.x limited feature but backwards compatible 
release sometime
in 2017, or would the community prefer a v3.x (not backwards 
compatible but potentially

more features) sometime in late 2017 to early 2018?

BOF attendees expressed an interest in having a list of features 
that might make it in to v2.2.x
and ones that the Open MPI developers think would be too hard to 
back port from

the development branch (master) to a v2.2.x release stream.

Here are the requested lists:

Features that we anticipate we could port to a v2.2.x release
• Improved collective performance (a new “tuned” module)
• Enable Linux CMA shared memory support by default
• PMIx 3.0 (If new functionality were to be used in this release of 
Open MPI)
Features that we anticipate would be too difficult to port to a 
v2.2.x release

• Revamped CUDA support
• MPI_ALLOC_MEM integration with memkind
• OpenMP affinity/placement integration
• THREAD_MULTIPLE improvements to MTLs (not so clear on the level of 
difficult for this one)
You can register your opinion on whether to go with a v2.2.x release 
next year
or to go from v2.1.x to v3.x in late 2017 or early 2018 at the link 
below:


https://www.open-mpi.org/sc16/



Thanks very much,

Howard

--
Howard Pritchard
HPC-DES
Los Alamos National Laboratory

___
announce mailing list
annou...@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/announce



--
Jeff Squyres
jsquy...@cisco.com 
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/


___
announce mailing list
annou...@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/announce



--
Jeff Squyres
jsquy...@cisco.com 
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/




___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] Problem getting job to start

[Jeff Layton]

Good afternoon sports fans!
I'm trying to run the ft code of NPB, class D, over 128 processors. I built the 
code with gfortran 4.4.7 (CentOS 6 platform) and Open MPI 1.8.1.  I'm using 
openlava as the resource manager. The error output is the following:

[ec2-user@ip-172-31-42-106 bin]$ more runit_ft_128_D.err
Warning: Permanently added the RSA host key for IP address '172.31.41.229' to 
the
 list of known hosts.
bash: orted: command not found
bash: orted: command not found


I realize this is something of a catch-all for startup problems but I'm not 
exactly sure where to start debugging. Any thoughts?
Thanks!
Jeff

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

ork/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include
CXX = /usr/bin/g++-4.8
CXXCPP = /usr/bin/g++-4.8 -E
CXXCPPFLAGS = -I$(top_srcdir)
CXXDEPMODE = depmode=gcc3
CXXFLAGS = -O3 -DNDEBUG -finline-functions -pthread
CYGPATH_W = echo
DEBUGGER_CFLAGS = -g
DEFS = -DHAVE_CONFIG_H
DEPDIR = .deps
DLLTOOL = false
DSYMUTIL =
DUMPBIN =
ECHO_C =
ECHO_N = -n
ECHO_T =
EGREP = /bin/grep -E
EXEEXT =
FC = /usr/bin/gfortran-4.8
FCFLAGS =
FCFLAGS_f =
FCFLAGS_f90 =
FFLAGS =
FGREP = /bin/grep -F
GREP = /bin/grep


Thanks!

Jeff



Jeff,

Raspbian is built for armv6
(unlike linaro which is built for armv7)

that being said, you are running on an armv7 cpu, so i am not sure 
"cross-compiling"
is appropriate. (e.g. if you run a 32 bits gcc binary on an x86_64 os 
to build a x86_64 binary,

is this called cross compilation ?)

my guess is raspbian gcc 4.6 is not able to compile for armv7, even 
with some extra flags.


can you try
sudo apt-get install gcc-4.8 g++-4.8
and configure with CC=gcc48 CXX=g++48
(or whatever the gcc 4.8 compiler name is)

if that works, i will think of updating configure so it fails with an 
understandable error message

when building for armv6

Cheers,

Gilles

On 6/10/2015 2:18 AM, Jeff Layton wrote:

Gilles,

I was looking in Raspbian a little and do you know what I found?
When I do "gcc -v" it says it was built with "--with-arch=armv6".
Since I'm trying to compile on a Pi 2, doesn't this mean I'm
cross-compiling?

Thanks!

Jeff


Jeff,

btw, did you try a pI 1 before a pi2 ?

I checked some forums, and you will likely have to upgrade gcc to 4.8

a simpler option could be linaro 
https://www.raspberrypi.org/forums/viewtopic.php?f=56=98997


Cheers,

Gilles

On Tuesday, June 9, 2015, Gilles Gouaillardet 
<gilles.gouaillar...@gmail.com 
<mailto:gilles.gouaillar...@gmail.com>> wrote:


Jeff,

can you
gcc -march=armv7-a foo.c

    Cheers,

    Gilles

On Tuesday, June 9, 2015, Jeff Layton <layto...@att.net
<javascript:_e(%7B%7D,'cvml','layto...@att.net');>> wrote:

Gilles,

I'm not cross-compiling - I'm building on the Pi 2.

I'm not sure how to check if gcc can generate armv7 code.
I'm using Raspbian and I'm just using the default compilers
that come with it (I think it's gcc 4.6.3).

Thanks!

Jeff

Jeff,

dmb is available only on ARMv7 (Pi 2)

if i remember correctly, you are building Open MPI on
ARMv7 as well (Pi 2),
so this is not a cross compilation issue.

if you configure with -march=armv7, the relevant log is
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I.
-I../../opal/include -I../../orte/include
-I../../ompi/include -I../../oshmem/include
-I../../opal/mca/hwloc/hwloc191/hwloc/include/private/autogen
-I../../opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen
-I../..
-I/work/pi/src/openmpi-1.8.5/opal/mca/hwloc/hwloc191/hwloc/include
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent

-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include
-march=armv7 -MT atomic-asm.lo -MD -MP -MF
.deps/atomic-asm.Tpo -c atomic-asm.S  -fPIC -DPIC -o
.libs/atomic-asm.o^M
atomic-asm.S:1:0: error: target CPU does not support ARM
mode

which is very puzzling ...

can you confirm
1) you are not cross compiling
2) your gcc can generate code for armv7 ?

Cheers,

    Gilles




On 6/9/2015 1:48 AM, Jeff Layton wrote:

Jeff,

Sorry - I was traveling for a week and didn't have
to the RPi.

What happens if you don't supply CCASFLAGS at all?


The output from "make" is below. It died when it
tried to compile
atomic-local. It says the processor doesn't support
ARM mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/opa

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Gilles,

I was looking in Raspbian a little and do you know what I found?
When I do "gcc -v" it says it was built with "--with-arch=armv6".
Since I'm trying to compile on a Pi 2, doesn't this mean I'm
cross-compiling?

Thanks!

Jeff


Jeff,

btw, did you try a pI 1 before a pi2 ?

I checked some forums, and you will likely have to upgrade gcc to 4.8

a simpler option could be linaro 
https://www.raspberrypi.org/forums/viewtopic.php?f=56=98997


Cheers,

Gilles

On Tuesday, June 9, 2015, Gilles Gouaillardet 
<gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> 
wrote:


Jeff,

can you
gcc -march=armv7-a foo.c

Cheers,

Gilles

    On Tuesday, June 9, 2015, Jeff Layton <layto...@att.net
<javascript:_e(%7B%7D,'cvml','layto...@att.net');>> wrote:

Gilles,

I'm not cross-compiling - I'm building on the Pi 2.

I'm not sure how to check if gcc can generate armv7 code.
I'm using Raspbian and I'm just using the default compilers
that come with it (I think it's gcc 4.6.3).

Thanks!

Jeff

Jeff,

dmb is available only on ARMv7 (Pi 2)

if i remember correctly, you are building Open MPI on
ARMv7 as well (Pi 2),
so this is not a cross compilation issue.

if you configure with -march=armv7, the relevant log is
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I.
-I../../opal/include -I../../orte/include
-I../../ompi/include -I../../oshmem/include
-I../../opal/mca/hwloc/hwloc191/hwloc/include/private/autogen
-I../../opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen -I../..
-I/work/pi/src/openmpi-1.8.5/opal/mca/hwloc/hwloc191/hwloc/include
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent

-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include
-march=armv7 -MT atomic-asm.lo -MD -MP -MF
.deps/atomic-asm.Tpo -c atomic-asm.S  -fPIC -DPIC -o
.libs/atomic-asm.o^M
atomic-asm.S:1:0: error: target CPU does not support ARM mode

which is very puzzling ...

can you confirm
1) you are not cross compiling
2) your gcc can generate code for armv7 ?

Cheers,

Gilles




On 6/9/2015 1:48 AM, Jeff Layton wrote:

Jeff,

Sorry - I was traveling for a week and didn't have to
the RPi.

What happens if you don't supply CCASFLAGS at all?


The output from "make" is below. It died when it tried
to compile
atomic-local. It says the processor doesn't support
ARM mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal'
Making all in include
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make  all-am
make[3]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[3]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[2]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
Making all in asm
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s"
atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not
support

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Gilles,

Let me know if you think I should upgrade to gcc-4.8 and I'll
try it.

Thanks!

Jeff



Jeff,

btw, did you try a pI 1 before a pi2 ?

I checked some forums, and you will likely have to upgrade gcc to 4.8

a simpler option could be linaro 
https://www.raspberrypi.org/forums/viewtopic.php?f=56=98997


Cheers,

Gilles

On Tuesday, June 9, 2015, Gilles Gouaillardet 
<gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> 
wrote:


Jeff,

can you
gcc -march=armv7-a foo.c

Cheers,

Gilles

On Tuesday, June 9, 2015, Jeff Layton <layto...@att.net
<javascript:_e(%7B%7D,'cvml','layto...@att.net');>> wrote:

Gilles,

I'm not cross-compiling - I'm building on the Pi 2.

I'm not sure how to check if gcc can generate armv7 code.
I'm using Raspbian and I'm just using the default compilers
that come with it (I think it's gcc 4.6.3).

Thanks!

Jeff

Jeff,

dmb is available only on ARMv7 (Pi 2)

if i remember correctly, you are building Open MPI on
ARMv7 as well (Pi 2),
so this is not a cross compilation issue.

if you configure with -march=armv7, the relevant log is
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I.
-I../../opal/include -I../../orte/include
-I../../ompi/include -I../../oshmem/include
-I../../opal/mca/hwloc/hwloc191/hwloc/include/private/autogen
-I../../opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen -I../..
-I/work/pi/src/openmpi-1.8.5/opal/mca/hwloc/hwloc191/hwloc/include
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent

-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include
-march=armv7 -MT atomic-asm.lo -MD -MP -MF
.deps/atomic-asm.Tpo -c atomic-asm.S  -fPIC -DPIC -o
.libs/atomic-asm.o^M
atomic-asm.S:1:0: error: target CPU does not support ARM mode

which is very puzzling ...

can you confirm
1) you are not cross compiling
2) your gcc can generate code for armv7 ?

Cheers,

Gilles




    On 6/9/2015 1:48 AM, Jeff Layton wrote:

Jeff,

Sorry - I was traveling for a week and didn't have to
the RPi.

What happens if you don't supply CCASFLAGS at all?


The output from "make" is below. It died when it tried
to compile
atomic-local. It says the processor doesn't support
ARM mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal'
Making all in include
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make  all-am
make[3]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[3]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[2]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
Making all in asm
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s"
atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not
support ARM mode `dmb'
 

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Gilles,

No I didn't. This is a fresh Raspbian for the Pi 2. I'm using the
05-05-raspbian-wheezy image, plus I did ant apt-get update
about a week ago.

Thanks!

Jeff


Jeff,

btw, did you try a pI 1 before a pi2 ?

I checked some forums, and you will likely have to upgrade gcc to 4.8

a simpler option could be linaro 
https://www.raspberrypi.org/forums/viewtopic.php?f=56=98997


Cheers,

Gilles

On Tuesday, June 9, 2015, Gilles Gouaillardet 
<gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> 
wrote:


Jeff,

can you
gcc -march=armv7-a foo.c

Cheers,

Gilles

On Tuesday, June 9, 2015, Jeff Layton <layto...@att.net
<javascript:_e(%7B%7D,'cvml','layto...@att.net');>> wrote:

Gilles,

I'm not cross-compiling - I'm building on the Pi 2.

I'm not sure how to check if gcc can generate armv7 code.
I'm using Raspbian and I'm just using the default compilers
that come with it (I think it's gcc 4.6.3).

Thanks!

Jeff

Jeff,

dmb is available only on ARMv7 (Pi 2)

if i remember correctly, you are building Open MPI on
ARMv7 as well (Pi 2),
so this is not a cross compilation issue.

if you configure with -march=armv7, the relevant log is
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I.
-I../../opal/include -I../../orte/include
-I../../ompi/include -I../../oshmem/include
-I../../opal/mca/hwloc/hwloc191/hwloc/include/private/autogen
-I../../opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen -I../..
-I/work/pi/src/openmpi-1.8.5/opal/mca/hwloc/hwloc191/hwloc/include
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent

-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include
-march=armv7 -MT atomic-asm.lo -MD -MP -MF
.deps/atomic-asm.Tpo -c atomic-asm.S  -fPIC -DPIC -o
.libs/atomic-asm.o^M
atomic-asm.S:1:0: error: target CPU does not support ARM mode

which is very puzzling ...

can you confirm
1) you are not cross compiling
2) your gcc can generate code for armv7 ?

Cheers,

Gilles




    On 6/9/2015 1:48 AM, Jeff Layton wrote:

Jeff,

Sorry - I was traveling for a week and didn't have to
the RPi.

What happens if you don't supply CCASFLAGS at all?


The output from "make" is below. It died when it tried
to compile
atomic-local. It says the processor doesn't support
ARM mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal'
Making all in include
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make  all-am
make[3]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[3]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[2]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
Making all in asm
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s"
atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:70: Error: selected pr

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Gilles,

Yes I can. I tried a simple C code ("hello world") and it compiled
and ran correctly.

Thanks!

Jeff


Jeff,

can you
gcc -march=armv7-a foo.c

Cheers,

Gilles

On Tuesday, June 9, 2015, Jeff Layton <layto...@att.net 
<mailto:layto...@att.net>> wrote:


Gilles,

I'm not cross-compiling - I'm building on the Pi 2.

I'm not sure how to check if gcc can generate armv7 code.
I'm using Raspbian and I'm just using the default compilers
that come with it (I think it's gcc 4.6.3).

Thanks!

Jeff

Jeff,

dmb is available only on ARMv7 (Pi 2)

if i remember correctly, you are building Open MPI on ARMv7 as
well (Pi 2),
so this is not a cross compilation issue.

if you configure with -march=armv7, the relevant log is
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I.
-I../../opal/include -I../../orte/include -I../../ompi/include
-I../../oshmem/include
-I../../opal/mca/hwloc/hwloc191/hwloc/include/private/autogen
-I../../opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen
-I../..
-I/work/pi/src/openmpi-1.8.5/opal/mca/hwloc/hwloc191/hwloc/include
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent

-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include
-march=armv7 -MT atomic-asm.lo -MD -MP -MF
.deps/atomic-asm.Tpo -c atomic-asm.S  -fPIC -DPIC -o
.libs/atomic-asm.o^M
atomic-asm.S:1:0: error: target CPU does not support ARM mode

which is very puzzling ...

can you confirm
1) you are not cross compiling
2) your gcc can generate code for armv7 ?

Cheers,

Gilles




    On 6/9/2015 1:48 AM, Jeff Layton wrote:

Jeff,

Sorry - I was traveling for a week and didn't have to the RPi.

What happens if you don't supply CCASFLAGS at all?


The output from "make" is below. It died when it tried to
compile
atomic-local. It says the processor doesn't support ARM
mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory
'/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory
'/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/opal'
Making all in include
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make  all-am
make[3]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[3]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
make[2]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/include'
Making all in asm
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s" atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:86: Error: selected processor does not
support ARM mode `ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not
support ARM mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not
support ARM mode `ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not
support ARM mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:130: Error: selected processor does not
support ARM mode `ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not
support ARM mode `dmb'
atomic-asm.S:136: Error: sele

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Gilles,

I'm not cross-compiling - I'm building on the Pi 2.

I'm not sure how to check if gcc can generate armv7 code.
I'm using Raspbian and I'm just using the default compilers
that come with it (I think it's gcc 4.6.3).

Thanks!

Jeff


Jeff,

dmb is available only on ARMv7 (Pi 2)

if i remember correctly, you are building Open MPI on ARMv7 as well 
(Pi 2),

so this is not a cross compilation issue.

if you configure with -march=armv7, the relevant log is
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. 
-I../../opal/include -I../../orte/include -I../../ompi/include 
-I../../oshmem/include 
-I../../opal/mca/hwloc/hwloc191/hwloc/include/private/autogen 
-I../../opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen -I../.. 
-I/work/pi/src/openmpi-1.8.5/opal/mca/hwloc/hwloc191/hwloc/include 
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent 
-I/work/pi/src/openmpi-1.8.5/opal/mca/event/libevent2021/libevent/include 
-march=armv7 -MT atomic-asm.lo -MD -MP -MF .deps/atomic-asm.Tpo -c 
atomic-asm.S  -fPIC -DPIC -o .libs/atomic-asm.o^M

atomic-asm.S:1:0: error: target CPU does not support ARM mode

which is very puzzling ...

can you confirm
1) you are not cross compiling
2) your gcc can generate code for armv7 ?

Cheers,

Gilles




On 6/9/2015 1:48 AM, Jeff Layton wrote:

Jeff,

Sorry - I was traveling for a week and didn't have to the RPi.


What happens if you don't supply CCASFLAGS at all?



The output from "make" is below. It died when it tried to compile
atomic-local. It says the processor doesn't support ARM mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/opal'
Making all in include
make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/include'
make  all-am
make[3]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/include'
make[3]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/include'
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/include'
Making all in asm
make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s" atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:15: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:23: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:55: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:70: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:86: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:130: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support ARM mode 
`dmb'
atomic-asm.S:136: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/06/27050.php




___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/06/27055.php

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Jeff,

Sorry - I was traveling for a week and didn't have to the RPi.


What happens if you don't supply CCASFLAGS at all?



The output from "make" is below. It died when it tried to compile
atomic-local. It says the processor doesn't support ARM mode "dmb".

Thanks!

Jeff



pi@raspberrypi /work/pi/src/openmpi-1.8.5 $ make
Making all in config
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/config'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/config'
Making all in contrib
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/contrib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/contrib'
Making all in opal
make[1]: Entering directory '/work/pi/src/openmpi-1.8.5/opal'
Making all in include
make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/include'
make  all-am
make[3]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/include'
make[3]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/include'
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/include'
Making all in asm
make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/asm'
  CC   asm.lo
rm -f atomic-asm.S
ln -s "../../opal/asm/generated/atomic-local.s" atomic-asm.S
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:86: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

atomic-asm.S:115: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:130: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'

atomic-asm.S:135: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:136: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Jeff,

The error happens during the configure step before compiling.
However, I ran the make command as you indicated and I'm
attaching the output to this email.

Thanks!

Jeff



Can you send the output of "make V=1"?

That will show the exact command line that is being used to build that file.



On May 29, 2015, at 2:17 PM, Jeff Layton <layto...@att.net> wrote:

George,

I changed my configure command to be:

./configure CCASFLAGS=-march=native

and I get an error while running configure:

...
*** Assembler
checking dependency style of gcc -std=gnu99... gcc3
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking for fgrep... /bin/grep -F
checking if need to remove -g from CCASFLAGS... no
checking whether to enable smp locks... yes
checking if .proc/endp is needed... no
checking directive for setting text section... .text
checking directive for exporting symbols... .globl
checking for objdump... objdump
checking if .note.GNU-stack is needed... yes
checking suffix for labels... :
checking prefix for global symbol labels... none
configure: error: Could not determine global symbol label prefix


Not sure why configure failed at this point.

Thanks!

Jeff



As you are not cross-compiling I would expect gcc to use the right assembly by 
default. What is happening is you force the native mode (-march=native) ?

   George.


On Fri, May 29, 2015 at 10:09 AM, Jeff Layton <layto...@att.net> wrote:
On 05/29/2015 09:35 AM, Jeff Layton wrote:

Gilles,

oops - yes, CFLAGS. But I also saw this posting:

https://www.open-mpi.org/community/lists/users/2013/01/2.php

where CCASFLAGS is used (I assume because for asm). I'm trying
this flag when I configure Open MPI.

I tried using the CCASFLAGS flag from the above link and it didn't work. The 
error
now reads:

Making all in mca/memory/linux
make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/mca/memory/linux'
   CC   memory_linux_component.lo
   CC   memory_linux_ptmalloc2.lo
   CC   memory_linux_munmap.lo
   CC   malloc.lo
/tmp/cc7g4mWi.s: Assembler messages:
/tmp/cc7g4mWi.s:948: Error: selected processor does not support ARM mode `dmb'
Makefile:1694: recipe for target 'malloc.lo' failed
make[2]: *** [malloc.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/mca/memory/linux'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I used the configure flag CCASFLAGS=-march=armv7-a

Not sure if that is correct or not. I'm guessing I'm using the wrong
architecture for the Pi 2. Suggestions?

Thanks!

Jeff




Thanks!

Jeff



Jeff,

shall I assume you made a typo and wrote CCFLAGS instead of CFLAGS ?

also, can you double check the flags are correctly passed to the assembler with
cd opal/asm
make -n atomic-asm.lo

Cheers,

Gilles

On Friday, May 29, 2015, Jeff Layton <layto...@att.net> wrote:
Good morning,

I'm building OpenMPI from source on a Raspberry Pi 2 and
I've hit an error. The error is:

make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/asm'
   CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:86: Error: selected processor does not support ARM mode `ldrexd 
r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM mode `strexd 
r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM mode `ldrexd 
r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM mode `strexd 
r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:130: Error: selected processor does not support ARM mode `ldrexd 
r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:136: Error: selected processor does not support ARM mode `strexd 
r1,r6,r7,[r0]'
Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I was doing some googling and I saw where I need to specify
CCFLAGS when I run configure but I want to make 100% sure
I have the right arguments. Can

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

George,

I changed my configure command to be:

./configure CCASFLAGS=-march=native

and I get an error while running configure:

...
*** Assembler
checking dependency style of gcc -std=gnu99... gcc3
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking for fgrep... /bin/grep -F
checking if need to remove -g from CCASFLAGS... no
checking whether to enable smp locks... yes
checking if .proc/endp is needed... no
checking directive for setting text section... .text
checking directive for exporting symbols... .globl
checking for objdump... objdump
checking if .note.GNU-stack is needed... yes
checking suffix for labels... :
checking prefix for global symbol labels... none
configure: error: Could not determine global symbol label prefix


Not sure why configure failed at this point.

Thanks!

Jeff


As you are not cross-compiling I would expect gcc to use the right 
assembly by default. What is happening is you force the native mode 
(-march=native) ?


  George.


On Fri, May 29, 2015 at 10:09 AM, Jeff Layton <layto...@att.net 
<mailto:layto...@att.net>> wrote:


On 05/29/2015 09:35 AM, Jeff Layton wrote:

Gilles,

oops - yes, CFLAGS. But I also saw this posting:

https://www.open-mpi.org/community/lists/users/2013/01/2.php

where CCASFLAGS is used (I assume because for asm). I'm trying
this flag when I configure Open MPI.


I tried using the CCASFLAGS flag from the above link and it didn't
work. The error
now reads:

Making all in mca/memory/linux
make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/mca/memory/linux'
  CC   memory_linux_component.lo
  CC   memory_linux_ptmalloc2.lo
  CC   memory_linux_munmap.lo
  CC   malloc.lo
/tmp/cc7g4mWi.s: Assembler messages:
/tmp/cc7g4mWi.s:948: Error: selected processor does not support
ARM mode `dmb'
Makefile:1694: recipe for target 'malloc.lo' failed
make[2]: *** [malloc.lo] Error 1
make[2]: Leaving directory
'/work/pi/src/openmpi-1.8.5/opal/mca/memory/linux'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I used the configure flag CCASFLAGS=-march=armv7-a

Not sure if that is correct or not. I'm guessing I'm using the wrong
architecture for the Pi 2. Suggestions?

Thanks!

Jeff





Thanks!

Jeff



Jeff,

shall I assume you made a typo and wrote CCFLAGS instead of CFLAGS ?

also, can you double check the flags are correctly passed to the
assembler with
cd opal/asm
make -n atomic-asm.lo

Cheers,

Gilles

On Friday, May 29, 2015, Jeff Layton <layto...@att.net
<mailto:layto...@att.net>> wrote:

Good morning,

I'm building OpenMPI from source on a Raspberry Pi 2 and
I've hit an error. The error is:

make[2]: Entering directory
'/work/pi/src/openmpi-1.8.5/opal/asm'
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:86: Error: selected processor does not support
ARM mode `ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support
ARM mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support
ARM mode `ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support
ARM mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:130: Error: selected processor does not support
ARM mode `ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support
ARM mode `dmb'
atomic-asm.S:136: Error: selected processor does not support
ARM mode `strexd r1,r6,r7,[r0]'
Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

On 05/29/2015 09:35 AM, Jeff Layton wrote:

Gilles,

oops - yes, CFLAGS. But I also saw this posting:

https://www.open-mpi.org/community/lists/users/2013/01/2.php

where CCASFLAGS is used (I assume because for asm). I'm trying
this flag when I configure Open MPI.


I tried using the CCASFLAGS flag from the above link and it didn't work. 
The error

now reads:

Making all in mca/memory/linux
make[2]: Entering directory 
'/work/pi/src/openmpi-1.8.5/opal/mca/memory/linux'

  CC   memory_linux_component.lo
  CC   memory_linux_ptmalloc2.lo
  CC   memory_linux_munmap.lo
  CC   malloc.lo
/tmp/cc7g4mWi.s: Assembler messages:
/tmp/cc7g4mWi.s:948: Error: selected processor does not support ARM mode 
`dmb'

Makefile:1694: recipe for target 'malloc.lo' failed
make[2]: *** [malloc.lo] Error 1
make[2]: Leaving directory 
'/work/pi/src/openmpi-1.8.5/opal/mca/memory/linux'

Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I used the configure flag CCASFLAGS=-march=armv7-a

Not sure if that is correct or not. I'm guessing I'm using the wrong
architecture for the Pi 2. Suggestions?

Thanks!

Jeff




Thanks!

Jeff



Jeff,

shall I assume you made a typo and wrote CCFLAGS instead of CFLAGS ?

also, can you double check the flags are correctly passed to the 
assembler with

cd opal/asm
make -n atomic-asm.lo

Cheers,

Gilles

On Friday, May 29, 2015, Jeff Layton <layto...@att.net 
<mailto:layto...@att.net>> wrote:


Good morning,

I'm building OpenMPI from source on a Raspberry Pi 2 and
I've hit an error. The error is:

make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/asm'
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:15: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:23: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:55: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:70: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:86: Error: selected processor does not support ARM
mode `ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM
mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM
mode `ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM
mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:130: Error: selected processor does not support ARM
mode `ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:136: Error: selected processor does not support ARM
mode `strexd r1,r6,r7,[r0]'
Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I was doing some googling and I saw where I need to specify
CCFLAGS when I run configure but I want to make 100% sure
I have the right arguments. Can anyone help?

Thanks!

Jeff

P.S. I'd rather build OpenMPI from source rather than use the
pre-built
packages in the Raspbian repository.

Re: [OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Gilles,

oops - yes, CFLAGS. But I also saw this posting:

https://www.open-mpi.org/community/lists/users/2013/01/2.php

where CCASFLAGS is used (I assume because for asm). I'm trying
this flag when I configure Open MPI.

Thanks!

Jeff



Jeff,

shall I assume you made a typo and wrote CCFLAGS instead of CFLAGS ?

also, can you double check the flags are correctly passed to the 
assembler with

cd opal/asm
make -n atomic-asm.lo

Cheers,

Gilles

On Friday, May 29, 2015, Jeff Layton <layto...@att.net 
<mailto:layto...@att.net>> wrote:


Good morning,

I'm building OpenMPI from source on a Raspberry Pi 2 and
I've hit an error. The error is:

make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/asm'
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:15: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:23: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:55: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:70: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:86: Error: selected processor does not support ARM
mode `ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM
mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM
mode `ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM
mode `strexd r1,r6,r7,[r0]'
atomic-asm.S:115: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:130: Error: selected processor does not support ARM
mode `ldrexd r4,r5,[r0]'
atomic-asm.S:135: Error: selected processor does not support ARM
mode `dmb'
atomic-asm.S:136: Error: selected processor does not support ARM
mode `strexd r1,r6,r7,[r0]'
Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I was doing some googling and I saw where I need to specify
CCFLAGS when I run configure but I want to make 100% sure
I have the right arguments. Can anyone help?

Thanks!

Jeff

P.S. I'd rather build OpenMPI from source rather than use the
pre-built
packages in the Raspbian repository.


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/05/26976.php



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/05/26977.php

[OMPI users] Building OpenMPI on Raspberry Pi 2

[Jeff Layton]

Good morning,

I'm building OpenMPI from source on a Raspberry Pi 2 and
I've hit an error. The error is:

make[2]: Entering directory '/work/pi/src/openmpi-1.8.5/opal/asm'
  CPPASatomic-asm.lo
atomic-asm.S: Assembler messages:
atomic-asm.S:7: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:15: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:23: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:55: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:70: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:86: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:91: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'
atomic-asm.S:107: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'
atomic-asm.S:112: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

atomic-asm.S:115: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:130: Error: selected processor does not support ARM mode 
`ldrexd r4,r5,[r0]'

atomic-asm.S:135: Error: selected processor does not support ARM mode `dmb'
atomic-asm.S:136: Error: selected processor does not support ARM mode 
`strexd r1,r6,r7,[r0]'

Makefile:1608: recipe for target 'atomic-asm.lo' failed
make[2]: *** [atomic-asm.lo] Error 1
make[2]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal/asm'
Makefile:2149: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/work/pi/src/openmpi-1.8.5/opal'
Makefile:1698: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I was doing some googling and I saw where I need to specify
CCFLAGS when I run configure but I want to make 100% sure
I have the right arguments. Can anyone help?

Thanks!

Jeff

P.S. I'd rather build OpenMPI from source rather than use the pre-built
packages in the Raspbian repository.

[OMPI users] Using orterun to start jobs with numactl

[Jeff Layton]

 Good afternoon,

I'm working with some AMD Interlagos nodes (16-core, 4-socket),
and I want to bind processes to cores in specific layouts. These
may not be in logical order so I need to be able to my MPI app
and point it to specific cores on specific nodes (although the
core ordering is the same for all nodes).

I'm assuming I need to "manually" start the processes using
orted (or orterun) and numactl on the specific cores. Since I've
never used orted or orterun to start jobs and I'm struggling with
the man pages a bit - any suggestions?

Thanks!

Jeff

Re: [OMPI users] Problem getting OpenMPI to run

[Jeff Layton]

Joe,

You are correct this is a ROCKS cluster. I didn't use the the --sge option when 
building (I tend to stay more generic, but I should have done that).

Not sure of the OFED release but I don't admin this cluster and the owners are 
picky about upgrades (tends to break Lustre).

BTW - the problem was solved. There was a configuration error for the specific 
queue. It was found and fixed and things seem to be running normally.

Thanks for help and I'm sorry for disturbing everyone. I wasn't familiar enough 
with the error messages to tell if it was OpenMPI or SGE.

TIA!

Jeff






From: Joe Landman <land...@scalableinformatics.com>
To: Open MPI Users <us...@open-mpi.org>
Sent: Monday, June 1, 2009 3:34:40 PM
Subject: Re: [OMPI users] Problem getting OpenMPI to run

Jeff Layton wrote:
> Jeff Squyres wrote:
>> On Jun 1, 2009, at 2:04 PM, Jeff Layton wrote:
>> 
>>> error: executing task of job 3084 failed: execution daemon on host
>>> "compute-2-2.local" didn't accept task
>>> 
>> 
>> This looks like an error message from the resource manager/scheduler -- not 
>> from OMPI (i.e., OMPI tried to launch a process on a node and the launch 
>> failed because something rejected it).
>> 
>> Which one are you using?

When you built Open-MPI, did you use the

--with-sge

switch?  Or if this is an OFED release, is it possible that this wasn't 
specified?

FWIW, this looks like a Rocks compute node ("compute-2-2.local" gives that 
away).  The OFED Rolls in Rocks have had a few issues in the past with how they 
were built, so you may be running into that.  If you didn't build it yourself, 
I'd suggest at least giving that a try.

Alternatively, OFED-1.4 is pretty good.  Has a later version of Open-MPI than 
1.3.x

Joe

> 
> SGE
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics,
email: land...@scalableinformatics.com
web  : http://scalableinformatics.com
  http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Problem getting OpenMPI to run

[Jeff Layton]

Jeff Squyres wrote:

On Jun 1, 2009, at 2:04 PM, Jeff Layton wrote:


error: executing task of job 3084 failed: execution daemon on host
"compute-2-2.local" didn't accept task



This looks like an error message from the resource manager/scheduler 
-- not from OMPI (i.e., OMPI tried to launch a process on a node and 
the launch failed because something rejected it).


Which one are you using?


SGE

[OMPI users] Problem getting OpenMPI to run

[Jeff Layton]

Good morning,

I think I sent this out last week but I did some "experimentation"
and kind-of/sort-of got my OpenMPI application to run. But I do
have a weird problem.

I can get the application (build with OpenMPI-1.3.2 with gcc and
the app is built with Intel 10.2) to run on the IB network (not sure
of the version of OFED but it might be 1.3.x) with certain CPUs.
For example I can run the application on AMD Shanghai processors
just fine. But when I try some other processors (also AMD), I
get the following error message:


error: executing task of job 3084 failed: execution daemon on host 
"compute-2-2.local" didn't accept task

--
A daemon (pid 27796) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--
--
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--
mpirun: clean termination accomplished



I've been googling my fingers off without any luck. My next steps are
to start putting printf's in OpenMPI to figure out where the problem
is occurring :)  Any ideas or things I can do to start? (I can provide all
kinds of information including ompi_info if you anyone cares to look
through it).

TIA!

Jeff

Re: [OMPI users] Problem with 1.3.2 - need tips on debugging

[Jeff Layton]

I've got some more information (after rebuilding Open MPI and the
application a few times). I put

-mca mpi_show_mca_params enviro


in my mpirun line to get some of the parameter information back.
I get the following information back (warning - it's long).

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01564] [[58307,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01565] [[58306,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01562] [[58309,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01560] [[58311,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01566] [[58305,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01563] [[58308,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  compute-0-0.local
Framework: ras
Component: proxy
--
[compute-0-0.local:01559] [[58312,0],0] ORTE_ERROR_LOG: Error in file 
ess_hnp_module.c at line 199

--
It looks 

[OMPI users] Problem with 1.3.2 - need tips on debugging

[Jeff Layton]

Good morning,

I just built 1.3.2 on a ROCKS 4.something system. I built my code
(CFL3D) with the Intel 10.1 compilers. I also linked in the
OpenMPI libs and the Intel libraries to make sure I had the paths
correct. When I try running my code, I get the following,


error: executing task of job 2951 failed: execution daemon on host 
"compute-2-3.local" didn't accept task

--
A daemon (pid 12015) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--
--
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--
mpirun: clean termination accomplished



Everything seems correct. I checked that the mpirun was correct
and the binary has the correct libraries (checked using ldd).

Can anyone tell me what the "status 1" means? Any tips on debugging
the problem?

Thanks!

Jeff

Re: [OMPI users] Open MPI 2009 released

[Jeff Layton]

I'm still looking for the new "-11" option that will allow OpenMPI compiled 
code to run even faster. I'm hoping that option makes it in soon (isn't it part 
of the 11.7 MPI standard?)

Jeff






From: George Bosilca 
To: Open MPI MPI Developers ; Open MPI Users 

Sent: Wednesday, April 1, 2009 6:04:15 PM
Subject: [OMPI users] Open MPI 2009 released

The Open MPI Team, representing a consortium of bailed-out banks, car
manufacturers, and insurance companies, is pleased to announce the
release of the "unbreakable" / bug-free version Open MPI 2009,
(expected to be available by mid-2011).  This release is essentially a
complete rewrite of Open MPI based on new technologies such as C#,
Java, and object-oriented Cobol (so say we all!).  Buffer overflows
and memory leaks are now things of the past.  We strongly recommend
that all users upgrade to Windows 7 to fully take advantage of the new
powers embedded in Open MPI.

This version can be downloaded from the The Onion web site or from
many BitTorrent networks (seeding now; the Open MPI ISO is
approximately 3.97GB -- please wait for the full upload).

Here is an abbreviated list of changes in Open MPI 2009 as compared to
the previous version:

- Dropped support for MPI 2 in favor of the newly enhanced MPI 11.7
standard.  MPI_COOK_DINNER support is only available with additional
equipment (some assembly may be required).  An experimental PVM-like
API has been introduced to deal with the current limitations of the
MPI 11.7 API.
- Added a Twitter network transport capable of achieving peta-scale
per second bandwidth (but only on useless data).
- Dropped support for the barely-used x86 and x86_64 architectures in
favor of the most recent ARM6 architecture.  As a direct result,
several Top500 sites are planning to convert from their now obsolete
peta-scale machines to high-reliability iPhone clusters using the
low-latency AT 3G network.
- The iPhone iMPI app (powered by iOpen MPI) is now downloadable from
the iTunes Store.  Blackberry support will be included in a future
release.
- Fix all compiler errors related to the PGI 8.0 compiler by
completely dropping support.
- Add some "green" features for energy savings.  The new "--bike"
mpirun option will only run your parallel jobs only during the
operation hours of the official Open MPI biking team.  The
"--preload-result" option will directly embed the final result in
the parallel execution, leading to more scalable and reliable runs
and decreasing the execution time of any parallel application under
the real-time limit of 1 second.  Open MPI is therefore EnergyStar
compliant when used with these options.
- In addition to moving Open MPI's lowest point-to-point transports to
be an external project, limited support will be offered for
industry-standard platforms.  Our focus will now be to develop
highly scalable transports based on widely distributed technologies
such as SMTP, High Performance Gopher (v3.8 and later), OLE COMM,
RSS/Atom, DNS, and Bonjour.
- Opportunistic integration with Conflicker in order to utilize free
resources distributed world-wide.
- Support for all Fortran versions prior to Fortran 2020 has been
dropped.

Make today an Open MPI day!


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] mlx4 error - looking for guidance

[Jeff Layton]

Oops. I ran it on the head node and not the compute node. Here is the
output from a compute node:

hca_id: mlx4_0
   fw_ver: 2.3.000
   node_guid:  0018:8b90:97fe:1b6d
   sys_image_guid: 0018:8b90:97fe:1b70
   vendor_id:  0x02c9
   vendor_part_id: 25418
   hw_ver: 0xA0
   board_id:   DEL08C001
   phys_port_cnt:  2
   max_mr_size:0x
   page_size_cap:  0xf000
   max_qp: 131008
   max_qp_wr:  16351
   device_cap_flags:   0x001c1c66
   max_sge:32
   max_sge_rd: 0
   max_cq: 65408
   max_cqe:4194303
   max_mr: 131056
   max_pd: 32764
   max_qp_rd_atom: 16
   max_ee_rd_atom: 0
   max_res_rd_atom:2096128
   max_qp_init_rd_atom:128
   max_ee_init_rd_atom:0
   atomic_cap: ATOMIC_HCA (1)
   max_ee: 0
   max_rdd:0
   max_mw: 0
   max_raw_ipv6_qp:0
   max_raw_ethy_qp:0
   max_mcast_grp:  8192
   max_mcast_qp_attach:56
   max_total_mcast_qp_attach:  458752
   max_ah: 0
   max_fmr:0
   max_srq:65472
   max_srq_wr: 16383
   max_srq_sge:31
   max_pkeys:  128
   local_ca_ack_delay: 15
   port:   1
   state:  PORT_ACTIVE (4)
   max_mtu:2048 (4)
   active_mtu: 2048 (4)
   sm_lid: 41
   port_lid:   70
   port_lmc:   0x00
   max_msg_sz: 0x4000
   port_cap_flags: 0x02510868
   max_vl_num: 8 (4)
   bad_pkey_cntr:  0x0
   qkey_viol_cntr: 0x0
   sm_sl:  0
   pkey_tbl_len:   128
   gid_tbl_len:128
   subnet_timeout: 18
   init_type_reply:0
   active_width:   4X (2)
   active_speed:   5.0 Gbps (2)
   phys_state: LINK_UP (5)
   GID[  0]:   
fe80::::0018:8b90:97fe:1b6e


   port:   2
   state:  PORT_DOWN (1)
   max_mtu:2048 (4)
   active_mtu: 2048 (4)
   sm_lid: 0
   port_lid:   0
   port_lmc:   0x00
   max_msg_sz: 0x4000
   port_cap_flags: 0x02510868
   max_vl_num: 8 (4)
   bad_pkey_cntr:  0x0
   qkey_viol_cntr: 0x0
   sm_sl:  0
   pkey_tbl_len:   128
   gid_tbl_len:128
   subnet_timeout: 0
   init_type_reply:0
   active_width:   4X (2)
   active_speed:   2.5 Gbps (1)
   phys_state: POLLING (2)
   GID[  0]:   
fe80::::0018:8b90:97fe:1b6f






Do you have the same HCA adapter type on all of your machines ?
In the error log I see mlx4 error message , and mlx4 is connectX driver,
but ibv_devinfo show some older hca.


Jeff,
Can you please provide more information about you HCA type 
(ibv_devinfo -v).
Do you see this error immediate during startup, or you get it during 
your run ?


Thanks,
Pasha

Jeff Layton wrote:

Evening everyone,

I'm running a CFD code on IB and I've encountered an error I'm not 
sure about and I'm looking for some guidance on where to start 
looking. Here's the error:


mlx4: local QP operation err (QPN 260092, WQE index 9a9e, 
vendor syndrome 6f, opcode = 5e)
[0,1,6][btl_openib_component.c:1392:btl_openib_component_progress] 
from compute-2-0.local to: compute-2-0.local erro
r polling HP CQ with status

Re: [OMPI users] mlx4 error - looking for guidance

[Jeff Layton]

Pasha,

Here you go... :) Thanks for looking at this.

Jeff

hca_id: mthca0
   fw_ver: 4.8.200
   node_guid:  0003:ba00:0100:38ac
   sys_image_guid: 0003:ba00:0100:38af
   vendor_id:  0x02c9
   vendor_part_id: 25208
   hw_ver: 0xA0
   board_id:   MT_00B0010001
   phys_port_cnt:  2
   max_mr_size:0x
   page_size_cap:  0xf000
   max_qp: 64512
   max_qp_wr:  65535
   device_cap_flags:   0x1c76
   max_sge:59
   max_sge_rd: 0
   max_cq: 65408
   max_cqe:131071
   max_mr: 131056
   max_pd: 32768
   max_qp_rd_atom: 4
   max_ee_rd_atom: 0
   max_res_rd_atom:258048
   max_qp_init_rd_atom:128
   max_ee_init_rd_atom:0
   atomic_cap: ATOMIC_HCA (1)
   max_ee: 0
   max_rdd:0
   max_mw: 0
   max_raw_ipv6_qp:0
   max_raw_ethy_qp:0
   max_mcast_grp:  8192
   max_mcast_qp_attach:56
   max_total_mcast_qp_attach:  458752
   max_ah: 0
   max_fmr:0
   max_srq:960
   max_srq_wr: 65535
   max_srq_sge:31
   max_pkeys:  64
   local_ca_ack_delay: 15
   port:   1
   state:  PORT_ACTIVE (4)
   max_mtu:2048 (4)
   active_mtu: 2048 (4)
   sm_lid: 41
   port_lid:   41
   port_lmc:   0x00
   max_msg_sz: 0x8000
   port_cap_flags: 0x02510a6a
   max_vl_num: 8 (4)
   bad_pkey_cntr:  0x0
   qkey_viol_cntr: 0x0
   sm_sl:  0
   pkey_tbl_len:   64
   gid_tbl_len:32
   subnet_timeout: 18
   init_type_reply:0
   active_width:   4X (2)
   active_speed:   2.5 Gbps (1)
   phys_state: LINK_UP (5)
   GID[  0]:   
fe80::::0003:ba00:0100:38ad


   port:   2
   state:  PORT_DOWN (1)
   max_mtu:2048 (4)
   active_mtu: 512 (2)
   sm_lid: 0
   port_lid:   0
   port_lmc:   0x00
   max_msg_sz: 0x8000
   port_cap_flags: 0x02510a68
   max_vl_num: 8 (4)
   bad_pkey_cntr:  0x0
   qkey_viol_cntr: 0x0
   sm_sl:  0
   pkey_tbl_len:   64
   gid_tbl_len:32
   subnet_timeout: 0
   init_type_reply:0
   active_width:   4X (2)
   active_speed:   2.5 Gbps (1)
   phys_state: POLLING (2)
   GID[  0]:   
fe80::::0003:ba00:0100:38ae




Jeff,
Can you please provide more information about you HCA type 
(ibv_devinfo -v).
Do you see this error immediate during startup, or you get it during 
your run ?


Thanks,
Pasha

Jeff Layton wrote:

Evening everyone,

I'm running a CFD code on IB and I've encountered an error I'm not 
sure about and I'm looking for some guidance on where to start 
looking. Here's the error:


mlx4: local QP operation err (QPN 260092, WQE index 9a9e, vendor 
syndrome 6f, opcode = 5e)
[0,1,6][btl_openib_component.c:1392:btl_openib_component_progress] 
from compute-2-0.local to: compute-2-0.local erro
r polling HP CQ with status LOCAL QP OPERATION ERROR status number 2 
for wr_id 37742320 opcode 0
mpirun noticed that job rank 0 with PID 21220 on node 
compute-2-0.local exited on signal 15 (Terminated).

78 additional processes aborted (not shown