Re: [OMPI users] Using mpicc to cross-compile MPI applications

[Graham Holland]

Hi Gilles,

Thanks for the quick response. I tried your suggestion of setting
the --enable-script-wrapper-compilers option to configure and ran
into the following bug (as I am on version 1.10.2):

https://www.mail-archive.com/users@lists.open-mpi.org/msg30166.html

I applied the patch found in your message above, and now have a
working mpicc perl script. However, this is the output of mpicc --showme:

arm-linux-gnueabihf-gcc -I/usr/include -pthread -L/usr/lib -Wl,-rpath 
-Wl,/usr/lib -Wl,--enable-new-dtags -lmpi -lm -ldl -lutil


As you can see the include and library paths refer to my host, instead
of my cross-compiler's sysroot as I might have expected. Do you know
why this is the case?

I noticed in the generated ompi_wrapper_script that the variables,
includedir and libdir are set as follows:

my $includedir = "/usr/include";
my $libdir = "/usr/lib";

Which I assume are generated from ompi_wrapper_script.in from the
following:

my $includedir = "@OMPI_WRAPPER_INCLUDEDIR@";
my $libdir = "@OMPI_WRAPPER_LIBDIR@";

Is there a way to fix this? I *think* what I need is for the -I
and -L options to use paths from $SYSROOT/usr/include and
$SYSROOT/usr/lib while the linker rpath option should use
/usr/lib

Please let me know if you need more information about my setup
and thanks for your help so far.

Cheers,
Graham


On 2017-03-02 05:39 PM, Gilles Gouaillardet wrote:

Graham,


you can configure Open MPI with '--enable-script-wrapper-compilers'

that will make wrappers as scripts instead of binaries.


Cheers,


Gilles


On 3/3/2017 10:23 AM, Graham Holland wrote:

Hello,

I am using OpenMPI version 1.10.2 on an arm development board and have
successfully cross-compiled the OpenMPI libraries and am able to use
them with simple MPI applications.

My question is, what is the best approach for cross-compiling more
complex MPI applications which use mpicc to determine compiler flags?
The mpicc that is built is an ARM executable and obviously cannot
be run on my host.

An example of such an application is MPPTEST, which uses mpicc from
within its Makefile: ftp://ftp.mcs.anl.gov/pub/mpi/tools/perftest.tar.gz

I have read in the FAQ about overriding the compiler wrapper default
flags using environment variables, so one possibility is to use my
host mpicc with these set appropriately.

I am wondering however if there is better way?

Thank you,

Graham


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Shared Windows and MPI_Accumulate

[Howard Pritchard]

Hello Joseph,

I'm still unable to reproduce this system on my SLES12 x86_64 node.

Are you building with CFLAGS=-O3?

If so, could you build without CFLAGS set and see if you still see the
failure?

Howard


2017-03-02 2:34 GMT-07:00 Joseph Schuchart :

> Hi Howard,
>
> Thanks for trying to reproduce this. It seems that on master the issue
> occurs less frequently but is still there. I used the following bash
> one-liner on my laptop and on our Linux Cluster (single node, 4 processes):
>
> ```
> $ for i in $(seq 1 100) ; do echo $i && mpirun -n 4
> ./mpi_shared_accumulate | grep \! && break ; done
> 1
> 2
> [0] baseptr[0]: 1004 (expected 1010) [!!!]
> [0] baseptr[1]: 1005 (expected 1011) [!!!]
> [0] baseptr[2]: 1006 (expected 1012) [!!!]
> [0] baseptr[3]: 1007 (expected 1013) [!!!]
> [0] baseptr[4]: 1008 (expected 1014) [!!!]
> ```
>
> Sometimes the error occurs after one or two iterations (like above),
> sometimes only at iteration 20 or later. However, I can reproduce it within
> the 100 runs every time I run the statement above. I am attaching the
> config.log and output of ompi_info of master on my laptop. Please let me
> know if I can help with anything else.
>
> Thanks,
> Joseph
>
> On 03/01/2017 11:24 PM, Howard Pritchard wrote:
>
> Hi Joseph,
>
> I built this test with craypich (Cray MPI) and it passed.  I also tried
> with Open MPI master and the test passed.  I also tried with 2.0.2
> and can't seem to reproduce on my system.
>
> Could you post the output of config.log?
>
> Also, how intermittent is the problem?
>
>
> Thanks,
>
> Howard
>
>
>
>
> 2017-03-01 8:03 GMT-07:00 Joseph Schuchart :
>
>> Hi all,
>>
>> We are seeing issues in one of our applications, in which processes in a
>> shared communicator allocate a shared MPI window and execute MPI_Accumulate
>> simultaneously on it to iteratively update each process' values. The test
>> boils down to the sample code attached. Sample output is as follows:
>>
>> ```
>> $ mpirun -n 4 ./mpi_shared_accumulate
>> [1] baseptr[0]: 1010 (expected 1010)
>> [1] baseptr[1]: 1011 (expected 1011)
>> [1] baseptr[2]: 1012 (expected 1012)
>> [1] baseptr[3]: 1013 (expected 1013)
>> [1] baseptr[4]: 1014 (expected 1014)
>> [2] baseptr[0]: 1005 (expected 1010) [!!!]
>> [2] baseptr[1]: 1006 (expected 1011) [!!!]
>> [2] baseptr[2]: 1007 (expected 1012) [!!!]
>> [2] baseptr[3]: 1008 (expected 1013) [!!!]
>> [2] baseptr[4]: 1009 (expected 1014) [!!!]
>> [3] baseptr[0]: 1010 (expected 1010)
>> [0] baseptr[0]: 1010 (expected 1010)
>> [0] baseptr[1]: 1011 (expected 1011)
>> [0] baseptr[2]: 1012 (expected 1012)
>> [0] baseptr[3]: 1013 (expected 1013)
>> [0] baseptr[4]: 1014 (expected 1014)
>> [3] baseptr[1]: 1011 (expected 1011)
>> [3] baseptr[2]: 1012 (expected 1012)
>> [3] baseptr[3]: 1013 (expected 1013)
>> [3] baseptr[4]: 1014 (expected 1014)
>> ```
>>
>> Each process should hold the same values but sometimes (not on all
>> executions) random processes diverge (marked through [!!!]).
>>
>> I made the following observations:
>>
>> 1) The issue occurs with both OpenMPI 1.10.6 and 2.0.2 but not with MPICH
>> 3.2.
>> 2) The issue occurs only if the window is allocated through
>> MPI_Win_allocate_shared, using MPI_Win_allocate works fine.
>> 3) The code assumes that MPI_Accumulate atomically updates individual
>> elements (please correct me if that is not covered by the MPI standard).
>>
>> Both OpenMPI and the example code were compiled using GCC 5.4.1 and run
>> on a Linux system (single node). OpenMPI was configure with
>> --enable-mpi-thread-multiple and --with-threads but the application is not
>> multi-threaded. Please let me know if you need any other information.
>>
>> Cheers
>> Joseph
>>
>> --
>> Dipl.-Inf. Joseph Schuchart
>> High Performance Computing Center Stuttgart (HLRS)
>> Nobelstr. 19
>> D-70569 Stuttgart
>>
>> Tel.: +49(0)711-68565890
>> Fax: +49(0)711-6856832
>> E-Mail: schuch...@hlrs.de
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>
>
>
>
> ___
> users mailing 
> listus...@lists.open-mpi.orghttps://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
>
> --
> Dipl.-Inf. Joseph Schuchart
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstr. 19
> D-70569 Stuttgart
>
> Tel.: +49(0)711-68565890 <+49%20711%2068565890>
> Fax: +49(0)711-6856832 <+49%20711%206856832>
> E-Mail: schuch...@hlrs.de
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] sharedfp/lockedfile collision between multiple program instances

[Edgar Gabriel]

done.  I added 2.0.3 as the milestone, since I am not sure what the 
timeline for 2.1.0 is. I will try to get the fix in over the weekened, 
and we can go from there.


Tanks

Edga


On 3/3/2017 9:00 AM, Howard Pritchard wrote:

Hi Edgar

Please open an issue too so we can track the fix.

Howard


Edgar Gabriel > schrieb am Fr. 3. März 2017 um 07:45:


Nicolas,

thank you for the bug report, I can confirm the behavior. I will
work on
a patch and will try to get that into the next release, should
hopefully
not be too complicated.

Thanks

Edgar


On 3/3/2017 7:36 AM, Nicolas Joly wrote:
> Hi,
>
> We just got hit by a problem with sharedfp/lockedfile component
under
> v2.0.1 (should be identical with v2.0.2). We had 2 instances of
an MPI
> program running conccurrently on the same input file and using
> MPI_File_read_shared() function ...
>
> If the shared file pointer is maintained with the lockedfile
> component, a "XXX.lockedfile" is created near to the data
> file. Unfortunately, this fixed name will collide with multiple
tools
> instances ;)
>
> Running 2 instances of the following command line (source code
> attached) on the same machine will show the problematic behaviour.
>
> mpirun -n 1 --mca sharedfp lockedfile ./shrread -v input.dat
>
> Confirmed with lsof(8) output :
>
> njoly@tars [~]> lsof input.dat.lockedfile
> COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> shrread 5876 njoly   21w   REG   0,308 13510798885996031
input.dat.lockedfile
> shrread 5884 njoly   21w   REG   0,308 13510798885996031
input.dat.lockedfile
>
> Thanks in advance.
>

___
users mailing list
users@lists.open-mpi.org 
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] sharedfp/lockedfile collision between multiple program instances

[Howard Pritchard]

Hi Edgar

Please open an issue too so we can track the fix.

Howard


Edgar Gabriel  schrieb am Fr. 3. März 2017 um
07:45:

> Nicolas,
>
> thank you for the bug report, I can confirm the behavior. I will work on
> a patch and will try to get that into the next release, should hopefully
> not be too complicated.
>
> Thanks
>
> Edgar
>
>
> On 3/3/2017 7:36 AM, Nicolas Joly wrote:
> > Hi,
> >
> > We just got hit by a problem with sharedfp/lockedfile component under
> > v2.0.1 (should be identical with v2.0.2). We had 2 instances of an MPI
> > program running conccurrently on the same input file and using
> > MPI_File_read_shared() function ...
> >
> > If the shared file pointer is maintained with the lockedfile
> > component, a "XXX.lockedfile" is created near to the data
> > file. Unfortunately, this fixed name will collide with multiple tools
> > instances ;)
> >
> > Running 2 instances of the following command line (source code
> > attached) on the same machine will show the problematic behaviour.
> >
> > mpirun -n 1 --mca sharedfp lockedfile ./shrread -v input.dat
> >
> > Confirmed with lsof(8) output :
> >
> > njoly@tars [~]> lsof input.dat.lockedfile
> > COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> > shrread 5876 njoly   21w   REG   0,308 13510798885996031
> input.dat.lockedfile
> > shrread 5884 njoly   21w   REG   0,308 13510798885996031
> input.dat.lockedfile
> >
> > Thanks in advance.
> >
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] sharedfp/lockedfile collision between multiple program instances

[Edgar Gabriel]

Nicolas,

thank you for the bug report, I can confirm the behavior. I will work on 
a patch and will try to get that into the next release, should hopefully 
not be too complicated.


Thanks

Edgar


On 3/3/2017 7:36 AM, Nicolas Joly wrote:

Hi,

We just got hit by a problem with sharedfp/lockedfile component under
v2.0.1 (should be identical with v2.0.2). We had 2 instances of an MPI
program running conccurrently on the same input file and using
MPI_File_read_shared() function ...

If the shared file pointer is maintained with the lockedfile
component, a "XXX.lockedfile" is created near to the data
file. Unfortunately, this fixed name will collide with multiple tools
instances ;)

Running 2 instances of the following command line (source code
attached) on the same machine will show the problematic behaviour.

mpirun -n 1 --mca sharedfp lockedfile ./shrread -v input.dat

Confirmed with lsof(8) output :

njoly@tars [~]> lsof input.dat.lockedfile
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
shrread 5876 njoly   21w   REG   0,308 13510798885996031 
input.dat.lockedfile
shrread 5884 njoly   21w   REG   0,308 13510798885996031 
input.dat.lockedfile

Thanks in advance.



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] sharedfp/lockedfile collision between multiple program instances

[Nicolas Joly]

Hi,

We just got hit by a problem with sharedfp/lockedfile component under
v2.0.1 (should be identical with v2.0.2). We had 2 instances of an MPI
program running conccurrently on the same input file and using
MPI_File_read_shared() function ...

If the shared file pointer is maintained with the lockedfile
component, a "XXX.lockedfile" is created near to the data
file. Unfortunately, this fixed name will collide with multiple tools
instances ;)

Running 2 instances of the following command line (source code
attached) on the same machine will show the problematic behaviour.

mpirun -n 1 --mca sharedfp lockedfile ./shrread -v input.dat

Confirmed with lsof(8) output :

njoly@tars [~]> lsof input.dat.lockedfile
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
shrread 5876 njoly   21w   REG   0,308 13510798885996031 
input.dat.lockedfile
shrread 5884 njoly   21w   REG   0,308 13510798885996031 
input.dat.lockedfile

Thanks in advance.

-- 
Nicolas Joly

Cluster & Computing Group
Biology IT Center
Institut Pasteur, Paris.

#include 
#include 

#include 
#include 
#include 
#include 
#include 

#include 

int main(int argc, char **argv) {
  char *rbuf, *file;
  int i, res, cnt, dry, inf;
  size_t len;
  struct statvfs stv; 
  struct utsname nam;

  MPI_Comm comm = MPI_COMM_WORLD;
  MPI_Info info = MPI_INFO_NULL;
  MPI_File rfh;
  MPI_Offset size, read, rsum;
  MPI_Status sts;

  dry = inf = 0;
  while ((i = getopt(argc, argv, "nv")) != -1) {
switch (i) {
case 'n': dry++; break; /* dryrun */
case 'v': inf++; break; /* verbose */
default: return EXIT_FAILURE; }
  }

  if (argc - optind != 1)
errx(1, "usage: %s ", *argv);
  file = *(argv+optind);

  res = statvfs(file, );
  assert(res == 0);

  len = 10UL * 1024 * 1024;
  len -= (len % stv.f_bsize);
  rbuf = malloc(len);
  assert(rbuf != NULL);

  res = MPI_Init(, );
  assert(res == MPI_SUCCESS);

  res = MPI_File_open(comm, file, MPI_MODE_RDONLY, info, );
  assert(res == MPI_SUCCESS);
  res = MPI_File_get_size(rfh, );
  assert(res == MPI_SUCCESS);

  read = 0;
  while (1) {

res = MPI_File_read_shared(rfh, rbuf, (int)len, MPI_CHAR, );
assert(res == MPI_SUCCESS);
res = MPI_Get_count(, MPI_CHAR, );
assert(res == MPI_SUCCESS);
if (cnt == 0) break;

read += cnt;
assert(read <= size);
  }

  res = MPI_File_close();
  assert(res == MPI_SUCCESS);

  rsum = 0;
  res = MPI_Allreduce(, , 1, MPI_OFFSET, MPI_SUM, comm);
  assert(res == MPI_SUCCESS);

  res = MPI_Finalize();
  assert(res == MPI_SUCCESS);

  free(rbuf);

  if (inf != 0) {
assert(uname() == 0);
printf("%s: %lld/%lld/%lld\n", nam.nodename, read, rsum, size); }
  assert(rsum == size);

  return 0; }
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Paul Kapinos]

Hi,

On 03/03/17 12:41, Mark Dixon wrote:

Your 20% memory bandwidth performance hit on 2.x and the OPA problem are
concerning - will look at that. Are there tickets open for them?


OPA performance issue on CP2K (15x slowdown) :
https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html
(cf. the thread) workaround is to disable IB failback on OPA,
> --mca btl ^tcp,openib
With this  tweak on OPA, OpenMPI's CP2K is less than 10% slower than Intel MPI's 
(the same result as on InfiniBand) - which is much much better that 1500%, huh. 
However Open MPI's CP2K still stays slower than Intel MPI's due to worse 
MPI_Alltoallv, as far as I understood the profiles.

I will mail to CP2K developers soon...



20% bandwidth with Open MPI 2.x: cf.
https://www.mail-archive.com/devel@lists.open-mpi.org/msg00043.html
- Nathan Hjelm mean the hooks are removed by intention. We have a (nasty) 
workaround, cf.

https://www.mail-archive.com/devel@lists.open-mpi.org/msg00052.html
As far as I can see this issue is on InfiniBand only.


Best

Paul

--
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, IT Center
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915



smime.p7s
Description: S/MIME Cryptographic Signature
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Mark Dixon]

On Fri, 3 Mar 2017, Paul Kapinos wrote:
...
Note that on 1.10.x series (even on 1.10.6), enabling of 
MPI_THREAD_MULTIPLE in lead to (silent) shutdown of the InfiniBand 
fabric for that application => SLOW!


2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand 
the right way up, however due to absence of memory hooks (= nut aligned 
memory allocation) we get 20% less bandwidth on IB with 2.x versions 
compared to 1.10.x versions of Open MPI (regardless with or without 
support of MPI_THREAD_MULTIPLE).


On Intel OmniPath network both above issues seem to be not present, but 
due to a performance bug in MPI_Free_mem your application can be 
horribly slow (seen: CP2K) if the InfiniBand failback of OPA not 
disabled manually, see 
https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html

...

Hi Paul,

All very useful - thanks :)

Our (limited) testing seems to show no difference on 2.x with 
MPI_THREAD_MULTIPLE enabled vs. disabled as well, which is good news. Glad 
to hear another opinion.


Your 20% memory bandwidth performance hit on 2.x and the OPA problem are 
concerning - will look at that. Are there tickets open for them?


Cheers,

Mark
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Paul Kapinos]

Hi Mark,


On 02/18/17 09:14, Mark Dixon wrote:

On Fri, 17 Feb 2017, r...@open-mpi.org wrote:


Depends on the version, but if you are using something in the v2.x range, you
should be okay with just one installed version




How good is MPI_THREAD_MULTIPLE support these days and how far up the wishlist
is it, please?


Note that on 1.10.x series (even on 1.10.6), enabling of MPI_THREAD_MULTIPLE in 
lead to (silent) shutdown of the InfiniBand fabric for that application => SLOW!


2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand the right 
way up, however due to absence of memory hooks (= nut aligned memory allocation) 
we get 20% less bandwidth on IB with 2.x versions compared to 1.10.x versions of 
Open MPI (regardless with or without support of MPI_THREAD_MULTIPLE).


On Intel OmniPath network both above issues seem to be not present, but due to a 
performance bug in MPI_Free_mem your application can be horribly slow (seen: 
CP2K) if the InfiniBand failback of OPA not disabled manually, see

https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html

Best,

Paul Kapinos



--
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, IT Center
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915



smime.p7s
Description: S/MIME Cryptographic Signature
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users