Re: [OMPI users] Addendum to: Assembler instruction errors for push and pop during make

[Jeremiah Willcock]

The file win_compat.h seems to be very strange (many #defines of function 
names, #defines of type names rather than typedefs, etc.).  It might make 
sense to avoid including it entirely for MinGW (it is included from 
opal/include/opal_config_bottom.h), or edit it to be correct for 64-bit 
systems.  You might want to try commenting out the entire body of 
win_compat.h and re-enabling only those parts that are truly necessary 
(and don't have MinGW headers that should be used instead, such as for 
ssize_t).


-- Jeremiah Willcock

 On Tue, 20 Aug 2013, Richard Haney wrote:

Ah!  Thanks, Jeff.  Here is a link to the relevant zip file 
"openmpi-1.6.5_configure_and_make.zip".  It contains the
modified configure (essentially replacing the compound "if" statement that 
assigns ompi_cv_asm_arch="IA32"
because my "long" is size 4); the replacement is the simply the alternative 
assignment that configure would chose if

my "long" size were not 4:
ompi_cv_asm_arch="AMD64"

The zip file contains:
configure_modified
config.log openmpi-1.6.5_configure.out   (the output of configure executed 
from within another script)

make_-j_2.out   (the output from the parallel attempt at make), and
make.out--(nonparallel make).out

The make command still does not get past the building of of asm.c in 
/openmpi-1.6.5/opal/asm


I ran the nonparallel make to eliminate any confusion over the parallel 
creation of object file atomic-asm.o in

subdirectory .libs by command
  CPPAS  atomic-asm.lo
at the same time that the command
  CC     asm.lo
was about to "crash" with lots of warnings and with seemingly uninformative 
"Error" messages.



On Tue, Aug 20, 2013 at 2:20 PM, Jeff Squyres (jsquyres) 
 wrote:

  Google drive?

[OMPI users] Call For Participation: EuroMPI 2013. Madrid, Spain. September 15-18th

[Javier Garcia Blas]

Dear Sir or Madam,

(We apologize if you receive multiple copies of this message)



Recent Advances in Message Passing Interface. 20th European MPI Users' Group 
Meeting (EuroMPI 2013) 

EuroMPI 2013 is being held in cooperation with SIGHPC 

Madrid, Spain, September 15-18, 2013 

www.eurompi2013.org

BACKGROUND AND TOPICS 
--- 

EuroMPI is the preeminent meeting for users, developers and researchers to 
interact and discuss new developments and applications of message-passing 
parallel computing, in particular in and related to the Message Passing 
Interface (MPI). The annual meeting has a long, rich tradition, and the 20th 
European MPI Users' Group Meeting will again be a lively forum for discussion 
of everything related to usage and implementation of MPI and other parallel 
programming interfaces. Traditionally, the meeting has focused on the efficient 
implementation of aspects of MPI, typically on high-performance computing 
platforms, benchmarking and tools for MPI, short-comings and extensions of MPI, 
parallel I/O and fault tolerance, as well as parallel applications using MPI. 
The meeting is open towards other topics, in particular application experience 
and alternative interfaces for high-performance heterogeneous, hybrid, 
distributed memory systems. 

The meeting will feature contributed talks on the selected, peer-reviewed 
papers, invited expert talks covering upcoming and future issues, a vendor 
session where selected vendors will present their new developments in hybrid 
and heterogeneous cluster and high-performance architectures, a poster session, 
and a tutorial day. 

TUTORIALS
--- 

- Understanding applications performance with Paraver

Writing parallel applications that make a good use of the resources is not an 
easy task. Performance analysis tools support developers on the evaluation, 
tuning and optimization of their codes. In this tutorial we will describe the 
performance tools developed at BSC (www.bsc.es/performance_tools), an 
open-source project targeting not only to detect performance problems but to 
understand the applications' behavior. The key component is Paraver, a 
performance analyzer based on traces with a great flexibility to explore the 
collected data. The Dimemas simulator can be used to predict the application's 
behavior under different scenarios. Provided with such tools, the analyst can 
generate and validate hypothesis to investigate the trails provided by the 
execution trace.

- Using Coprocessors in MPI programs

Implementing the high performance version 2.2 of the Message Passing 
Interface(MPI)-2 specification on multiple fabrics, Intel® MPI Library 4.1 
focuses on making applications perform better on IA-based clusters. The ease of 
use of Intel® MPI and analysis tools like the Intel® Trace Analyzer and 
Collector is available for both the Intel® Xeon and Intel® Many Integrated 
Core (MIC) architecture. Intel® MPI Library supports different programming 
models on Xeon Ph(TM) coprocessors and we will demonstrate these models with 
examples in addition to the basics of debugging Intel® MPI codes on Xeon 
Ph(TM) coprocessors.
 
In this tutorial, as a starting point, you will learn how a basic MPI program 
can utilize the card as a coprocessor by leveraging the offload model where all 
MPI ranks are run on the main Xeon host, and the application utilizes offload 
directives to run on the Intel Xeon Phi coprocessor card. Well-known threading 
technologies, such as OpenMP, can implement the offload regions.  We'll then 
focus on how natively-compiled MPI applications can be executed directly on the 
card.  The final method explored will emphasize how to run on a mix of 
architectures via distribution of the ranks on both the coprocessor card and 
the host node.  As part of the final segment, we'll also discuss how to analyze 
the load balance of an MPI application which runs in such a heterogeneous 
environment.
 

KEYNOTE SPEAKERS
--- 

- Prof. Alok N. Choudhary: Exascale Systems, Big Data and Knowledge Discovery - 
Challenges and Opportunities

The primary goal of exascale systems is to accelerate scientific discoveries 
and engineering innovation, yet the impact of exascale will not only be 
measured by how fast a single simulation is performed, but rather by the speed 
and acceleration on overall knowledge discoveries. Modern experiments and 
simulations involving satellites, telescopes, high-throughput instruments, 
imaging devices, sensor networks, accelerators, and supercomputers yield 
massive amounts of data. That is, processing, mining and analyzing this data 
effectively and efficiently will be a critical component of the knowledge 
discovery process, as we can no longer rely upon traditional ways of dealing 
with the data due to its scale and speed. Interestingly, an exascale system can 

Re: [OMPI users] Addendum to: Assembler instruction errors for push and pop during make

[Richard Haney]

Ah!  Thanks, Jeff.  Here is a link to the relevant zip file "
openmpi-1.6.5_configure_and_make.zip".
 It contains the modified configure (essentially replacing the compound
"if" statement that assigns
ompi_cv_asm_arch="IA32"
because my "long" is size 4); the replacement is the simply the alternative
assignment that configure would chose if my "long" size were not 4:
ompi_cv_asm_arch="AMD64"

The zip file contains:
configure_modified
config.log openmpi-1.6.5_configure.out   (the output of configure executed
from within another script)
make_-j_2.out   (the output from the parallel attempt at make), and
make.out--(nonparallel make).out

The make command still does not get past the building of of asm.c in
/openmpi-1.6.5/opal/asm

I ran the nonparallel make to eliminate any confusion over the parallel
creation of object file atomic-asm.o in subdirectory .libs by command
  CPPAS  atomic-asm.lo
at the same time that the command
  CC asm.lo
was about to "crash" with lots of warnings and with seemingly uninformative
"Error" messages.


On Tue, Aug 20, 2013 at 2:20 PM, Jeff Squyres (jsquyres)  wrote:

> Google drive?
>
>

Re: [OMPI users] Addendum to: Assembler instruction errors for push and pop during make

[Jeff Squyres (jsquyres)]

Google drive?

On Aug 20, 2013, at 5:13 PM, Richard Haney  wrote:

> Thanks, Jeff, for the info and suggestion.
> 
> I know of no web site where I could post the compressed files.  I tried to 
> create a blog at http://www.blogger.com/home to post the info, but it does 
> not seem to take compressed-file uploads.  I then tried copying text to the 
> blog and the screen went "Aw snap!".  So it looks like that resource will not 
> be much help either.
> 
> Richard Haney
> 
> On Tue, Aug 20, 2013 at 12:58 PM, Jeff Squyres (jsquyres) 
>  wrote:
> FWIW, the only restriction we have on the OMPI list is a max size of the 
> message (including attachments).  That might be a bit arcane and from when 
> people didn't have big mailboxes, but... oh well.  :-\
> 
> Can you just post the attachment somewhere we can all download it?
> 
> ...
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Addendum to: Assembler instruction errors for push and pop during make

[Richard Haney]

Thanks, Jeff, for the info and suggestion.

I know of no web site where I could post the compressed files.  I tried to
create a blog at http://www.blogger.com/home to post the info, but it does
not seem to take compressed-file uploads.  I then tried copying text to the
blog and the screen went "Aw snap!".  So it looks like that resource will
not be much help either.

Richard Haney

On Tue, Aug 20, 2013 at 12:58 PM, Jeff Squyres (jsquyres) <
jsquy...@cisco.com> wrote:

> FWIW, the only restriction we have on the OMPI list is a max size of the
> message (including attachments).  That might be a bit arcane and from when
> people didn't have big mailboxes, but... oh well.  :-\
>
> Can you just post the attachment somewhere we can all download it?
>
> ...
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>

Re: [OMPI users] MPI_Init_thread hangs in OpenMPI 1.7.1 when using --enable-mpi-thread-multiple

[Ralph Castain]

On Aug 20, 2013, at 12:40 PM, RoboBeans  wrote:

> I can confirm that mpi program still hangs if one uses these options while 
> configuring openmpi-1.7.2
> 
> --enable-event-thread-support --enable-opal-multi-threads 
> --enable-orte-progress-threads --enable-mpi-thread-multiple 
> 
> 
> cd /opt
> gunzip -c openmpi-1.7.2.tar.gz | tar xf -
> cd openmpi-1.7.2
> ./configure --with-openib=/usr --with-psm=/usr --prefix=/opt/openmpi-1.7.2 
> --enable-event-thread-support --enable-opal-multi-threads 
> --enable-orte-progress-threads --enable-mpi-thread-multiple
> make all install
> cd ..
> chmod -R 775 openmpi-1.7.2
> ldconfig
> 

Yep, sure will


> Is there any plan to support multiple threads any time soon?

Not in the immediate future, but hopefully not too far into infinity - however, 
we should fix the hang in the near future, at least

> 
> $ uname -r 
> 
> 2.6.32-358.14.1.el6.x86_64
> 
> 
> Thanks!
> 
> On 6/19/13 12:51 PM, Hans Ekkehard Plesser wrote:
>> I can confirm that
>> 
>> --enable-event-thread-support --enable-opal-multi-threads 
>> --enable-orte-progress-threads --enable-mpi-thread-multiple
>> 
>> worked for me as well under OSX 10.8.4 with clang. The orte-progress flag is 
>> needed, without it the parallel process hangs.
>> 
>> I will report this to the MacPorts people so they can fix it.
>> 
>> Best,
>> Hans
>> 
>> On 19. juni 2013, at 15.08, Ralph Castain wrote:
>> 
>>> While I'm glad it worked, having it depend on orte progress thread wasn't 
>>> done by intent, hence the comment in the configure help output. I'm not 
>>> seeing any problems with using that option, but please let us know if you 
>>> run into something. It hasn't been as fully tested as we'd like.
>>> 
>>> 
>>> On Jun 19, 2013, at 12:10 AM, Elias Rudberg  wrote:
>>> 
 This has now been solved.
 
 The solution (as suggested by Ralph) is to use the following four 
 configure options together:
 ./configure --enable_event_thread_support --enable_opal_multi_threads 
 --enable_orte_progress_threads --enable_mpi_thread_multiple
 
 So, apparently the --enable_mpi_thread_multiple option cannot be used by 
 itself, it must be combined with the other three options, otherwise 
 MPI_Init_thread will hang.
 
 / Elias
 
 
 Quoting Elias Rudberg :
 
> Hi,
> 
> I now tried using --enable-opal-multi-threads (I added an "s" in the end, 
> since "--enable-opal-multi-thread" was not recognized by the configure 
> script).
> 
> However, my test code still hangs in the same way as before.
> 
> To be clear, I now configure like this:
> ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads
> 
> but the problem remains: MPI_Init_thread still hangs when trying to run 
> the test case with -np 2.
> 
> Any other ideas?
> Are there other configure options that must be used together with 
> --enable-mpi-thread-multiple to make it work?
> 
> Best regards,
> Elias
> 
> 
> Quoting Ralph Castain :
> 
>> Hmmm...well, your code runs fine for me:
>> 
>> Ralphs-iMac:mpi rhc$ mpirun -n 2 ./thread_init
>> Calling MPI_Init_thread...
>> Calling MPI_Init_thread...
>> MPI_Init_thread returned, provided = 3
>> MPI_Init_thread returned, provided = 3
>> Ralphs-iMac:mpi rhc$
>> 
>> I think the key, however, is that you also have to configure with 
>> --enable-opal-multi-thread.
>> 
>> 
>> On Jun 16, 2013, at 8:54 AM, Elias Rudberg  
>> wrote:
>> 
>>> Hello!
>>> 
>>> I would like to report what seems to be a bug in MPI_Init_thread in 
>>> OpenMPI 1.7.1.
>>> 
>>> The bug can be reproduced with the following test program 
>>> (test_mpi_thread_support.c):
>>> ===
>>> #include 
>>> #include 
>>> int main(int argc, const char* argv[]) {
>>> int provided = -1;
>>> printf("Calling MPI_Init_thread...\n");
>>> MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, );
>>> printf("MPI_Init_thread returned, provided = %d\n", provided);
>>> MPI_Finalize();
>>> return 0;
>>> }
>>> ===
>>> 
>>> When trying to run this when OpenMPI was configured with 
>>> --enable-mpi-thread-multiple, the program hangs when trying to run 
>>> using anything more than one process.
>>> 
>>> Steps I use to reproduce this in Ubuntu:
>>> 
>>> (1) Download openmpi-1.7.1.tar.gz
>>> 
>>> (2) Configure like this:
>>> ./configure --enable-mpi-thread-multiple
>>> 
>>> (3) make
>>> 
>>> (4) Compile test program like this:
>>> mpicc test_mpi_thread_support.c
>>> 
>>> (5) Run like this:
>>> mpirun -np 2 ./a.out
>>> Then you see the following two 

Re: [OMPI users] Addendum to: Assembler instruction errors for push and pop during make

[Jeff Squyres (jsquyres)]

FWIW, the only restriction we have on the OMPI list is a max size of the 
message (including attachments).  That might be a bit arcane and from when 
people didn't have big mailboxes, but... oh well.  :-\

Can you just post the attachment somewhere we can all download it?


On Aug 20, 2013, at 3:56 PM, Richard Haney  wrote:

> This is my fourth attempt to get past the robot censor.  Not even text format 
> passed.  If this message passes the censor, it is likely because attachments 
> are considered part of the "body" of the message, which the censor seems to 
> object to due to size.  I am deleting the attachments.
> 
> Re-sending:
> 
> On Tue, Aug 20, 2013 at 12:16 PM, Richard Haney  wrote:
> Thanks  Jeremiah,
> 
> Your comments were most helpful.
> 
> I did find where configure sets
> 
> ompi_cv_asm_arch="IA32"
> 
> and subsequently prints out [sic]
> 
> checking for asssembly architecture... IA32
> 
> I tried changing the 'ompi_cv_asm_arch="IA32" ' to 
> 
> ompi_cv_asm_arch="AMD64"
> 
> which was an alternative not chosen in the original configure because my long 
> has size 4 -- a very puzzling excuse for not using "AMD64".
> 
> A more complete message from me "awaits moderator approval".
> 
> It is a puzzle to me why the OpenMPI issue-reporting info asks for lots and 
> lots of info, including several compressed files of rather large files and 
> yet the listserv seems to have rather stringent size limits on the body of 
> messages.
> 
> It seems as if the original designers and developers and maintainers had 
> grand, ambitious ideas for OpenMPI but that now the maintainers have 
> considerably deflated ambitions and hopes for OpenMPI.
> 
> I am new to OpenMPI and I am only trying to build and install it because an 
> opt++2.4 numerical optimization package I want to install says it has 
> facilities to use any installed MPI.
> 
> But I am having sooo much trouble with trying to build the package -- 
> specifically trying to get past the make of asm.c in /openmpi-1.6.5/opal/asm  
> -- that I'm wondering if it is worthwhile.
> 
> Richard Haney
> 
> 
> On Mon, Aug 19, 2013 at 8:04 AM, Jeremiah Willcock  
> wrote:
>   
> ...
>  
> Are you sure that the code is building with the right assembly file? 
> AMD64.asm looks to be using pushq, which is the right form to use for 64-bit 
> code.  I suspect that this is a configure issue (detecting your architecture 
> as IA-32 and not x86-64), not a problem in the assembly files themselves.
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Addendum to: Assembler instruction errors for push and pop during make

[Richard Haney]

This is my fourth attempt to get past the robot censor.  Not even text
format passed.  If this message passes the censor, it is likely because
attachments are considered part of the "body" of the message, which the
censor seems to object to due to size.  I am deleting the attachments.

Re-sending:

On Tue, Aug 20, 2013 at 12:16 PM, Richard Haney  wrote:

> Thanks  Jeremiah,
>
> Your comments were most helpful.
>
> I did find where configure sets
>
> ompi_cv_asm_arch="IA32"
>
> and subsequently prints out [sic]
>
> checking for asssembly architecture... IA32
>
> I tried changing the 'ompi_cv_asm_arch="IA32" ' to
>
> ompi_cv_asm_arch="AMD64"
>
> which was an alternative not chosen in the original configure because my
> long has size 4 -- a very puzzling excuse for not using "AMD64".
>
> A more complete message from me "awaits moderator approval".
>
> It is a puzzle to me why the OpenMPI issue-reporting info asks for lots
> and lots of info, including several compressed files of rather large files
> and yet the listserv seems to have rather stringent size limits on the body
> of messages.
>
> It seems as if the original designers and developers and maintainers had
> grand, ambitious ideas for OpenMPI but that now the maintainers have
> considerably deflated ambitions and hopes for OpenMPI.
>
> I am new to OpenMPI and I am only trying to build and install it because
> an opt++2.4 numerical optimization package I want to install says it has
> facilities to use any installed MPI.
>
>  But I am having sooo much trouble with trying to build the package
> -- specifically trying to get past the make of asm.c in
> /openmpi-1.6.5/opal/asm  -- that I'm wondering if it is worthwhile.
>
> Richard Haney
>
>
> On Mon, Aug 19, 2013 at 8:04 AM, Jeremiah Willcock wrote:
>
>
>> ...
>>
>
>
>> Are you sure that the code is building with the right assembly file?
>> AMD64.asm looks to be using pushq, which is the right form to use for
>> 64-bit code.  I suspect that this is a configure issue (detecting your
>> architecture as IA-32 and not x86-64), not a problem in the assembly files
>> themselves.
>>
>>

Re: [OMPI users] MPI_Init_thread hangs in OpenMPI 1.7.1 when using --enable-mpi-thread-multiple

[RoboBeans]

I can confirm that mpi program still hangs if one uses these options 
while configuring openmpi-1.7.2


--enable-event-thread-support --enable-opal-multi-threads 
--enable-orte-progress-threads --enable-mpi-thread-multiple



*cd /opt**
**gunzip -c openmpi-1.7.2.tar.gz | tar xf -**
**cd openmpi-1.7.2**
**./configure --with-openib=/usr --with-psm=/usr 
--prefix=/opt/openmpi-1.7.2 --enable-event-thread-support 
--enable-opal-multi-threads --enable-orte-progress-threads 
--enable-mpi-thread-multiple**

**make all install**
**cd ..**
**chmod -R 775 openmpi-1.7.2**
**ldconfig*

Is there any plan to support multiple threads any time soon?

*$ uname -r*

2.6.32-358.14.1.el6.x86_64


Thanks!

On 6/19/13 12:51 PM, Hans Ekkehard Plesser wrote:

I can confirm that

--enable-event-thread-support --enable-opal-multi-threads 
--enable-orte-progress-threads --enable-mpi-thread-multiple

worked for me as well under OSX 10.8.4 with clang. The orte-progress flag is 
needed, without it the parallel process hangs.

I will report this to the MacPorts people so they can fix it.

Best,
Hans

On 19. juni 2013, at 15.08, Ralph Castain wrote:


While I'm glad it worked, having it depend on orte progress thread wasn't done 
by intent, hence the comment in the configure help output. I'm not seeing any 
problems with using that option, but please let us know if you run into 
something. It hasn't been as fully tested as we'd like.


On Jun 19, 2013, at 12:10 AM, Elias Rudberg  wrote:


This has now been solved.

The solution (as suggested by Ralph) is to use the following four configure 
options together:
./configure --enable_event_thread_support --enable_opal_multi_threads 
--enable_orte_progress_threads --enable_mpi_thread_multiple

So, apparently the --enable_mpi_thread_multiple option cannot be used by 
itself, it must be combined with the other three options, otherwise 
MPI_Init_thread will hang.

/ Elias


Quoting Elias Rudberg :


Hi,

I now tried using --enable-opal-multi-threads (I added an "s" in the end, since 
"--enable-opal-multi-thread" was not recognized by the configure script).

However, my test code still hangs in the same way as before.

To be clear, I now configure like this:
./configure --enable-mpi-thread-multiple --enable-opal-multi-threads

but the problem remains: MPI_Init_thread still hangs when trying to run the 
test case with -np 2.

Any other ideas?
Are there other configure options that must be used together with 
--enable-mpi-thread-multiple to make it work?

Best regards,
Elias


Quoting Ralph Castain :


Hmmm...well, your code runs fine for me:

Ralphs-iMac:mpi rhc$ mpirun -n 2 ./thread_init
Calling MPI_Init_thread...
Calling MPI_Init_thread...
MPI_Init_thread returned, provided = 3
MPI_Init_thread returned, provided = 3
Ralphs-iMac:mpi rhc$

I think the key, however, is that you also have to configure with 
--enable-opal-multi-thread.


On Jun 16, 2013, at 8:54 AM, Elias Rudberg  wrote:


Hello!

I would like to report what seems to be a bug in MPI_Init_thread in OpenMPI 
1.7.1.

The bug can be reproduced with the following test program 
(test_mpi_thread_support.c):
===
#include 
#include 
int main(int argc, const char* argv[]) {
int provided = -1;
printf("Calling MPI_Init_thread...\n");
MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, );
printf("MPI_Init_thread returned, provided = %d\n", provided);
MPI_Finalize();
return 0;
}
===

When trying to run this when OpenMPI was configured with 
--enable-mpi-thread-multiple, the program hangs when trying to run using 
anything more than one process.

Steps I use to reproduce this in Ubuntu:

(1) Download openmpi-1.7.1.tar.gz

(2) Configure like this:
./configure --enable-mpi-thread-multiple

(3) make

(4) Compile test program like this:
mpicc test_mpi_thread_support.c

(5) Run like this:
mpirun -np 2 ./a.out
Then you see the following two lines of output:
Calling MPI_Init_thread...
Calling MPI_Init_thread...
And then it hangs.

MPI_Init_thread did not hang in earlier OpenMPI versions (for example it worked 
in 1.5.* and 1.6.*), so it seems like a bug introduced in 1.7.

The description above shows how I reproduce this in Ubuntu on my local desktop 
computer, but the same problem exists for the OpenMPI 1.7.1 installation at the 
UPPMAX computer center where I wan to run my code in the end. I don't know all 
details about how they installed it there, but I know they set 
--enable-mpi-thread-multiple. So maybe it hangs in 1.7.1 on any computer as 
long as you use MPI_THREAD_MULTIPLE. At least I have not seen it work anywhere.

Do you agree that this is a bug, or am I doing something wrong?

Best regards,
Elias


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Mixing Linux's CPU-shielding with mpirun's bind-to-core

[Jeff Squyres (jsquyres)]

On Aug 20, 2013, at 1:18 PM, "Jeff Squyres (jsquyres)"  
wrote:

> $ cat my_binding_script.sh
> #!/bin/sh
> exec hwloc-bind socket.1:core.$OMPI_COMM_WORLD_LOCAL_RANK $1


Oops!  Typo.  That last line should be:

exec hwloc-bind socket:1.core:$OMPI_COMM_WORLD_LOCAL_RANK $1

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Mixing Linux's CPU-shielding with mpirun's bind-to-core

[Jeff Squyres (jsquyres)]

I know I'm late to this conversation, but I was on vacation last week.  Some 
random points:

1. If you use OMPI's --bind-to-core option and then re-bind yourself to some 
other core, then all the memory affinity that MPI setup during MPI_Init() will 
be "wrong" (possibly on a remote numa node).  I would advise against doing this.

2. Instead of #1, as Ralph stated, if you're going to do your own process 
affinity, then don't use OMPI's --bind-to-core (or any --bind-to-* option).  
Then MPI won't setup any affinity stuff, and you're good.

3. Rather that setting up cpu shielding, you can just use simple API calls or 
scripting calls to bind each MPI process to wherever you want.  For example:

$ mpirun --host a,b -np 4 my_binding_script.sh my_mpi_app

Where my_binding_script.sh simply invokes a tool like hwloc-bind to bind 
yourself to whatever socket/core combination you want, and then invokes 
my_mpi_app (i.e., the real MPI application).  For example:

$ cat my_binding_script.sh
#!/bin/sh
exec hwloc-bind socket.1:core.$OMPI_COMM_WORLD_LOCAL_RANK $1

Where $OMPI_COMM_WORLD_LOCAL_RANK is an environment variable that mpirun will 
put in the environment of the processes that it launches.  Each process will 
have $OMPI_COMM_WORLD_LOCAL_RANK set to a value in the range of [0,N), where N 
is the number processes on that server.  In the above example of launching 4 
processes (2 on each server a and b), each of the 4 processes would get an 
$OMPI_COMM_WORLD_LOCAL_RANK value of 0 or 1.

If you don't know about hwloc, you should -- it's very, very helpful for all 
this kind of process affinity stuff.  See 
http://www.open-mpi.org/projects/hwloc/ (hwloc-bind is one of the tools in the 
hwloc suite).


On Aug 18, 2013, at 7:01 PM, Siddhartha Jana  wrote:

> Noted. Thanks again
> -- Sid
> 
> 
> On 18 August 2013 18:40, Ralph Castain  wrote:
> It only has to come after MPI_Init *if* you are telling mpirun to bind you as 
> well. Otherwise, you could just not tell mpirun to bind (it doesn't by 
> default) and then bind anywhere, anytime you like
> 
> 
> On Aug 18, 2013, at 3:24 PM, Siddhartha Jana  
> wrote:
> 
>> 
>> A process can always change its binding by "re-binding" to wherever it wants 
>> after MPI_Init completes.
>> Noted. Thanks. I guess the important thing that I wanted to know was that 
>> the binding needs to happen *after* MPI_Init() completes. 
>> 
>> Thanks all
>> 
>> -- Siddhartha
>> 
>>  
>> 
>> 
>> On Aug 18, 2013, at 9:38 AM, Siddhartha Jana  
>> wrote:
>> 
>>> Firstly, I would like my program to dynamically assign it self to one of 
>>> the cores it pleases and remain bound to it until it later reschedules 
>>> itself.
>>> 
>>> Ralph Castain wrote:
>>> >> "If you just want mpirun to respect an external cpuset limitation, it 
>>> >> already does so when binding - it will bind within the external 
>>> >> limitation"
>>> 
>>> In my case, the limitation is enforced "internally", by the application 
>>> once in begins execution. I enforce this during program execution, after 
>>> the mpirun has finished "binding within the external limitation". 
>>> 
>>> 
>>> Brice Goglin said:
>>> >>  "MPI can bind at two different times: inside mpirun after ssh before 
>>> >> running the actual program (this one would ignore your cpuset), later at 
>>> >> MPI_Init inside your program (this one will ignore your cpuset only if 
>>> >> you call MPI_Init before creating the cpuset)."
>>> 
>>> Noted. In that case, during program execution, whose binding is respected - 
>>> mpirun's or MPI_Init()'s? From the above, is my understanding correct? That 
>>> MPI_Init() will be responsible for the 2nd round of attempting to bind 
>>> processes to cores and can override what mpirun or the programmer had 
>>> enforced before its call (using hwloc/cpuset/sched_load_balance() and other 
>>> compatible cousins) ? 
>>> 
>>> 
>>> 
>>> If this is so, in my case the flow of events is thus:
>>> 
>>> 1. mpirun binds an MPI process which is yet to begin execution. So mpirun 
>>> says: "Bind to some core - A" (I don't use any hostfile/rankfile. but I do 
>>> use the --bind-to-core flag) 
>>> 
>>> 2. Process begins execution on core A
>>> 
>>> 3. I enforce: "Bind to core B". (we must remember, it is only at runtime 
>>> that  I know what core I want to be bound to and not while launching the 
>>> processes using mpirun). So my process shifts over to core B
>>> 
>>> 4. MPI_Init() once again honors rankfile mapping(if any, default policy, 
>>> otherwise ) and rebinds my process to core A
>>> 
>>> 5. process finished execution and calls MPI_Finalize(), all the time on 
>>> core A
>>> 
>>> 6. mpirun exits
>>> 
>>> 
>>> So if I place step-3 above after step-4, my request will hold for the rest 
>>> of the execution. Please do let me know, if my understanding 

Re: [OMPI users] Finalize() does not return

[Jeff Squyres (jsquyres)]

On Aug 14, 2013, at 9:23 AM, "Hazelrig, Chris CTR (US)" 
 wrote:

> Thanks for your suggestions.  I had already tested for which threads were 
> reaching the Finalize() call and all of them are.  Also, the Finalize() call 
> is not inside a conditional.  This seems to suggest there may be a prior 
> communication left unfinished, but based on the documentation I have read I 
> would think the Finalize() routine would error/exception out in that 
> situation.  

Sorry for the delayed reply -- I was on vacation last week.

Not necessarily -- you can definitely deadlock in Finalize if you've done a 
send that isn't matched with a receive, for example.

> It seems significant that the software was performing as expected under the 
> previous OS and OpenMPI versions (although, the older OpenMPI version is only 
> slightly older than what is being used now), but I don't know yet what the 
> differences are.

Possibly, but not definitely.  Just because an application runs properly under 
an MPI implementation does not mean that that application is correct (that 
sounds snobby, but I don't mean it that way).  Buffer allocations and blocking 
patterns change from release to release of a given MPI implementation, such 
that if you have an erroneous MPI application, it may work fine under version A 
of that MPI implementation but fail under version B of that same MPI 
implementation.

> Is there any other information I could provide that might be useful?

You might want to audit the code and ensure that you have no pending 
communications that haven't finished -- check all your sends and receives, not 
just in the code, but at run-time (e.g., use an MPI profiling tool to match up 
the sends and receives, and see what's left at Finalize time).

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

[OMPI users] Problem compiling 1.7.2 on Mac OS 10.8 with gcc 4.8

[Hugo Gagnon]

Hi,

I'm encountering this error during "make all":

Undefined symbols for architecture x86_64:
  "___atomic_fetch_add_4", referenced from:
  std::locale::locale(std::locale const&,
  SpaceSeparator*) in otfprofile-create_latex.o

See attached for the complete output.

Thanks,
-- 
  Hugo Gagnon


ompi-output.tar.bz2
Description: BZip2 compressed data