Re: [OMPI users] MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

[Gus Correa]

I never used MPI_Type_create_subarray, only MPI_Type_Vector.
What I like about MPI_Type_Vector is that you can define a stride,
hence you can address any regular pattern in memory.
However, it envisages the array layout in memory as a big 1-D array,
with a linear index progressing in either Fortran or C order.

Somebody correct me please if I am wrong, but at first sight 
MPI_Type_Vector sounds more flexible to me than 
MPI_Type_create_subarray, exactly because the latter doesn't have strides.


The downside is that you need to do some index arithmetic to figure
the right strides, etc, to match the corresponding
Fortran90 array sections.

There are good examples in the "MPI - The complete reference" books I 
suggested to you before (actually in vol 1).


Online I could find the two man pages (good information, but no example):

http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php

There is a very simple 2D example of MPI_Type_vector using strides here:

https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types

and a similar one here:

http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html

Gus Correa

On 01/15/2015 06:53 PM, Diego Avesani wrote:

dear George, dear Gus, dear all,
Could you please tell me where I can find a good example?
I am sorry but I can not understand the 3D array.


Really Thanks

Diego


On 15 January 2015 at 20:13, George Bosilca mailto:bosi...@icl.utk.edu>> wrote:



On Jan 15, 2015, at 06:02 , Diego Avesani mailto:diego.aves...@gmail.com>> wrote:

Dear Gus, Dear all,
Thanks a lot.
MPI_Type_Struct works well for the first part of my problem, so I
am very happy to be able to use it.

Regarding MPI_TYPE_VECTOR.

I have studied it and for simple case it is clear to me what id
does (at least I believe). Foe example if I have a matrix define as:
REAL, ALLOCATABLE (AA(:,:))
ALLOCATE AA(100,5)

I could send part of it defining

CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)

after that I can send part of it with

CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
MPI_COMM_WORLD );

Have I understood correctly?

What I can do in case of three dimensional array? for example
AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY.
Is that the correct way?

Thanks again


Indeed, using the subarray is the right approach independent on the
number of dimensions of the data (you can use it instead of
MPI_TYPE_VECTOR as well).

   George.







Diego


On 13 January 2015 at 19:04, Gus Correa mailto:g...@ldeo.columbia.edu>> wrote:

Hi Diego
I guess MPI_Type_Vector is the natural way to send and receive
Fortran90 array sections (e.g. your QQMLS(:,50:100,:)).
I used that before and it works just fine.
I think that is pretty standard MPI programming style.
I guess MPI_Type_Struct tries to emulate Fortran90 and C
structures
(as you did in your previous code, with all the surprises
regarding alignment, etc), not array sections.
Also, MPI type vector should be more easy going (and probably
more efficient) than MPI type struct, with less memory
alignment problems.
I hope this helps,
Gus Correa

PS - These books have a quite complete description and several
examples
of all MPI objects and functions, including MPI types (native
and user defined):
http://mitpress.mit.edu/books/__mpi-complete-reference-0

http://mitpress.mit.edu/books/__mpi-complete-reference-1


[They cover MPI 1 and 2. I guess there is a new/upcoming book
with MPI 3, but for what you're doing 1 and 2 are more than
enough.]


On 01/13/2015 09:22 AM, Diego Avesani wrote:

Dear all,

I had some wonderful talking about MPI_type_create_struct adn
isend\irecv with
Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is
more clear and my
program works.

Now I have another question. In may program I have matrix:

/QQMLS(:,:,:) /that is allocate as

/ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of
particles

QQMLS is double precision.

I would like to sent form a CPU to another part of it, for
example,
sending QQMLS(:,50:100,:). I mean sending the QQMLS of the
particles
between 50 to 100.
I suppose that i could use MPI_Type_vector but I am not
sure. The
particle that I want to sent could be from 25 to 50 ecc..
ecc..so
  blocklength changes everytime.

Do I h

Re: [OMPI users] MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

[Diego Avesani]

dear George, dear Gus, dear all,
Could you please tell me where I can find a good example?
I am sorry but I can not understand the 3D array.


Really Thanks

Diego


On 15 January 2015 at 20:13, George Bosilca  wrote:

>
> On Jan 15, 2015, at 06:02 , Diego Avesani  wrote:
>
> Dear Gus, Dear all,
> Thanks a lot.
> MPI_Type_Struct works well for the first part of my problem, so I am very
> happy to be able to use it.
>
> Regarding MPI_TYPE_VECTOR.
>
> I have studied it and for simple case it is clear to me what id does (at
> least I believe). Foe example if I have a matrix define as:
> REAL, ALLOCATABLE (AA(:,:))
> ALLOCATE AA(100,5)
>
> I could send part of it defining
>
> CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,*MY_NEW_TYPE*)
>
> after that I can send part of it with
>
> CALL MPI_SEND( AA(1:*10*,:), *10*, *MY_NEW_TYPE*, 1, 0, MPI_COMM_WORLD );
>
> Have I understood correctly?
>
> What I can do in case of three dimensional array? for example AA(:,:,:), I
> am looking to MPI_TYPE_CREATE_SUBARRAY.
> Is that the correct way?
>
> Thanks again
>
>
> Indeed, using the subarray is the right approach independent on the number
> of dimensions of the data (you can use it instead of MPI_TYPE_VECTOR as
> well).
>
>   George.
>
>
>
>
>
>
> Diego
>
>
> On 13 January 2015 at 19:04, Gus Correa  wrote:
>
>> Hi Diego
>> I guess MPI_Type_Vector is the natural way to send and receive Fortran90
>> array sections (e.g. your QQMLS(:,50:100,:)).
>> I used that before and it works just fine.
>> I think that is pretty standard MPI programming style.
>> I guess MPI_Type_Struct tries to emulate Fortran90 and C structures
>> (as you did in your previous code, with all the surprises regarding
>> alignment, etc), not array sections.
>> Also, MPI type vector should be more easy going (and probably more
>> efficient) than MPI type struct, with less memory alignment problems.
>> I hope this helps,
>> Gus Correa
>>
>> PS - These books have a quite complete description and several examples
>> of all MPI objects and functions, including MPI types (native and user
>> defined):
>> http://mitpress.mit.edu/books/mpi-complete-reference-0
>> http://mitpress.mit.edu/books/mpi-complete-reference-1
>>
>> [They cover MPI 1 and 2. I guess there is a new/upcoming book
>> with MPI 3, but for what you're doing 1 and 2 are more than enough.]
>>
>>
>> On 01/13/2015 09:22 AM, Diego Avesani wrote:
>>
>>> Dear all,
>>>
>>> I had some wonderful talking about MPI_type_create_struct adn
>>> isend\irecv with
>>> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is more clear and my
>>> program works.
>>>
>>> Now I have another question. In may program I have matrix:
>>>
>>> /QQMLS(:,:,:) /that is allocate as
>>>
>>> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of particles
>>>
>>> QQMLS is double precision.
>>>
>>> I would like to sent form a CPU to another part of it, for example,
>>> sending QQMLS(:,50:100,:). I mean sending the QQMLS of the particles
>>> between 50 to 100.
>>> I suppose that i could use MPI_Type_vector but I am not sure. The
>>> particle that I want to sent could be from 25 to 50 ecc.. ecc..so
>>>   blocklength changes everytime.
>>>
>>> Do I have to use MPI_type_create_struct?
>>> Do I have correctly understood MPI_Type_vector?
>>>
>>> Thanks a lot
>>>
>>>
>>> Diego
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: http://www.open-mpi.org/
>>> community/lists/users/2015/01/26171.php
>>>
>>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: http://www.open-mpi.org/community/lists/users/2015/01/
>> 26172.php
>>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26184.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26192.php
>

Re: [OMPI users] MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

[George Bosilca]

> On Jan 15, 2015, at 06:02 , Diego Avesani  wrote:
> 
> Dear Gus, Dear all,
> Thanks a lot.
> MPI_Type_Struct works well for the first part of my problem, so I am very 
> happy to be able to use it.
> 
> Regarding MPI_TYPE_VECTOR.
> 
> I have studied it and for simple case it is clear to me what id does (at 
> least I believe). Foe example if I have a matrix define as:
> REAL, ALLOCATABLE (AA(:,:))
> ALLOCATE AA(100,5)
> 
> I could send part of it defining
> 
> CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,MY_NEW_TYPE)
> 
> after that I can send part of it with
> 
> CALL MPI_SEND( AA(1:10,:), 10, MY_NEW_TYPE, 1, 0, MPI_COMM_WORLD );
> 
> Have I understood correctly?
> 
> What I can do in case of three dimensional array? for example AA(:,:,:), I am 
> looking to MPI_TYPE_CREATE_SUBARRAY.
> Is that the correct way?
> 
> Thanks again

Indeed, using the subarray is the right approach independent on the number of 
dimensions of the data (you can use it instead of MPI_TYPE_VECTOR as well).

  George.


> 
> 
> 
> 
> Diego
> 
> 
> On 13 January 2015 at 19:04, Gus Correa  > wrote:
> Hi Diego
> I guess MPI_Type_Vector is the natural way to send and receive Fortran90 
> array sections (e.g. your QQMLS(:,50:100,:)).
> I used that before and it works just fine.
> I think that is pretty standard MPI programming style.
> I guess MPI_Type_Struct tries to emulate Fortran90 and C structures
> (as you did in your previous code, with all the surprises regarding 
> alignment, etc), not array sections.
> Also, MPI type vector should be more easy going (and probably more efficient) 
> than MPI type struct, with less memory alignment problems.
> I hope this helps,
> Gus Correa
> 
> PS - These books have a quite complete description and several examples
> of all MPI objects and functions, including MPI types (native and user 
> defined):
> http://mitpress.mit.edu/books/mpi-complete-reference-0 
> 
> http://mitpress.mit.edu/books/mpi-complete-reference-1 
> 
> 
> [They cover MPI 1 and 2. I guess there is a new/upcoming book
> with MPI 3, but for what you're doing 1 and 2 are more than enough.]
> 
> 
> On 01/13/2015 09:22 AM, Diego Avesani wrote:
> Dear all,
> 
> I had some wonderful talking about MPI_type_create_struct adn
> isend\irecv with
> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is more clear and my
> program works.
> 
> Now I have another question. In may program I have matrix:
> 
> /QQMLS(:,:,:) /that is allocate as
> 
> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of particles
> 
> QQMLS is double precision.
> 
> I would like to sent form a CPU to another part of it, for example,
> sending QQMLS(:,50:100,:). I mean sending the QQMLS of the particles
> between 50 to 100.
> I suppose that i could use MPI_Type_vector but I am not sure. The
> particle that I want to sent could be from 25 to 50 ecc.. ecc..so
>   blocklength changes everytime.
> 
> Do I have to use MPI_type_create_struct?
> Do I have correctly understood MPI_Type_vector?
> 
> Thanks a lot
> 
> 
> Diego
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
> 
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26171.php 
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
> 
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26172.php 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26184.php

Re: [OMPI users] libevent hangs on app finalize stage

[Ralph Castain]

Fixed - sorry about that!


> On Jan 15, 2015, at 10:39 AM, Ralph Castain  wrote:
> 
> Ah, indeed - I found the problem. Fix coming momentarily
> 
>> On Jan 15, 2015, at 10:31 AM, Ralph Castain  wrote:
>> 
>> Hmmm…I’m not seeing a failure. Let me try on another system.
>> 
>> 
>> Modifying libevent is not a viable solution :-(
>> 
>> 
>>> On Jan 15, 2015, at 10:26 AM, Leonid  wrote:
>>> 
>>> Hi Ralph.
>>> 
>>> Of course that may indicate an issue with custom compiler, but given that 
>>> it fails with gcc and inserted delay I still think it is a OMPI bug, since 
>>> such a delay could be caused by operating system at that exact point.
>>> 
>>> For me simply commenting out "base->event_gotterm = base->event_break = 0;" 
>>> seems to do the trick, but I am not completely sure if that won't cause any 
>>> other troubles.
>>> 
>>> I've tried to update my master branch to the latest version (including your 
>>> fix) but now it just crashes for me on *all* benchmarks that I am trying 
>>> (both with gcc and our compiler).
>>> 
>>> On 15.01.2015 18:57, Ralph Castain wrote:
 Thought about this some more and realized that the orte progress engine 
 wasn’t using the opal_progress_thread support functions, which include a 
 “break” event to kick us out of just such problems. So I changed it on the 
 master. From your citing of libevent 2.0.22, I believe that must be where 
 you are working, yes?
 
 If so, give the changed version a try and see if your problem is resolved.
 
 
> On Jan 15, 2015, at 12:55 AM, Ralph Castain  wrote:
> 
> Given that you could only reproduce it with either your custom compiler 
> or by forcibly introducing a delay, is this indicating an issue with the 
> custom compiler? It does seem strange that we don't see this anywhere 
> else, given the number of times that code gets run.
> 
> Only alternative solution I can think of would be to push the finalize 
> request into the event loop, and thus execute the loopbreak from within 
> an event. You might try and see if that solves the problem.
> 
> 
>> On Jan 14, 2015, at 11:54 PM, Leonid  wrote:
>> 
>> Hi all.
>> 
>> I believe there is a bug in event_base_loop() function from file event.c 
>> (opal/mca/event/libevent2022/libevent/).
>> 
>> Consider the case when application is going to be finalized and both 
>> event_base_loop() and event_base_loopbreak() are called in the same time 
>> in parallel threads.
>> 
>> Then if event_base_loopbreak() happens to acquire lock first, it will 
>> set "event_base->event_break = 1", but won't send any signal to event 
>> loop, because it did not started yet.
>> 
>> After that, event_base_loop() will acquire the lock and will clear 
>> event_break flag with the following statement: "base->event_gotterm = 
>> base->event_break = 0;". Then it will go into polling with timeout = -1 
>> and thus block forever.
>> 
>> This issue was reproduced on a custom compiler (using Lulesh benchmark 
>> and x86 4-core PC), but it can be also reproduced for me with GCC 
>> compiler (on almost any benchmark and in same HW settings) by putting 
>> some delay to orte_progress_thread_engine() function:
>> 
>> static void* orte_progress_thread_engine(opal_object_t *obj)
>> {
>> while (orte_event_base_active) {
>>   usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set 
>> orte_event_base_active flag to false
>>   opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
>> }
>> return OPAL_THREAD_CANCELLED;
>> }
>> 
>> I am not completely sure what should be the best fix for described 
>> problem.
>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26181.php
 ___
 users mailing list
 us...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
 Link to this post: 
 http://www.open-mpi.org/community/lists/users/2015/01/26185.php
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2015/01/26188.php
>> 
> 

Re: [OMPI users] libevent hangs on app finalize stage

[Ralph Castain]

Ah, indeed - I found the problem. Fix coming momentarily

> On Jan 15, 2015, at 10:31 AM, Ralph Castain  wrote:
> 
> Hmmm…I’m not seeing a failure. Let me try on another system.
> 
> 
> Modifying libevent is not a viable solution :-(
> 
> 
>> On Jan 15, 2015, at 10:26 AM, Leonid  wrote:
>> 
>> Hi Ralph.
>> 
>> Of course that may indicate an issue with custom compiler, but given that it 
>> fails with gcc and inserted delay I still think it is a OMPI bug, since such 
>> a delay could be caused by operating system at that exact point.
>> 
>> For me simply commenting out "base->event_gotterm = base->event_break = 0;" 
>> seems to do the trick, but I am not completely sure if that won't cause any 
>> other troubles.
>> 
>> I've tried to update my master branch to the latest version (including your 
>> fix) but now it just crashes for me on *all* benchmarks that I am trying 
>> (both with gcc and our compiler).
>> 
>> On 15.01.2015 18:57, Ralph Castain wrote:
>>> Thought about this some more and realized that the orte progress engine 
>>> wasn’t using the opal_progress_thread support functions, which include a 
>>> “break” event to kick us out of just such problems. So I changed it on the 
>>> master. From your citing of libevent 2.0.22, I believe that must be where 
>>> you are working, yes?
>>> 
>>> If so, give the changed version a try and see if your problem is resolved.
>>> 
>>> 
 On Jan 15, 2015, at 12:55 AM, Ralph Castain  wrote:
 
 Given that you could only reproduce it with either your custom compiler or 
 by forcibly introducing a delay, is this indicating an issue with the 
 custom compiler? It does seem strange that we don't see this anywhere 
 else, given the number of times that code gets run.
 
 Only alternative solution I can think of would be to push the finalize 
 request into the event loop, and thus execute the loopbreak from within an 
 event. You might try and see if that solves the problem.
 
 
> On Jan 14, 2015, at 11:54 PM, Leonid  wrote:
> 
> Hi all.
> 
> I believe there is a bug in event_base_loop() function from file event.c 
> (opal/mca/event/libevent2022/libevent/).
> 
> Consider the case when application is going to be finalized and both 
> event_base_loop() and event_base_loopbreak() are called in the same time 
> in parallel threads.
> 
> Then if event_base_loopbreak() happens to acquire lock first, it will set 
> "event_base->event_break = 1", but won't send any signal to event loop, 
> because it did not started yet.
> 
> After that, event_base_loop() will acquire the lock and will clear 
> event_break flag with the following statement: "base->event_gotterm = 
> base->event_break = 0;". Then it will go into polling with timeout = -1 
> and thus block forever.
> 
> This issue was reproduced on a custom compiler (using Lulesh benchmark 
> and x86 4-core PC), but it can be also reproduced for me with GCC 
> compiler (on almost any benchmark and in same HW settings) by putting 
> some delay to orte_progress_thread_engine() function:
> 
> static void* orte_progress_thread_engine(opal_object_t *obj)
> {
>  while (orte_event_base_active) {
>usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set 
> orte_event_base_active flag to false
>opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
>  }
>  return OPAL_THREAD_CANCELLED;
> }
> 
> I am not completely sure what should be the best fix for described 
> problem.
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26181.php
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/users/2015/01/26185.php
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26188.php
> 

Re: [OMPI users] libevent hangs on app finalize stage

[Ralph Castain]

Hmmm…I’m not seeing a failure. Let me try on another system.


Modifying libevent is not a viable solution :-(


> On Jan 15, 2015, at 10:26 AM, Leonid  wrote:
> 
> Hi Ralph.
> 
> Of course that may indicate an issue with custom compiler, but given that it 
> fails with gcc and inserted delay I still think it is a OMPI bug, since such 
> a delay could be caused by operating system at that exact point.
> 
> For me simply commenting out "base->event_gotterm = base->event_break = 0;" 
> seems to do the trick, but I am not completely sure if that won't cause any 
> other troubles.
> 
> I've tried to update my master branch to the latest version (including your 
> fix) but now it just crashes for me on *all* benchmarks that I am trying 
> (both with gcc and our compiler).
> 
> On 15.01.2015 18:57, Ralph Castain wrote:
>> Thought about this some more and realized that the orte progress engine 
>> wasn’t using the opal_progress_thread support functions, which include a 
>> “break” event to kick us out of just such problems. So I changed it on the 
>> master. From your citing of libevent 2.0.22, I believe that must be where 
>> you are working, yes?
>> 
>> If so, give the changed version a try and see if your problem is resolved.
>> 
>> 
>>> On Jan 15, 2015, at 12:55 AM, Ralph Castain  wrote:
>>> 
>>> Given that you could only reproduce it with either your custom compiler or 
>>> by forcibly introducing a delay, is this indicating an issue with the 
>>> custom compiler? It does seem strange that we don't see this anywhere else, 
>>> given the number of times that code gets run.
>>> 
>>> Only alternative solution I can think of would be to push the finalize 
>>> request into the event loop, and thus execute the loopbreak from within an 
>>> event. You might try and see if that solves the problem.
>>> 
>>> 
 On Jan 14, 2015, at 11:54 PM, Leonid  wrote:
 
 Hi all.
 
 I believe there is a bug in event_base_loop() function from file event.c 
 (opal/mca/event/libevent2022/libevent/).
 
 Consider the case when application is going to be finalized and both 
 event_base_loop() and event_base_loopbreak() are called in the same time 
 in parallel threads.
 
 Then if event_base_loopbreak() happens to acquire lock first, it will set 
 "event_base->event_break = 1", but won't send any signal to event loop, 
 because it did not started yet.
 
 After that, event_base_loop() will acquire the lock and will clear 
 event_break flag with the following statement: "base->event_gotterm = 
 base->event_break = 0;". Then it will go into polling with timeout = -1 
 and thus block forever.
 
 This issue was reproduced on a custom compiler (using Lulesh benchmark and 
 x86 4-core PC), but it can be also reproduced for me with GCC compiler (on 
 almost any benchmark and in same HW settings) by putting some delay to 
 orte_progress_thread_engine() function:
 
 static void* orte_progress_thread_engine(opal_object_t *obj)
 {
   while (orte_event_base_active) {
 usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set 
 orte_event_base_active flag to false
 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
   }
   return OPAL_THREAD_CANCELLED;
 }
 
 I am not completely sure what should be the best fix for described problem.
 
 
 
 ___
 users mailing list
 us...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
 Link to this post: 
 http://www.open-mpi.org/community/lists/users/2015/01/26181.php
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26185.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26188.php

Re: [OMPI users] libevent hangs on app finalize stage

[Leonid]

Hi Ralph.

Of course that may indicate an issue with custom compiler, but given 
that it fails with gcc and inserted delay I still think it is a OMPI 
bug, since such a delay could be caused by operating system at that 
exact point.


For me simply commenting out "base->event_gotterm = base->event_break = 
0;" seems to do the trick, but I am not completely sure if that won't 
cause any other troubles.


I've tried to update my master branch to the latest version (including 
your fix) but now it just crashes for me on *all* benchmarks that I am 
trying (both with gcc and our compiler).


On 15.01.2015 18:57, Ralph Castain wrote:

Thought about this some more and realized that the orte progress engine wasn’t 
using the opal_progress_thread support functions, which include a “break” event 
to kick us out of just such problems. So I changed it on the master. From your 
citing of libevent 2.0.22, I believe that must be where you are working, yes?

If so, give the changed version a try and see if your problem is resolved.



On Jan 15, 2015, at 12:55 AM, Ralph Castain  wrote:

Given that you could only reproduce it with either your custom compiler or by 
forcibly introducing a delay, is this indicating an issue with the custom 
compiler? It does seem strange that we don't see this anywhere else, given the 
number of times that code gets run.

Only alternative solution I can think of would be to push the finalize request 
into the event loop, and thus execute the loopbreak from within an event. You 
might try and see if that solves the problem.



On Jan 14, 2015, at 11:54 PM, Leonid  wrote:

Hi all.

I believe there is a bug in event_base_loop() function from file event.c 
(opal/mca/event/libevent2022/libevent/).

Consider the case when application is going to be finalized and both 
event_base_loop() and event_base_loopbreak() are called in the same time in 
parallel threads.

Then if event_base_loopbreak() happens to acquire lock first, it will set 
"event_base->event_break = 1", but won't send any signal to event loop, because 
it did not started yet.

After that, event_base_loop() will acquire the lock and will clear event_break flag with the 
following statement: "base->event_gotterm = base->event_break = 0;". Then it 
will go into polling with timeout = -1 and thus block forever.

This issue was reproduced on a custom compiler (using Lulesh benchmark and x86 
4-core PC), but it can be also reproduced for me with GCC compiler (on almost 
any benchmark and in same HW settings) by putting some delay to 
orte_progress_thread_engine() function:

static void* orte_progress_thread_engine(opal_object_t *obj)
{
   while (orte_event_base_active) {
 usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set 
orte_event_base_active flag to false
 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
   }
   return OPAL_THREAD_CANCELLED;
}

I am not completely sure what should be the best fix for described problem.



___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26181.php

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26185.php

Re: [OMPI users] mpirun hangs without internet connection

[Marco Atzeri]

On 1/15/2015 5:39 PM, Klara Hornisova wrote:

I have installed OpenMPI 1.6.5 under cygwin. When trying test example

$mpirun hello


current cygwin package is 1.8.4-1, could you test it ?



or, e.g., more complex examples from scalapack, such as

$mpirun -np 4 xslu

everything works fine when there is an internet connection. However,
when the cable is disconnected, mpirun hangs without any error message.
With -d option the output stops before the line


there is a wireless on?


[my_computer...] [[3169,1],0] node[0].name my_computer... daemon0

which is included in the output when the internet is on.

The firewall is turned off.
I tried also to add options: -host localhost, -mca btl self, --mca
btl_tcp_if_include “127.0.0.1/8 ” and their
combinations, but nothing has changed.

Thank you in advance for advice.

Klara Hornisova



Regards
Marco

[OMPI users] mpirun hangs without internet connection

[Klara Hornisova]

I have installed OpenMPI 1.6.5 under cygwin. When trying test example

$mpirun hello

or, e.g., more complex examples from scalapack, such as

$mpirun -np 4 xslu

everything works fine when there is an internet connection. However, when
the cable is disconnected, mpirun hangs without any error message. With -d
option the output stops before the line

[my_computer...] [[3169,1],0] node[0].name my_computer... daemon0

which is included in the output when the internet is on.

The firewall is turned off.
I tried also to add options: -host localhost, -mca btl self, --mca
btl_tcp_if_include “127.0.0.1/8” and their combinations, but nothing has
changed.

Thank you in advance for advice.

Klara Hornisova

Re: [OMPI users] libevent hangs on app finalize stage

[Ralph Castain]

Thought about this some more and realized that the orte progress engine wasn’t 
using the opal_progress_thread support functions, which include a “break” event 
to kick us out of just such problems. So I changed it on the master. From your 
citing of libevent 2.0.22, I believe that must be where you are working, yes?

If so, give the changed version a try and see if your problem is resolved.


> On Jan 15, 2015, at 12:55 AM, Ralph Castain  wrote:
> 
> Given that you could only reproduce it with either your custom compiler or by 
> forcibly introducing a delay, is this indicating an issue with the custom 
> compiler? It does seem strange that we don't see this anywhere else, given 
> the number of times that code gets run.
> 
> Only alternative solution I can think of would be to push the finalize 
> request into the event loop, and thus execute the loopbreak from within an 
> event. You might try and see if that solves the problem.
> 
> 
>> On Jan 14, 2015, at 11:54 PM, Leonid  wrote:
>> 
>> Hi all.
>> 
>> I believe there is a bug in event_base_loop() function from file event.c 
>> (opal/mca/event/libevent2022/libevent/).
>> 
>> Consider the case when application is going to be finalized and both 
>> event_base_loop() and event_base_loopbreak() are called in the same time in 
>> parallel threads.
>> 
>> Then if event_base_loopbreak() happens to acquire lock first, it will set 
>> "event_base->event_break = 1", but won't send any signal to event loop, 
>> because it did not started yet.
>> 
>> After that, event_base_loop() will acquire the lock and will clear 
>> event_break flag with the following statement: "base->event_gotterm = 
>> base->event_break = 0;". Then it will go into polling with timeout = -1 and 
>> thus block forever.
>> 
>> This issue was reproduced on a custom compiler (using Lulesh benchmark and 
>> x86 4-core PC), but it can be also reproduced for me with GCC compiler (on 
>> almost any benchmark and in same HW settings) by putting some delay to 
>> orte_progress_thread_engine() function:
>> 
>> static void* orte_progress_thread_engine(opal_object_t *obj)
>> {
>>   while (orte_event_base_active) {
>> usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set 
>> orte_event_base_active flag to false
>> opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
>>   }
>>   return OPAL_THREAD_CANCELLED;
>> }
>> 
>> I am not completely sure what should be the best fix for described problem.
>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/01/26181.php
> 

Re: [OMPI users] MPI_type_create_struct + MPI_Type_vector + MPI_Type_contiguous

[Diego Avesani]

Dear Gus, Dear all,
Thanks a lot.
MPI_Type_Struct works well for the first part of my problem, so I am very
happy to be able to use it.

Regarding MPI_TYPE_VECTOR.

I have studied it and for simple case it is clear to me what id does (at
least I believe). Foe example if I have a matrix define as:
REAL, ALLOCATABLE (AA(:,:))
ALLOCATE AA(100,5)

I could send part of it defining

CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,*MY_NEW_TYPE*)

after that I can send part of it with

CALL MPI_SEND( AA(1:*10*,:), *10*, *MY_NEW_TYPE*, 1, 0, MPI_COMM_WORLD );

Have I understood correctly?

What I can do in case of three dimensional array? for example AA(:,:,:), I
am looking to MPI_TYPE_CREATE_SUBARRAY.
Is that the correct way?

Thanks again




Diego


On 13 January 2015 at 19:04, Gus Correa  wrote:

> Hi Diego
> I guess MPI_Type_Vector is the natural way to send and receive Fortran90
> array sections (e.g. your QQMLS(:,50:100,:)).
> I used that before and it works just fine.
> I think that is pretty standard MPI programming style.
> I guess MPI_Type_Struct tries to emulate Fortran90 and C structures
> (as you did in your previous code, with all the surprises regarding
> alignment, etc), not array sections.
> Also, MPI type vector should be more easy going (and probably more
> efficient) than MPI type struct, with less memory alignment problems.
> I hope this helps,
> Gus Correa
>
> PS - These books have a quite complete description and several examples
> of all MPI objects and functions, including MPI types (native and user
> defined):
> http://mitpress.mit.edu/books/mpi-complete-reference-0
> http://mitpress.mit.edu/books/mpi-complete-reference-1
>
> [They cover MPI 1 and 2. I guess there is a new/upcoming book
> with MPI 3, but for what you're doing 1 and 2 are more than enough.]
>
>
> On 01/13/2015 09:22 AM, Diego Avesani wrote:
>
>> Dear all,
>>
>> I had some wonderful talking about MPI_type_create_struct adn
>> isend\irecv with
>> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is more clear and my
>> program works.
>>
>> Now I have another question. In may program I have matrix:
>>
>> /QQMLS(:,:,:) /that is allocate as
>>
>> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of particles
>>
>> QQMLS is double precision.
>>
>> I would like to sent form a CPU to another part of it, for example,
>> sending QQMLS(:,50:100,:). I mean sending the QQMLS of the particles
>> between 50 to 100.
>> I suppose that i could use MPI_Type_vector but I am not sure. The
>> particle that I want to sent could be from 25 to 50 ecc.. ecc..so
>>   blocklength changes everytime.
>>
>> Do I have to use MPI_type_create_struct?
>> Do I have correctly understood MPI_Type_vector?
>>
>> Thanks a lot
>>
>>
>> Diego
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: http://www.open-mpi.org/community/lists/users/2015/01/
>> 26171.php
>>
>>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: http://www.open-mpi.org/community/lists/users/2015/01/
> 26172.php
>

Re: [OMPI users] Valgrind reports a plenty of Invalid read's in osc_rdma_data_move.c

[Victor Vysotskiy]

Hi Nathan,

surely, OpenMPI was compiled with the Valgrind support:

%/opt/mpi/openmpi-1.8.4.dbg/bin/ompi_info | grep -i memchecker
  MCA memchecker: valgrind (MCA v2.0, API v2.0, Component v1.8.4)

The following configure options were used:

--enable-mem-debug --enable-debug --enable-memchecker --with-valgrind 
--with-mpi-param-check

Best,
Victor.

Re: [OMPI users] libevent hangs on app finalize stage

[Ralph Castain]

Given that you could only reproduce it with either your custom compiler or by 
forcibly introducing a delay, is this indicating an issue with the custom 
compiler? It does seem strange that we don't see this anywhere else, given the 
number of times that code gets run.

Only alternative solution I can think of would be to push the finalize request 
into the event loop, and thus execute the loopbreak from within an event. You 
might try and see if that solves the problem.


> On Jan 14, 2015, at 11:54 PM, Leonid  wrote:
> 
> Hi all.
> 
> I believe there is a bug in event_base_loop() function from file event.c 
> (opal/mca/event/libevent2022/libevent/).
> 
> Consider the case when application is going to be finalized and both 
> event_base_loop() and event_base_loopbreak() are called in the same time in 
> parallel threads.
> 
> Then if event_base_loopbreak() happens to acquire lock first, it will set 
> "event_base->event_break = 1", but won't send any signal to event loop, 
> because it did not started yet.
> 
> After that, event_base_loop() will acquire the lock and will clear 
> event_break flag with the following statement: "base->event_gotterm = 
> base->event_break = 0;". Then it will go into polling with timeout = -1 and 
> thus block forever.
> 
> This issue was reproduced on a custom compiler (using Lulesh benchmark and 
> x86 4-core PC), but it can be also reproduced for me with GCC compiler (on 
> almost any benchmark and in same HW settings) by putting some delay to 
> orte_progress_thread_engine() function:
> 
> static void* orte_progress_thread_engine(opal_object_t *obj)
> {
>while (orte_event_base_active) {
>  usleep(1000); // add sleep to allow orte_ess_base_app_finalize() set 
> orte_event_base_active flag to false
>  opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
>}
>return OPAL_THREAD_CANCELLED;
> }
> 
> I am not completely sure what should be the best fix for described problem.
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26181.php

[OMPI users] libevent hangs on app finalize stage

[Leonid]

Hi all.

I believe there is a bug in event_base_loop() function from file event.c 
(opal/mca/event/libevent2022/libevent/).


Consider the case when application is going to be finalized and both 
event_base_loop() and event_base_loopbreak() are called in the same time 
in parallel threads.


Then if event_base_loopbreak() happens to acquire lock first, it will 
set "event_base->event_break = 1", but won't send any signal to event 
loop, because it did not started yet.


After that, event_base_loop() will acquire the lock and will clear 
event_break flag with the following statement: "base->event_gotterm = 
base->event_break = 0;". Then it will go into polling with timeout = -1 
and thus block forever.


This issue was reproduced on a custom compiler (using Lulesh benchmark 
and x86 4-core PC), but it can be also reproduced for me with GCC 
compiler (on almost any benchmark and in same HW settings) by putting 
some delay to orte_progress_thread_engine() function:


static void* orte_progress_thread_engine(opal_object_t *obj)
{
while (orte_event_base_active) {
  usleep(1000); // add sleep to allow orte_ess_base_app_finalize() 
set orte_event_base_active flag to false

  opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE);
}
return OPAL_THREAD_CANCELLED;
}

I am not completely sure what should be the best fix for described problem.