date:20170728

Re: [OMPI users] Groups and Communicators

2017-07-28 Thread Diego Avesani

Dear George, Dear all,

I have just rewritten the code to make it more clear:

* INTEGER :: colorl,colorglobal*
* INTEGER :: LOCAL_COMM,MASTER_COMM*
* !*
* !---*
* ! create WORLD communicator*
* !---*
* CALL MPI_INIT(MPIworld%iErr)*
* CALL MPI_COMM_SIZE(MPI_COMM_WORLD, MPIworld%nCPU, MPIworld%iErr)  !get
the nCPU*
* CALL MPI_COMM_RANK(MPI_COMM_WORLD, MPIworld%rank, MPIworld%iErr)  !get
the rank*
* !*
* colorl = MPIworld%rank/4*
* !*
* !---*
* ! create LOCAL communicator*
* !---*
* CALL
MPI_COMM_SPLIT(MPI_COMM_WORLD,colorl,MPIworld%rank,LOCAL_COMM,MPIworld%iErr)*
* CALL MPI_COMM_RANK(LOCAL_COMM, MPIlocal%rank,MPIlocal%iErr)*
* CALL MPI_COMM_SIZE(LOCAL_COMM, MPIlocal%nCPU,MPIlocal%iErr)*
* !*
* !*
* !WRITE(*,'(A15,I3,A15,I3)') 'WORLD RANK  ',MPIworld%rank,'LOCAL RANK
 ',MPIlocal%rank*
* !*
* !---*
* ! create MASTER communicator*
* !---*
* IF(MOD(MPIworld%rank,4).EQ.0)THEN*
*colorglobal = MOD(MPIworld%rank,4)*
* ELSE*
*colorglobal = MPI_COMM_NULL*
* ENDIF*
* !*
* CALL
MPI_COMM_SPLIT(MPI_COMM_WORLD,colorglobal,MPIworld%rank,MASTER_COMM,MPIworld%iErr)*
* CALL MPI_COMM_RANK(MASTER_COMM, MPImaster%rank,MPImaster%iErr)*
* CALL MPI_COMM_SIZE(MASTER_COMM, MPImaster%nCPU,MPImaster%iErr)*
* !*
* !*
* WRITE(*,'(A15,I3,A15,I3,A15,I3)') 'WORLD RANK  ',MPIworld%rank,'LOCAL
RANK  ',MPIlocal%rank,'MASTER  ',MPImaster%nCPU *

This is the result:

WORLD RANK2   LOCAL RANK2   MASTER   12
WORLD RANK3   LOCAL RANK3   MASTER   12
WORLD RANK   10   LOCAL RANK2   MASTER   12
*WORLD RANK   12   LOCAL RANK0   MASTER4*
WORLD RANK1   LOCAL RANK1   MASTER   12
WORLD RANK5   LOCAL RANK1   MASTER   12
*WORLD RANK4   LOCAL RANK0   MASTER4*
WORLD RANK6   LOCAL RANK2   MASTER   12
WORLD RANK   13   LOCAL RANK1   MASTER   12
WORLD RANK   14   LOCAL RANK2   MASTER   12
WORLD RANK7   LOCAL RANK3   MASTER   12
WORLD RANK   11   LOCAL RANK3   MASTER   12
*WORLD RANK0   LOCAL RANK0   MASTER4*
*WORLD RANK8   LOCAL RANK0   MASTER4*
WORLD RANK9   LOCAL RANK1   MASTER   12
WORLD RANK   15   LOCAL RANK3   MASTER   12

I am expecting only a new communicator only for the master but it seems
that I get two new groups despite I set "*colorglobal = MPI_COMM_NULL*"

What do think? Is there something that I haven't understood properly?

Thanks again, I am trying to learn better MPI_Comm_create_group.

Thanks


Diego


On 28 July 2017 at 16:59, Diego Avesani  wrote:

> Dear George, Dear all,
>
> here the code:
>
> PROGRAM TEST
> USE MPI
> IMPLICIT NONE
> ! mpif90 -r8 *.f90
> !+++
> +
> INTEGER   :: rank
> INTEGER   :: subrank,leader_rank
> INTEGER   :: nCPU
> INTEGER   :: subnCPU
> INTEGER   :: ierror
> INTEGER   :: tag
> INTEGER   :: status(MPI_STATUS_SIZE)
> INTEGER   :: colorloc,colorglobal
> INTEGER   :: NEW_COMM,LEADER_COMM
> !+++
> +
>  CALL MPI_INIT(ierror)
>  CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nCPU, ierror)
>  CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
>  !
>  colorloc = rank/4
>  !
>  !
>  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorloc,rank,NEW_COMM,ierror)
>  CALL MPI_COMM_RANK(NEW_COMM, subrank,ierror);
>  CALL MPI_COMM_SIZE(NEW_COMM, subnCPU,ierror);
>  !
>  IF(MOD(rank,4).EQ.0)THEN
> colorglobal = MOD(rank,4)
>  ELSE
> colorglobal = *MPI_COMM_NULL*
>  ENDIF
>  !
>  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorglobal,rank,LEADER_COMM,ierror)
>  CALL MPI_COMM_RANK(LEADER_COMM, leader_rank,ierror);
>  !
>  CALL MPI_FINALIZE(ierror)
> ENDPROGRAM
>
> Now, it works.
>
> Could you please explain me "*MPI_Comm_create_group*". I am trying buy my
> self, but it seems quite different from MPI_SPLIT_COMM.
>
> Again, really, really thanks
>
> Diego
>
>
> On 28 July 2017 at 16:02, George Bosilca  wrote:
>
>> I guess the second comm_rank call is invalid on all non-leader processes,
>> as their LEADER_COMM communicator is MPI_COMM_NULL.
>>
>> george
>>
>> On Fri, Jul 28, 2017 at 05:06 Diego Avesani 
>> wrote:
>>
>>> Dear George, Dear all,
>>>
>>> thanks, thanks a lot. I will tell you everything.
>>> I will try also to implement your suggestion.
>>>
>>> Unfortunately,  the program that I have show to you is not working. I
>>> get the following error:
>>>
>>> [] *** An error occurred in MPI_Comm_rank
>>> [] *** reported by process [643497985,7]
>>> [] *** on communicator MPI_COMM_WORLD
>>> [] *** MPI_ERR_COMM: invalid communicator
>>> [] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will n

[OMPI users] Memory leak in Open MPI 2.1.1

2017-07-28 Thread McGrattan, Kevin B. Dr. (Fed)

I am using Open MPI 2.1.1 along with Intel Fortran 17 update 4 and I am 
experiencing what I think is a memory leak with a job that uses 184 MPI 
processes. The memory used per process appears to be increasing by about 1 to 2 
percent per hour. My code uses mostly persistent sends and receives to exchange 
packets that are on the order of 1 MB. I am using not using a special form of 
MPI_INIT_SEND, like buffered or synchronous. What can I do to identify where 
the leak is occurring? I have run this came case with Intel MPI and I did not 
see the leak.
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Groups and Communicators

2017-07-28 Thread Diego Avesani

Dear George, Dear all,

here the code:

PROGRAM TEST
USE MPI
IMPLICIT NONE
! mpif90 -r8 *.f90
!
INTEGER   :: rank
INTEGER   :: subrank,leader_rank
INTEGER   :: nCPU
INTEGER   :: subnCPU
INTEGER   :: ierror
INTEGER   :: tag
INTEGER   :: status(MPI_STATUS_SIZE)
INTEGER   :: colorloc,colorglobal
INTEGER   :: NEW_COMM,LEADER_COMM
!
 CALL MPI_INIT(ierror)
 CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nCPU, ierror)
 CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
 !
 colorloc = rank/4
 !
 !
 CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorloc,rank,NEW_COMM,ierror)
 CALL MPI_COMM_RANK(NEW_COMM, subrank,ierror);
 CALL MPI_COMM_SIZE(NEW_COMM, subnCPU,ierror);
 !
 IF(MOD(rank,4).EQ.0)THEN
colorglobal = MOD(rank,4)
 ELSE
colorglobal = *MPI_COMM_NULL*
 ENDIF
 !
 CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorglobal,rank,LEADER_COMM,ierror)
 CALL MPI_COMM_RANK(LEADER_COMM, leader_rank,ierror);
 !
 CALL MPI_FINALIZE(ierror)
ENDPROGRAM

Now, it works.

Could you please explain me "*MPI_Comm_create_group*". I am trying buy my
self, but it seems quite different from MPI_SPLIT_COMM.

Again, really, really thanks

Diego

On 28 July 2017 at 16:02, George Bosilca  wrote:

> I guess the second comm_rank call is invalid on all non-leader processes,
> as their LEADER_COMM communicator is MPI_COMM_NULL.
>
> george
>
> On Fri, Jul 28, 2017 at 05:06 Diego Avesani 
> wrote:
>
>> Dear George, Dear all,
>>
>> thanks, thanks a lot. I will tell you everything.
>> I will try also to implement your suggestion.
>>
>> Unfortunately,  the program that I have show to you is not working. I get
>> the following error:
>>
>> [] *** An error occurred in MPI_Comm_rank
>> [] *** reported by process [643497985,7]
>> [] *** on communicator MPI_COMM_WORLD
>> [] *** MPI_ERR_COMM: invalid communicator
>> [] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now
>> abort,
>> [] ***and potentially your MPI job)
>> [warn] Epoll ADD(4) on fd 47 failed.  Old events were 0; read change was
>> 0 (none); write change was 1 (add): Bad file descriptor
>> [warn] Epoll ADD(4) on fd 65 failed.  Old events were 0; read change was
>> 0 (none); write change was 1 (add): Bad file descriptor
>> [] 8 more processes have sent help message help-mpi-errors.txt /
>> mpi_errors_are_fatal
>> [] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help /
>> error messages
>>
>> What do you think could be the error?
>>
>> Really, Really thanks again
>>
>>
>>
>>
>>
>> Diego
>>
>>
>> On 27 July 2017 at 15:57, George Bosilca  wrote:
>>
>>> This looks good. If performance is critical you can speed up the entire
>>> process by using MPI_Comm_create_group instead of the second
>>> MPI_COMM_SPLIT. The MPI_Comm_create_group is collective only over the
>>> resulting communicator and not over the source communicator, so its cost is
>>> only dependent of the number of groups and not on the total number of
>>> processes.
>>>
>>> You can also try to replace the first MPI_COMM_SPLIT by the same
>>> approach. I would be curious to see the outcome.
>>>
>>>   George.
>>>
>>>
>>> On Thu, Jul 27, 2017 at 9:44 AM, Diego Avesani 
>>> wrote:
>>>
 Dear George, Dear all,

 I have tried to create a simple example. In particular, I would like to
 use 16 CPUs and to create four groups according to rank is and then a
 communicator between masters of each group.I have tried to follow the first
 part of this example
 .
 In the last part I have tried to create a communicator be masters as
 suggested by George.

 Here my example:

  CALL MPI_INIT(ierror)
  CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nCPU, ierror)
  CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
  !
  colorloc = rank/4
  !
  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorloc,rank,*NEW_COMM*,ierror)
  CALL MPI_COMM_RANK(*NEW_COMM*, subrank,ierror);
  CALL MPI_COMM_SIZE(*NEW_COMM*, subnCPU,ierror);
  !
  IF(MOD(rank,4).EQ.0)THEN
 *! where I set color for the masters*
 colorglobal = MOD(rank,4)
  ELSE
 colorglobal = MPI_UNDEFINED
  ENDIF
  !
  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorglobal,rank,LEADER_COMM,
 ierror)
  CALL MPI_COMM_RANK(*LEADER_COMM*, leader_rank,ierror);
  CALL MPI_FINALIZE(ierror)

 I would like to know if this could be correct. I mean if I have
 understood correctly what George told me about the code design. Now, this
 example does not work, but probably there is some coding error.

 Really, Really thanks
 Diego

 Diego

 On 27 July 2017 at 10:42, Diego Avesani 
 wrote:

> Dear George, Dear all,
>
> A question regarding program design:
> The draft that I have sent to yo

Re: [OMPI users] Groups and Communicators

2017-07-28 Thread George Bosilca

I guess the second comm_rank call is invalid on all non-leader processes,
as their LEADER_COMM communicator is MPI_COMM_NULL.

george

On Fri, Jul 28, 2017 at 05:06 Diego Avesani  wrote:

> Dear George, Dear all,
>
> thanks, thanks a lot. I will tell you everything.
> I will try also to implement your suggestion.
>
> Unfortunately,  the program that I have show to you is not working. I get
> the following error:
>
> [] *** An error occurred in MPI_Comm_rank
> [] *** reported by process [643497985,7]
> [] *** on communicator MPI_COMM_WORLD
> [] *** MPI_ERR_COMM: invalid communicator
> [] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> [] ***and potentially your MPI job)
> [warn] Epoll ADD(4) on fd 47 failed.  Old events were 0; read change was 0
> (none); write change was 1 (add): Bad file descriptor
> [warn] Epoll ADD(4) on fd 65 failed.  Old events were 0; read change was 0
> (none); write change was 1 (add): Bad file descriptor
> [] 8 more processes have sent help message help-mpi-errors.txt /
> mpi_errors_are_fatal
> [] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help /
> error messages
>
> What do you think could be the error?
>
> Really, Really thanks again
>
>
>
>
>
> Diego
>
>
> On 27 July 2017 at 15:57, George Bosilca  wrote:
>
>> This looks good. If performance is critical you can speed up the entire
>> process by using MPI_Comm_create_group instead of the second
>> MPI_COMM_SPLIT. The MPI_Comm_create_group is collective only over the
>> resulting communicator and not over the source communicator, so its cost is
>> only dependent of the number of groups and not on the total number of
>> processes.
>>
>> You can also try to replace the first MPI_COMM_SPLIT by the same
>> approach. I would be curious to see the outcome.
>>
>>   George.
>>
>>
>> On Thu, Jul 27, 2017 at 9:44 AM, Diego Avesani 
>> wrote:
>>
>>> Dear George, Dear all,
>>>
>>> I have tried to create a simple example. In particular, I would like to
>>> use 16 CPUs and to create four groups according to rank is and then a
>>> communicator between masters of each group.I have tried to follow the first
>>> part of this example
>>> .
>>> In the last part I have tried to create a communicator be masters as
>>> suggested by George.
>>>
>>> Here my example:
>>>
>>>  CALL MPI_INIT(ierror)
>>>  CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nCPU, ierror)
>>>  CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
>>>  !
>>>  colorloc = rank/4
>>>  !
>>>  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorloc,rank,*NEW_COMM*,ierror)
>>>  CALL MPI_COMM_RANK(*NEW_COMM*, subrank,ierror);
>>>  CALL MPI_COMM_SIZE(*NEW_COMM*, subnCPU,ierror);
>>>  !
>>>  IF(MOD(rank,4).EQ.0)THEN
>>> *! where I set color for the masters*
>>> colorglobal = MOD(rank,4)
>>>  ELSE
>>> colorglobal = MPI_UNDEFINED
>>>  ENDIF
>>>  !
>>>  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorglobal,rank,LEADER_COMM,ierror)
>>>  CALL MPI_COMM_RANK(*LEADER_COMM*, leader_rank,ierror);
>>>  CALL MPI_FINALIZE(ierror)
>>>
>>> I would like to know if this could be correct. I mean if I have
>>> understood correctly what George told me about the code design. Now, this
>>> example does not work, but probably there is some coding error.
>>>
>>> Really, Really thanks
>>> Diego
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Diego
>>>
>>>
>>> On 27 July 2017 at 10:42, Diego Avesani  wrote:
>>>
 Dear George, Dear all,

 A question regarding program design:
 The draft that I have sent to you has to be done many and many times.
 Does the splitting procedure ensure efficiency?

 I will try, at a lest to create groups and split them. I am a beginner
 in the MPI groups environment.
 really, really thanks.

 You are my lifesaver.



 Diego


 On 26 July 2017 at 15:09, George Bosilca  wrote:

> Diego,
>
> As all your processes are started under the umbrella of a single
> mpirun, they have a communicator in common, the MPI_COMM_WORLD.
>
> One possible implementation, using MPI_Comm_split, will be the
> following:
>
> MPI_Comm small_comm, leader_comm;
>
> /* Create small_comm on all processes */
>
> /* Now use MPI_Comm_split on MPI_COMM_WORLD to select the leaders */
> MPI_Comm_split( MPI_COMM_WORLD,
>  i_am_leader(small_comm) ? 1 :
> MPI_UNDEFINED,
>  rank_in_comm_world,
>  &leader_Comm);
>
> The leader_comm will be a valid communicator on all leaders processes,
> and MPI_COMM_NULL on all others.
>
>   George.
>
>
>
> On Wed, Jul 26, 2017 at 4:29 AM, Diego Avesani <
> diego.aves...@gmail.com> wrote:
>
>> Dear George, Dear all,
>>
>> I use "mpirun -np xx ./a.out"
>>
>> I do not know if I have some common  grounds. I mean, I have to
>> desig

Re: [OMPI users] Groups and Communicators

2017-07-28 Thread Diego Avesani

Dear George, Dear all,

thanks, thanks a lot. I will tell you everything.
I will try also to implement your suggestion.

Unfortunately,  the program that I have show to you is not working. I get
the following error:

[] *** An error occurred in MPI_Comm_rank
[] *** reported by process [643497985,7]
[] *** on communicator MPI_COMM_WORLD
[] *** MPI_ERR_COMM: invalid communicator
[] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[] ***and potentially your MPI job)
[warn] Epoll ADD(4) on fd 47 failed.  Old events were 0; read change was 0
(none); write change was 1 (add): Bad file descriptor
[warn] Epoll ADD(4) on fd 65 failed.  Old events were 0; read change was 0
(none); write change was 1 (add): Bad file descriptor
[] 8 more processes have sent help message help-mpi-errors.txt /
mpi_errors_are_fatal
[] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help /
error messages

What do you think could be the error?

Really, Really thanks again





Diego


On 27 July 2017 at 15:57, George Bosilca  wrote:

> This looks good. If performance is critical you can speed up the entire
> process by using MPI_Comm_create_group instead of the second
> MPI_COMM_SPLIT. The MPI_Comm_create_group is collective only over the
> resulting communicator and not over the source communicator, so its cost is
> only dependent of the number of groups and not on the total number of
> processes.
>
> You can also try to replace the first MPI_COMM_SPLIT by the same approach.
> I would be curious to see the outcome.
>
>   George.
>
>
> On Thu, Jul 27, 2017 at 9:44 AM, Diego Avesani 
> wrote:
>
>> Dear George, Dear all,
>>
>> I have tried to create a simple example. In particular, I would like to
>> use 16 CPUs and to create four groups according to rank is and then a
>> communicator between masters of each group.I have tried to follow the first
>> part of this example
>> .
>> In the last part I have tried to create a communicator be masters as
>> suggested by George.
>>
>> Here my example:
>>
>>  CALL MPI_INIT(ierror)
>>  CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nCPU, ierror)
>>  CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
>>  !
>>  colorloc = rank/4
>>  !
>>  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorloc,rank,*NEW_COMM*,ierror)
>>  CALL MPI_COMM_RANK(*NEW_COMM*, subrank,ierror);
>>  CALL MPI_COMM_SIZE(*NEW_COMM*, subnCPU,ierror);
>>  !
>>  IF(MOD(rank,4).EQ.0)THEN
>> *! where I set color for the masters*
>> colorglobal = MOD(rank,4)
>>  ELSE
>> colorglobal = MPI_UNDEFINED
>>  ENDIF
>>  !
>>  CALL MPI_COMM_SPLIT(MPI_COMM_WORLD,colorglobal,rank,LEADER_COMM,ierror)
>>  CALL MPI_COMM_RANK(*LEADER_COMM*, leader_rank,ierror);
>>  CALL MPI_FINALIZE(ierror)
>>
>> I would like to know if this could be correct. I mean if I have
>> understood correctly what George told me about the code design. Now, this
>> example does not work, but probably there is some coding error.
>>
>> Really, Really thanks
>> Diego
>>
>>
>>
>>
>>
>>
>>
>> Diego
>>
>>
>> On 27 July 2017 at 10:42, Diego Avesani  wrote:
>>
>>> Dear George, Dear all,
>>>
>>> A question regarding program design:
>>> The draft that I have sent to you has to be done many and many times.
>>> Does the splitting procedure ensure efficiency?
>>>
>>> I will try, at a lest to create groups and split them. I am a beginner
>>> in the MPI groups environment.
>>> really, really thanks.
>>>
>>> You are my lifesaver.
>>>
>>>
>>>
>>> Diego
>>>
>>>
>>> On 26 July 2017 at 15:09, George Bosilca  wrote:
>>>
 Diego,

 As all your processes are started under the umbrella of a single
 mpirun, they have a communicator in common, the MPI_COMM_WORLD.

 One possible implementation, using MPI_Comm_split, will be the
 following:

 MPI_Comm small_comm, leader_comm;

 /* Create small_comm on all processes */

 /* Now use MPI_Comm_split on MPI_COMM_WORLD to select the leaders */
 MPI_Comm_split( MPI_COMM_WORLD,
  i_am_leader(small_comm) ? 1 :
 MPI_UNDEFINED,
  rank_in_comm_world,
  &leader_Comm);

 The leader_comm will be a valid communicator on all leaders processes,
 and MPI_COMM_NULL on all others.

   George.



 On Wed, Jul 26, 2017 at 4:29 AM, Diego Avesani >>> > wrote:

> Dear George, Dear all,
>
> I use "mpirun -np xx ./a.out"
>
> I do not know if I have some common  grounds. I mean, I have to
> design everything from the begging. You can find what I would like to do 
> in
> the attachment. Basically, an MPI cast in another MPI. Consequently, I am
> thinking to MPI groups or MPI virtual topology with a 2D cart, using the
> columns as "groups" and the first rows as the external groups to handle 
> the
> columns.
>
> What do think? What do you suggest?
>

Re: [OMPI users] Groups and Communicators

[OMPI users] Memory leak in Open MPI 2.1.1

Re: [OMPI users] Groups and Communicators

Re: [OMPI users] Groups and Communicators

Re: [OMPI users] Groups and Communicators

5 matches

Site Navigation

Mail list logo

Footer information