Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Wolfgang Bangerth]

On 05/23/2017 07:24 PM, Jack wrote:
Originally, I suspected that when solving two linear systems simultaneously by 
two threads would reduce time. But now it seems that this idea increases the 
complexity of communication between MPI communicators and coding, and also not 
surely to decrease computing time. I’m not going to try this idea at present. 
When I’m free, I will test its feasibility and post here the result.


I think we'd all be curious about the result. As I mentioned, it's not a 
stupid idea, but it requires significant experience in both thread and MPI 
programming to deal with the errors you're likely going to get if you get 
anything wrong.


Best
 W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
   www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Jack]

Hi Prof. Wolfgang,

 

Many thanks!

 

Originally, I suspected that when solving two linear systems simultaneously 
by two threads would reduce time. But now it seems that this idea increases 
the complexity of communication between MPI communicators and coding, and 
also not surely to decrease computing time. I’m not going to try this idea 
at present. When I’m free, I will test its feasibility and post here the 
result.

 

Thank you very much!


All my best wishes,


Jack

在 2017年5月24日星期三 UTC+8上午1:59:17，Wolfgang Bangerth写道：
>
>
> Jack, 
>
> > “The way to do this is to clone the MPI communicator you use for your 
> > overall 
> > problem once for each linear system. ” 
> > 
> > That means for my problem I have to copy the Vector and Matrix of one 
> > linear system（either the thermal diffusion or stokes flow） to another 
> > Vector and Matrix which are built with a clone of present communicator 
> > before starting a thread to solve this linear system. 
> > 
> > And then I have to copy the solution of this linear system to the 
> > targeted Vector which is built using the present MPI communicator for 
> > assembling systems. 
>
> That seems too complicated. Just set up the linear system (=matrix, 
> vectors) with a different communicator from the beginning. 
>
>
> > By the way is it possible to copy Vector and Matrix of TrilinosWrappers 
> > to another ones those are initialized with another MPI communicator? 
>
> I don't remember. It is certainly nor a usual operation. You'll have to 
> investigate whether copying also copies the communicator or not by 
> looking at the actual source. 
>
> It is not common to solve linear systems on different threads in 
> parallel when using MPI. I don't want to say that that is not a good 
> idea -- it seems like a good idea for sure because it can hide some of 
> the latency associated with MPI communication, but it is not usually 
> done. I suspect that you will not save much time compared to just 
> solving the two linear systems one after the other. 
>
> Best 
>   W. 
>
> -- 
>  
> Wolfgang Bangerth  email: bang...@colostate.edu 
>  
> www: http://www.math.colostate.edu/~bangerth/ 
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Wolfgang Bangerth]

Jack,

“The way to do this is to clone the MPI communicator you use for your 
overall

problem once for each linear system. ”

That means for my problem I have to copy the Vector and Matrix of one 
linear system（either the thermal diffusion or stokes flow） to another 
Vector and Matrix which are built with a clone of present communicator 
before starting a thread to solve this linear system.


And then I have to copy the solution of this linear system to the 
targeted Vector which is built using the present MPI communicator for 
assembling systems.


That seems too complicated. Just set up the linear system (=matrix, 
vectors) with a different communicator from the beginning.



By the way is it possible to copy Vector and Matrix of TrilinosWrappers 
to another ones those are initialized with another MPI communicator?


I don't remember. It is certainly nor a usual operation. You'll have to 
investigate whether copying also copies the communicator or not by 
looking at the actual source.


It is not common to solve linear systems on different threads in 
parallel when using MPI. I don't want to say that that is not a good 
idea -- it seems like a good idea for sure because it can hide some of 
the latency associated with MPI communication, but it is not usually 
done. I suspect that you will not save much time compared to just 
solving the two linear systems one after the other.


Best
 W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
   www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Jack]

Hi Prof. Wolfgang，


Thanks so much！


“The way to do this is to clone the MPI communicator you use for your 
overall 
problem once for each linear system. ”

That means for my problem I have to copy the Vector and Matrix of one 
linear system（either the thermal diffusion or stokes flow） to another 
Vector and Matrix which are built with a clone of present communicator 
before starting a thread to solve this linear system.

 

And then I have to copy the solution of this linear system to the targeted 
Vector which is built using the present MPI communicator for assembling 
systems.

 

By the way is it possible to copy Vector and Matrix of TrilinosWrappers to 
another ones those are initialized with another MPI communicator?

 

Thanks,

 

All best，

 

Jack

在 2017年5月22日星期一 UTC+8下午9:50:32，Wolfgang Bangerth写道：
>
>
> Jack, 
> unrelated to the question about versions of MPI libraries, when you solve 
> two 
> linear systems on separate threads, you need to pay attention that the two 
> solvers (and all associated objects such as matrices, vectors, etc) use 
> separate MPI communicators. Otherwise you will have the two solvers send 
> messages on the same communicator at the same time, and this is certain to 
> lead to trouble. 
>
> The way to do this is to clone the MPI communicator you use for your 
> overall 
> problem once for each linear system. This way, each linear system uses its 
> own 
> communication channel, and one solver can not get in the way of the other. 
>
> Best 
>   W. 
>
> -- 
>  
> Wolfgang Bangerth  email: bang...@colostate.edu 
>  
> www: http://www.math.colostate.edu/~bangerth/ 
>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Bruno Turcksin]

 Jack,

2017-05-19 22:06 GMT-04:00 Jack :
> I should update my OpenMPI to the latest version and such problem will be
> avoided and I do not need to change my code for initialization of MPI and
> TBB myself?
Unfortunately, you wil need to initialize MPI yourself or you can
change this line
https://github.com/dealii/dealii/blob/master/source/base/mpi.cc#L321
in
dealii/source/base/mpi.cc (replace MPI_THREAD_SERIALIZED by
MPI_THREAD_MULTIPLE) and recompile deal. Here is some explanation on
the different options available
https://www.open-mpi.org/doc/current/man3/MPI_Init_thread.3.php If you
still have the problem then you can try to update you version of
OpenMPI.

Best,

Bruno

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Jack]

Hi Bruno,

 

I appreciated it very much for your responses.

 

The Matrices and Vectors are of TrilinosWrappers, the solvers should use 
MPI, too. I used the OpenMPI 1.8.1. The release date is Apr 22 2014, later 
than the post on the Github. I initialized the MPI as following

try

{

 using namespace dealii;

 using namespace problem;

 Utilities::MPI::MPI_InitFinalize mpi_initialization (argc, argv,

numbers::invalid_unsigned_int);

 Problem<2> app;

 app.infofile = argv[1];

 app.run ();

}

So, It seems this error is because I called two functions which use MPI 
during solving thermal diffusion and stokes flow equations.

 According to this post https://github.com/open-mpi/ompi/issues/1081

 “It turns out that if one tries to use Open MPI with MPI THREAD MULTIPLE 
(at least master) and actually has multiple threads making concurrent MPI 
calls, one gets a flood of messages of the following ilk:

opal_libevent2022_event_base_loop: reentrant invocation. Only one 
event_base_loop can run on each event_base at once.

showing up in stderr.

This needs to be fixed before release of v2.x”

I should update my OpenMPI to the latest version and such problem will be 
avoided and I do not need to change my code for initialization of MPI and 
TBB myself?

 

Thanks for your kind help!

 

Regards,


Jack

在 2017年5月19日星期五 UTC+8下午8:25:55，Bruno Turcksin写道：
>
> Jack,
>
> are your solvers using MPI?  This looks similar to this problem 
> https://github.com/open-mpi/ompi/issues/1081 Which version of MPI are you 
> using? How do you initialize MPI? MPI_InitFinalize set 
> MPI_THREAD_SERIALIZED which "tells MPI that 
> we might use several threads but never call two MPI functions at the same 
> time." This is not want you want if your solvers use MPI. Then you need to 
> initialize MPI and TBB yourself and use MPI THREAD MULTIPLE.
>
> Best,
>
> Bruno
>
> On Friday, May 19, 2017 at 3:52:39 AM UTC-4, Jack wrote:
>>
>> Dear all,
>>
>>  
>>
>> I’m trying to solve the thermal diffusion and Stokes flow problem 
>> simultaneously, similar to the step-32. I opened two threads to solve 
>> thermal diffusion and Stokes equations during solving by linear solvers 
>> (the former is solved by CG solver and the latter is solved by GMRES 
>> solver) after assembling the matrices and RHS. I encountered a problem when 
>> I using parallel computing in combination with distributed and shared 
>> memory.
>>
>>  The code is like following:
>>
>>
>> assemble_thermal_diffusion_system();
>>
>> assemble_stokes_flow_system ();
>>
>> Threads::Task task =
>>
>> Threads::new_task (::solve_thermal_diffsuion, *this);
>>
>> solve_stokes_flow ();
>>
>> task.join();
>>
>>
>> The program ran successfully with single process and showed a high 
>> performance of thread parallel computing, unfortunately, when I ran the 
>> program using several processes I got following error. I have googled this 
>> error and can not find any solution. Could someone help me?
>>
>> Thanks in advance!
>>
>>  
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only 
>> one event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
>> event_base_loop can run on each event_base at once.
>>
>>
>> Thank you very much.
>>
>>  
>>
>> Best,
>>
>>  
>>
>> Jack
>>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit 

[deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Bruno Turcksin]

Jack,

are your solvers using MPI?  This looks similar to this problem 
https://github.com/open-mpi/ompi/issues/1081 Which version of MPI are you 
using? How do you initialize MPI? MPI_InitFinalize set 
MPI_THREAD_SERIALIZED which "tells MPI that 
we might use several threads but never call two MPI functions at the same 
time." This is not want you want if your solvers use MPI. Then you need to 
initialize MPI and TBB yourself and use MPI THREAD MULTIPLE.

Best,

Bruno

On Friday, May 19, 2017 at 3:52:39 AM UTC-4, Jack wrote:
>
> Dear all,
>
>  
>
> I’m trying to solve the thermal diffusion and Stokes flow problem 
> simultaneously, similar to the step-32. I opened two threads to solve 
> thermal diffusion and Stokes equations during solving by linear solvers 
> (the former is solved by CG solver and the latter is solved by GMRES 
> solver) after assembling the matrices and RHS. I encountered a problem when 
> I using parallel computing in combination with distributed and shared 
> memory.
>
>  The code is like following:
>
>
> assemble_thermal_diffusion_system();
>
> assemble_stokes_flow_system ();
>
> Threads::Task task =
>
> Threads::new_task (::solve_thermal_diffsuion, *this);
>
> solve_stokes_flow ();
>
> task.join();
>
>
> The program ran successfully with single process and showed a high 
> performance of thread parallel computing, unfortunately, when I ran the 
> program using several processes I got following error. I have googled this 
> error and can not find any solution. Could someone help me?
>
> Thanks in advance!
>
>  
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
> [warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one 
> event_base_loop can run on each event_base at once.
>
>
> Thank you very much.
>
>  
>
> Best,
>
>  
>
> Jack
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.