Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Wolfgang Bangerth]

On 05/23/2017 07:24 PM, Jack wrote:
Originally, I suspected that when solving two linear systems simultaneously by 
two threads would reduce time. But now it seems that this idea increases the 
complexity of communication between MPI communicators and coding, and also not 
surely to decrease computing time. I’m not going to try this idea at present. 
When I’m free, I will test its feasibility and post here the result.


I think we'd all be curious about the result. As I mentioned, it's not a 
stupid idea, but it requires significant experience in both thread and MPI 
programming to deal with the errors you're likely going to get if you get 
anything wrong.


Best
 W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
   www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Jack]

Hi Prof. Wolfgang,

 

Many thanks!

 

Originally, I suspected that when solving two linear systems simultaneously 
by two threads would reduce time. But now it seems that this idea increases 
the complexity of communication between MPI communicators and coding, and 
also not surely to decrease computing time. I’m not going to try this idea 
at present. When I’m free, I will test its feasibility and post here the 
result.

 

Thank you very much!


All my best wishes,


Jack

在 2017年5月24日星期三 UTC+8上午1:59:17，Wolfgang Bangerth写道：
>
>
> Jack, 
>
> > “The way to do this is to clone the MPI communicator you use for your 
> > overall 
> > problem once for each linear system. ” 
> > 
> > That means for my problem I have to copy the Vector and Matrix of one 
> > linear system（either the thermal diffusion or stokes flow） to another 
> > Vector and Matrix which are built with a clone of present communicator 
> > before starting a thread to solve this linear system. 
> > 
> > And then I have to copy the solution of this linear system to the 
> > targeted Vector which is built using the present MPI communicator for 
> > assembling systems. 
>
> That seems too complicated. Just set up the linear system (=matrix, 
> vectors) with a different communicator from the beginning. 
>
>
> > By the way is it possible to copy Vector and Matrix of TrilinosWrappers 
> > to another ones those are initialized with another MPI communicator? 
>
> I don't remember. It is certainly nor a usual operation. You'll have to 
> investigate whether copying also copies the communicator or not by 
> looking at the actual source. 
>
> It is not common to solve linear systems on different threads in 
> parallel when using MPI. I don't want to say that that is not a good 
> idea -- it seems like a good idea for sure because it can hide some of 
> the latency associated with MPI communication, but it is not usually 
> done. I suspect that you will not save much time compared to just 
> solving the two linear systems one after the other. 
>
> Best 
>   W. 
>
> -- 
>  
> Wolfgang Bangerth  email: bang...@colostate.edu 
>  
> www: http://www.math.colostate.edu/~bangerth/ 
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Wolfgang Bangerth]

Jack,

“The way to do this is to clone the MPI communicator you use for your 
overall

problem once for each linear system. ”

That means for my problem I have to copy the Vector and Matrix of one 
linear system（either the thermal diffusion or stokes flow） to another 
Vector and Matrix which are built with a clone of present communicator 
before starting a thread to solve this linear system.


And then I have to copy the solution of this linear system to the 
targeted Vector which is built using the present MPI communicator for 
assembling systems.


That seems too complicated. Just set up the linear system (=matrix, 
vectors) with a different communicator from the beginning.



By the way is it possible to copy Vector and Matrix of TrilinosWrappers 
to another ones those are initialized with another MPI communicator?


I don't remember. It is certainly nor a usual operation. You'll have to 
investigate whether copying also copies the communicator or not by 
looking at the actual source.


It is not common to solve linear systems on different threads in 
parallel when using MPI. I don't want to say that that is not a good 
idea -- it seems like a good idea for sure because it can hide some of 
the latency associated with MPI communication, but it is not usually 
done. I suspect that you will not save much time compared to just 
solving the two linear systems one after the other.


Best
 W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
   www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Jack]

Hi Prof. Wolfgang，


Thanks so much！


“The way to do this is to clone the MPI communicator you use for your 
overall 
problem once for each linear system. ”

That means for my problem I have to copy the Vector and Matrix of one 
linear system（either the thermal diffusion or stokes flow） to another 
Vector and Matrix which are built with a clone of present communicator 
before starting a thread to solve this linear system.

 

And then I have to copy the solution of this linear system to the targeted 
Vector which is built using the present MPI communicator for assembling 
systems.

 

By the way is it possible to copy Vector and Matrix of TrilinosWrappers to 
another ones those are initialized with another MPI communicator?

 

Thanks,

 

All best，

 

Jack

在 2017年5月22日星期一 UTC+8下午9:50:32，Wolfgang Bangerth写道：
>
>
> Jack, 
> unrelated to the question about versions of MPI libraries, when you solve 
> two 
> linear systems on separate threads, you need to pay attention that the two 
> solvers (and all associated objects such as matrices, vectors, etc) use 
> separate MPI communicators. Otherwise you will have the two solvers send 
> messages on the same communicator at the same time, and this is certain to 
> lead to trouble. 
>
> The way to do this is to clone the MPI communicator you use for your 
> overall 
> problem once for each linear system. This way, each linear system uses its 
> own 
> communication channel, and one solver can not get in the way of the other. 
>
> Best 
>   W. 
>
> -- 
>  
> Wolfgang Bangerth  email: bang...@colostate.edu 
>  
> www: http://www.math.colostate.edu/~bangerth/ 
>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [deal.II] Re: Problems in solving two PDE systems using MPI+Thread parallel computing

[Bruno Turcksin]

 Jack,

2017-05-19 22:06 GMT-04:00 Jack :
> I should update my OpenMPI to the latest version and such problem will be
> avoided and I do not need to change my code for initialization of MPI and
> TBB myself?
Unfortunately, you wil need to initialize MPI yourself or you can
change this line
https://github.com/dealii/dealii/blob/master/source/base/mpi.cc#L321
in
dealii/source/base/mpi.cc (replace MPI_THREAD_SERIALIZED by
MPI_THREAD_MULTIPLE) and recompile deal. Here is some explanation on
the different options available
https://www.open-mpi.org/doc/current/man3/MPI_Init_thread.3.php If you
still have the problem then you can try to update you version of
OpenMPI.

Best,

Bruno

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.