Kyle:
I have a question to which I think the answer may be "no" but I thought I
would ask. I'll just ask and then explain the "why?" at the end in case there
is a better work around from the outset.
I am initializing MPI myself with MPI_THREAD_MULTIPLE so that threads can each
call MPI functions without interfering. To the extent possible each thread has
its own copy of MPI_COMM_WORLD so that simultaneous calls do not get
convoluted. However, I have a matrix of type TrilinosWrappers::SparseMatrix
which BOTH threads need simultaneous access to. Since you must give one and
only one MPI_Comm object in the constructor, these sorts of conflicts are
inevitable.
For obvious reasons I would not like to require a copy of this matrix for each
thread. The other obvious solution is a mutex on the matrix, but this could
easily get costly as both threads are calling matrix.vmult(...) in an
iterative solver. I thus have two questions:
1) Is initializing MPI with MPI_THREAD_MULTIPLE going to break the deal.ii
internals for some reason and I should just not investigate this further?
This should work. deal.II uses MPI_THREAD_SERIALIZED internally.
2) I think the best solution, if possible, would be to get pointers to the
internal data of my matrix which I can then associate with different MPI_Comm
objects. Is this possible?
No. You should never try to use anything but the public interfaces of classes.
Everything is bound to break things in unpredictable ways sooner or later.
Probably sooner.
Why am I doing this?
This is a bit of a simplification, but imagine that I am solving a linear
deferred correction problem. This means at each time step I solve A . x_1 =
b_1 and A . x_2 = b_2. Let us assume that the matrix A does not have a
well-known preconditioner which scales nicely with the number of processors.
Then instead of using 2n processors on each linear system in series, we could
instead use n processors on each linear system simultaneously and expect this
to be faster. I hope this makes sense.
Yes, this makes sense. But you should not expect to be able to solve multiple
linear systems at the same time over the same communicator. Each step in a
linear solver (vector dot products, matrix-vector products, etc.) consists of
multiple MPI messages where process wait for data to be sent from other
processes. If you have multiple solves running on the same process, you will
receive messages in unpredictable orders that may or may not belong to the
current solve. Nothing good can come out of this.
But if the linear solve is the bottleneck, you can always build the matrix
multiple times (or create copies), with different (sub-)communicators and run
one solve on each communicator.
Best
W.
--
------------------------------------------------------------------------
Wolfgang Bangerth email: [email protected]
www: http://www.math.colostate.edu/~bangerth/
--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see
https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/dealii/700631c0-e021-425f-9e4e-605583606cb3%40colostate.edu.
