Thanks a lot to both of you! First, I modified my code in a way that the "colored iterators" (still one color) only contains locally owned cells. Unfortunately, this didn't help to avoid the crash and I still got the same random error as before.
Next, I tested Martin's suggestion: > The solution to this problem is to either initialize the vector with the > appropriate ghost entries (locally relevant dofs), i.e., the reinit > method/constructor that takes two index sets, an MPI_Comm and a > vector_writable flag. Alternatively, you can use > parallel::distributed::Vector that you initialize with the same two index > sets, locally owned dofs as first argument and locally relevant dofs as the > second element (i.e., ghosts are the locally relevant minus the locally > owned dofs). Trilinos matrices and preconditioners should know how to > collaborate with these vectors. And it works fine. To be clear, instead of using dealii::TrilinosWrappers::MPI::Vector dst(locally_owned_dofs,MPI_COMM_WORLD ); the correct way would be dealii::TrilinosWrappers::MPI::Vector dst; dst.reinit(locally_owned_dofs,locally_relevant_dofs,MPI_COMM_WORLD,true); This attempt also works fine in a much more sophisticated code of mine. And maybe just to note it, it runs for both cases, using the whole range of iterators as well as having a range consisting only of locally owned iterators. In my eyes this is fulfilled by the fact that cell_action() checks, if the given cell is locally owned or not. Again, thanks a lot for all your kind responses and your offered help. Best, Julius P.S. I will upload the corrected code after the holidays as it is on my local machine in the university, unfortunately. Am Mittwoch, 25. Mai 2016 15:30:04 UTC+2 schrieb Martin Kronbichler: > > Hi Julius, > > Is the implementation of the (colored) WorkStream::run() compatible with a > hybrid parallel code-architecture? > > > Besides checking for the validity of the ranges (as Bruno points out), > there is at least one potential problem as seen from the stack trace: You > cannot write into TrilinosWrappers::MPI::Vector in a multithreaded way > unless you take special care at construction. The way the vector works is > that it accumulates non-local entries in a data structure a bit similar to > an std::map. Of course, that is not thread-safe. This also explains why the > problem only appears sporadically: It happens when one thread changes the > shared data structure while the other is looking at it (or trying to modify > the same data structure at the same time). > > The solution to this problem is to either initialize the vector with the > appropriate ghost entries (locally relevant dofs), i.e., the reinit > method/constructor that takes two index sets, an MPI_Comm and a > vector_writable flag. Alternatively, you can use > parallel::distributed::Vector that you initialize with the same two index > sets, locally owned dofs as first argument and locally relevant dofs as the > second element (i.e., ghosts are the locally relevant minus the locally > owned dofs). Trilinos matrices and preconditioners should know how to > collaborate with these vectors. > > There is potentially the same problem for Trilinos sparse matrices in case > you start using them, but there it is simpler to arrive to the thread-safe > path (if you initialize them by a DynamicSparsityPattern and set > 'exchange_data' to true). > > #10 0x00007ffff0735a05 in std::vector<int, std::allocator<int> >::insert ( > >> this=this@entry=0x8ecb40, __position=..., __x=@0x7fffda9c85f8: 1) >> at /usr/include/c++/4.8/bits/vector.tcc:127 >> #11 0x00007ffff077ff62 in Epetra_FEVector::inputNonlocalValues<int> ( >> this=this@entry=0x8eca40, GID=771, numValues=numValues@entry=1, >> values=values@entry=0x7fffda9c8680, suminto=suminto@entry=true, >> vectorIndex=vectorIndex@entry=0) >> at >> /export/home/jwitte/trilinos-12.6.1-Source/packages/epetra/src/Epetra_FEVector.cpp:427 >> > > It is this frame #11 that tells me that you use > Epetra_FEVector::inputNonlocalValues that is a non-thread-safe method. > > Best, > Martin > -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
