Hi guys, I have two ghosted vectors fxx, fyy which should store derivatives of a quantity f. They are both obtained via calling VecDuplicate on f. To minimize communications when computing both fxx and fyy I do the following:
// loop over boundary points and compute fxx and fyy VecGhostUpdateBegin(fxx, INSERT_VALUES, SCATTER_FORWARD); VecGhostUpdateBegin(fyy, INSERT_VALUES, SCATTER_FORWARD); // compute fxx and fyy for internal points VecGhostUpdateEnd(fxx, INSERT_VALUES, SCATTER_FORWARD); VecGhostUpdateEnd(fyy, INSERT_VALUES, SCATTER_FORWARD); However this does not work and I get the following error: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Object is in wrong state! [1]PETSC ERROR: Scatter ctx already in use! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Development HG revision: d0a314355399bb0c38e384fde4b1feb01b721896 HG Date: Sat Nov 03 17:18:52 2012 -0500 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./derivatives on a arch-linu named mohammad-Desktop by mohammad Tue Oct 1 15:47:38 2013 [1]PETSC ERROR: Libraries linked from /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib [1]PETSC ERROR: Configure run at Sat Nov 3 15:26:57 2012 [1]PETSC ERROR: Configure options --download-blacs=1 --download-f-blas-lapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 --with-clanguage=cxx --with-mpi-dir=/usr/bin --with-pthreadclasses=1 PETSC_ARCH=arch-linux2-cxx-debug [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: VecScatterBegin() line 1546 in /home/mohammad/soft/petsc-dev/src/vec/vec/utils/vscat.c [1]PETSC ERROR: VecGhostUpdateBegin() line 235 in /home/mohammad/soft/petsc-dev/src/vec/vec/impls/mpi/commonmpvec.c [1]PETSC ERROR: constructing node neighboring information ... done in 0.0314 secs. on process 0 [Note: only showing root's timings] Computing derivatives the new way ... dxx_and_dyy_central() line 659 in "unknowndirectory/"../../src/my_p4est_node_neighbors.cpp terminate called after throwing an instance of 'PETSc::Exception' what(): std::exception [mohammad-Desktop:19589] *** Process received signal *** [mohammad-Desktop:19589] Signal: Aborted (6) [mohammad-Desktop:19589] Signal code: (-6) [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Scatter ctx already in use! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d0a314355399bb0c38e384fde4b1feb01b721896 HG Date: Sat Nov 03 17:18:52 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./derivatives on a arch-linu named mohammad-Desktop by mohammad Tue Oct 1 15:47:38 2013 [0]PETSC ERROR: Libraries linked from /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib [0]PETSC ERROR: Configure run at Sat Nov 3 15:26:57 2012 [0]PETSC ERROR: Configure options --download-blacs=1 --download-f-blas-lapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 --with-clanguage=cxx --with-mpi-dir=/usr/bin --with-pthreadclasses=1 PETSC_ARCH=arch-linux2-cxx-debug [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecScatterBegin() line 1546 in /home/mohammad/soft/petsc-dev/src/vec/vec/utils/vscat.c [0]PETSC ERROR: VecGhostUpdateBegin() line 235 in /home/mohammad/soft/petsc-dev/src/vec/vec/impls/mpi/commonmpvec.c [0]PETSC ERROR: dxx_and_dyy_central() line 659 in "unknowndirectory/"../../src/my_p4est_node_neighbors.cpp terminate called after throwing an instance of 'PETSc::Exception' what(): std::exception [mohammad-Desktop:19588] *** Process received signal *** [mohammad-Desktop:19588] Signal: Aborted (6) [mohammad-Desktop:19588] Signal code: (-6) [mohammad-Desktop:19589] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f12290f4cb0] [mohammad-Desktop:19589] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f1228a60425] [mohammad-Desktop:19589] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f1228a63b8b] [mohammad-Desktop:19589] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d) [0x7f122988069d] [mohammad-Desktop:19589] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f122987e846] [mohammad-Desktop:19589] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f122987e873] [mohammad-Desktop:19589] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f122987e96e] [mohammad-Desktop:19589] [ 7] ./derivatives(_Z10PetscErrorP19ompi_communicator_tiPKcS2_S2_i14PetscErrorTypeS2_z+0x53c) [0x5bf83c] [mohammad-Desktop:19589] [ 8] ./derivatives(_ZNK25my_p4est_node_neighbors_t19dxx_and_dyy_centralEP6_p_VecS1_S1_+0x950) [0x505c30] [mohammad-Desktop:19589] [ 9] ./derivatives(main+0xe89) [0x4e4678] [mohammad-Desktop:19589] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f1228a4b76d] [mohammad-Desktop:19589] [11] ./derivatives() [0x4e2ab9] [mohammad-Desktop:19589] *** End of error message *** [mohammad-Desktop:19588] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f48f260bcb0] [mohammad-Desktop:19588] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f48f1f77425] [mohammad-Desktop:19588] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f48f1f7ab8b] [mohammad-Desktop:19588] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d) [0x7f48f2d9769d] [mohammad-Desktop:19588] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f48f2d95846] [mohammad-Desktop:19588] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f48f2d95873] [mohammad-Desktop:19588] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f48f2d9596e] [mohammad-Desktop:19588] [ 7] ./derivatives(_Z10PetscErrorP19ompi_communicator_tiPKcS2_S2_i14PetscErrorTypeS2_z+0x53c) [0x5bf83c] [mohammad-Desktop:19588] [ 8] ./derivatives(_ZNK25my_p4est_node_neighbors_t19dxx_and_dyy_centralEP6_p_VecS1_S1_+0x950) [0x505c30] [mohammad-Desktop:19588] [ 9] ./derivatives(main+0xe89) [0x4e4678] [mohammad-Desktop:19588] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f48f1f6276d] [mohammad-Desktop:19588] [11] ./derivatives() [0x4e2ab9] [mohammad-Desktop:19588] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 1 with PID 19589 on node mohammad-Desktop exited on signal 6 (Aborted). -------------------------------------------------------------------------- I assume I'm getting this since they somehow share a VecScatter internally because of VecDuplicate? If so how can I fix this? Also as a side question, if I stored fxx and fyy inside a block vector, would I gain any performance in terms of MPI traffic? What about in 3D? Thanks
