Progress Report: So I modified my code to remove all the wrong usages of MatCopy where I was trying to AllScatter/AllGather a parallel matrix, and replaced with usage of MatGetRedundantMatrix(). The highlighted diff is here<http://goo.gl/yyJVcV> .
Now, running it with 1 proc crashes, I guess because the documentation says<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRedundantMatrix.html>that only parallel matrices are supported by MatGetRedundantMatrix(), and running with 1 proc means the parallel matrices are sequantial (from what I read in the PetSc users manual). However, running it with 2,3,4,5+ procs now yields identical and consistent results (unlike before), so the problem appears to have been solved. I would be very grateful if an extra pair of eyes could take a quick gander at the diff and confirm if my changes are correct. Thanks a bunch and Regards, On Wed, Sep 25, 2013 at 12:55 PM, Analabha Roy <[email protected]>wrote: > > > > On Wed, Sep 25, 2013 at 12:12 PM, Jed Brown <[email protected]> wrote: > >> Analabha Roy <[email protected]> writes: >> >> > There is one thing >> > >> > In the code, the evaluation of each element of AVG_BIBJ requires a >> > read-only matrix U_parallel that I input from another program, and a >> > writeable sequential vector BDB_AA that is different for each element. >> > >> > I sequentiate U_parallel to U_seq by using MatCopy here in lines >> > 242+< >> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242 >> > >> >> /* >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - - - - - - - - - - - - - >> Create a sequential matrix and copy U_parallel to it >> so that each processor has complete local copy of U >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - - - - - - - - - - - - - >> */ >> ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL, >> &U_seq); >> CHKERRQ (ierr); >> ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR, >> PETSC_FALSE); >> CHKERRQ (ierr); >> ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN); >> >> This is wrong and should raise an error (definitely in a debugging >> build). Are you intending to create a redundant matrix containing all >> of U_parallel replicated onto each process? If so, you may as well >> assemble the whole thing redundantly, but if you'd rather distribute the >> computation of entries, you can use MatGetRedundantMatrix(). If you >> want something else, what is it? >> >> > > Thanks for the informative post. You're right in that I need to distribute > the computation of entries, but need the data scattered/gathered to all > procs afterwards. I thought MatCopy would do this. I'm not surprised that > this is wrong. This part is probably what's wrong with the code. > > I'm looking at MatGetRedundantMatrix() docs and using it to fix the > problem. Will post results as soon as I'm done... > > > >> CHKERRQ (ierr); >> ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY); >> CHKERRQ (ierr); >> ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY); >> CHKERRQ (ierr); >> >> >> > and each process is supposed to update its copy of BDB_AA at every loop >> > iteration here in line >> > 347+< >> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347 >> > >> > >> > >> > Is this right? Or are sequential vectors/matrices handled by the root >> > process only? >> >> PETSC_COMM_SELF means the current process, so every process does that >> stuff. >> >> > I know how to scatter a parallel vector to all processes using PetSc >> > scatter contexts but don't see any way to do so to a matrix other than >> > MatCopy. How do I ensure that each process has its own private >> > writeable copy of a sequential vector? >> >> Is the last sentence supposed to say "vector"? If so, see >> VecScatterCreateToAll. >> > > > > -- > --- > *Analabha Roy* > C.S.I.R <http://www.csir.res.in> Senior Research > Associate<http://csirhrdg.res.in/poolsra.htm> > Saha Institute of Nuclear Physics <http://www.saha.ac.in> > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: [email protected], [email protected] > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- --- *Analabha Roy* C.S.I.R <http://www.csir.res.in> Senior Research Associate<http://csirhrdg.res.in/poolsra.htm> Saha Institute of Nuclear Physics <http://www.saha.ac.in> Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: [email protected], [email protected] *Webpage*: http://www.ph.utexas.edu/~daneel/
