On Thu, Apr 9, 2015 at 2:12 PM, Steena M <[email protected]> wrote: > Thanks, Barry! I patched the master, modified > src/mat/examples/tests/ex190.c to suit my data, and the fix works. > > I need to execute MatMult and I am assigning vectors on each rank using > MatCreateVecs(A, &x, &y). However, I don't think multiplication is > happening at all. (I inserted a few printf statements inside > MatMult_MPIBAIJ and inside MatMult_SeqBAIJ1 to check). VecSum(&ysum) is > always zero. Maybe I'm assigning data incorrectly? > > Code snippet in ex190.c after partitioning the matrix unequally > > ierr = MatLoad(A,fd);CHKERRQ(ierr); > ierr = MatCreateVecs(A, &x, &y);CHKERRQ(ierr); > > ierr = VecSet(x,one);CHKERRQ(ierr); //PetscScalar one = 1.0; > ierr = VecSet(y,zero); CHKERRQ(ierr); //PetscScalar zero = 0.0; >
MatView(A, 0) gives you the matrix, so you can see what Ax should be. > ierr = MatMult(A,x,y); CHKERRQ(ierr); > VecView(y, 0) gives you the output Matt > ierr = VecSum(y,&ysum);CHKERRQ(ierr); //ysum is always zero. > > Data is a 20x20 matrix and block size is 1. > > Any thoughts? > > Thanks, > Steena > > > . > > > > > On Friday, April 3, 2015 4:17 PM, Barry Smith <[email protected]> > wrote: > > > > Steena, > > Sorry for all the delays. Our code was just totally wrong for a user > provided decomposition for two reasons 1) the logic in MatLoad_MPIXAIJ() > was too convoluted to work in all cases and 2) we didn't have a single test > case. > > I have attached a patch that fixes it for petsc-3.5.3 > . The fix is also in the branch barry/fix-matload-uneven-rows > and I'll put it into next then after testing it will got into maint and > master and the next PETSc patch release. > > Please let us know if the patch doesn't work for you > > Thanks for your patience, > > Barry > > > On Apr 1, 2015, at 3:10 PM, Steena M <[email protected]> wrote: > > > > Thanks Barry. Attached is the driver program, the binary mat file, and > the corresponding mtx file. > > > > Runtime command used: > > > > sierra324@monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows > -fin trefethen.dat -matload_block_size 1 > > > > > > > > > > On Wednesday, April 1, 2015 12:28 PM, Barry Smith <[email protected]> > wrote: > > > > > > > > Send a data file you generated and your reader program and we'll debug > it. > > > > Barry > > > > > On Apr 1, 2015, at 2:18 PM, Steena M <[email protected]> wrote: > > > > > > Thanks Barry. I removed the Preallocation calls. It is still > complaining about the malloc and incorrect data in the matrix file. I > generate binary matrix files using PETSc's pythonscript to loop through a > set of UFL sparse matrices. For this use case: > > > > > > mtx_mat = scipy.io.mmread('trefethen.mtx') > > > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), > mtx_mat) > > > > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith <[email protected]> > wrote: > > > > > > > > > > > > You should not need to call any preallocation routines when using > MatLoad() > > > > > > > > > How did you generate the file? Are you sure it has the correct > information for the matrix? > > > > > > Barry > > > > > > > > > > > > > On Mar 31, 2015, at 11:05 PM, Steena M <[email protected]> wrote: > > > > > > > > Thanks Matt. I'm still getting the malloc error > > > > > > > > [0]PETSC ERROR: Argument out of range! > > > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > > > > > and > > > > > > > > a new incorrect matrix file error: > > > > > > > > [0]PETSC ERROR: Unexpected data in file! > > > > [0]PETSC ERROR: not matrix object! > > > > > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > > > > > if (rank ==0) > > > > { > > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > > > CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > > CHKERRQ( MatLoad(A,fd)); > > > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > > } > > > > > > > > else > > > > { > > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > > > CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > > CHKERRQ(MatLoad(A,fd)); > > > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > > } > > > > > > > > Is there something I'm missing? > > > > > > > > Thanks, > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley < > [email protected]> wrote: > > > > > > > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M <[email protected]> wrote: > > > > Thanks Barry. I'm still getting the malloc error with NULL. Is there > a way to distribute the matrix without explicit preallocation? Different > matrices will be loaded during runtime and assigning preallocation > parameters would mean an additional preprocessing step. > > > > > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > > > > > > > 2) Note that this is never ever ever more efficient than making > another pass and preallocating > > > > > > > > Thanks, > > > > > > > > Matt > > > > -------------------------------------------- > > > > On Sun, 3/29/15, Barry Smith <[email protected]> wrote: > > > > > > > > Subject: Re: [petsc-users] Unequal sparse matrix row distribution > for MPI MatMult > > > > To: "Steena M" <[email protected]> > > > > Cc: "Matthew Knepley" <[email protected]>, [email protected] > > > > Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > > > > > > > On > > > > Mar 29, 2015, at 11:05 PM, Steena M <[email protected]> > > > > wrote: > > > > > > > > > > Thanks > > > > Matt. I used PETSC_DETERMINE but I'm now getting an > > > > allocation-based error: > > > > > > > > > > [0]PETSC ERROR: --------------------- > > > > Error Message ------------------------------------ > > > > > [0]PETSC ERROR: Argument out of range! > > > > > [0]PETSC ERROR: New nonzero at (2,18) > > > > caused a malloc! > > > > > [0]PETSC ERROR: > > > > > ------------------------------------------------------------------------ > > > > > > > > > > I tried > > > > preallocating on each rank for the diagonal and off diagonal > > > > section of the matrix as the next step My current > > > > approximations for preallocation > > > > > > > > > > CHKERRQ( > > > > MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > > > > > > > These > > > > arguments where you pass PETSC_DEFAULT are expecting a > > > > pointer not an integer. You can pass NULL in those > > > > locations. Though it is better to provide the correct > > > > preallocation rather than some defaults. > > > > > > > > Barry > > > > > > > > > > > > > > are throwing segmentation errors. > > > > > > > > > > [0]PETSC ERROR: > > > > Caught signal number 11 SEGV: Segmentation Violation, > > > > probably memory access out of range > > > > > > > > > > Any insights into what I'm doing > > > > wrong? > > > > > > > > > > Thanks, > > > > > Steena > > > > > > > > > > > > > > > > > > > > On Sun, 3/29/15, Matthew Knepley <[email protected]> > > > > wrote: > > > > > > > > > > Subject: > > > > Re: [petsc-users] Unequal sparse matrix row distribution for > > > > MPI MatMult > > > > > To: "Steena M" > > > > <[email protected]> > > > > > Cc: "Barry Smith" <[email protected]>, > > > > [email protected] > > > > > Date: Sunday, March 29, 2015, 10:02 PM > > > > > > > > > > On Sun, Mar 29, 2015 > > > > at > > > > > 9:56 PM, Steena M <[email protected]> > > > > > wrote: > > > > > Hi > > > > > Barry, > > > > > > > > > > > > > > > > > > > > I am trying to partition a 20 row and 20 > > > > col sparse matrix > > > > > between two procs > > > > such that proc 0 has 15 rows and 20 cols > > > > > and proc 1 has 5 rows and 20 cols. The > > > > code snippet: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > > > // > > > > > at runtime: -matload_block_size 1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if (rank ==0) > > > > > > > > > > { > > > > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 15, 20, 20, > > > > > 20) ); //rank 0 gets 75% of the rows > > > > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > > ); > > > > > > > > > > CHKERRQ( > > > > MatLoad(A,fd) ); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > else > > > > > > > > > > > > > > { > > > > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 5, > > > > 20, 20, > > > > > 20) ); //rank 1 gets 25% of the > > > > rows > > > > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > > ); > > > > > > > > > > CHKERRQ( > > > > MatLoad(A,fd) ); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > This throws the following error (probably > > > > from psplit.c): > > > > > > > > > > > > > > [1]PETSC ERROR: --------------------- Error Message > > > > > ------------------------------------ > > > > > > > > > > [1]PETSC ERROR: > > > > Nonconforming object sizes! > > > > > > > > > > [1]PETSC ERROR: Sum of local lengths 40 > > > > does not equal > > > > > global length 20, my > > > > local length 20 > > > > > > > > > > likely a call to > > > > VecSetSizes() or MatSetSizes() is > > > > > > > > > wrong. > > > > > > > > > > See > > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > > > > > > > > > > > > > > > > > > > This error printout > > > > doesn't quite make sense to me. > > > > > > > > > I'm trying to specify a total matrix size of 20x20... > > > > I > > > > > haven't yet figured out where the > > > > '40' comes > > > > > from in the error > > > > message. > > > > > > > > > > > > > > > > > > > > Any thoughts on what > > > > might be going wrong? > > > > > > > > > > Its the column specification. Just > > > > > use PETSC_DETERMINE for the local columns > > > > since all our > > > > > sparse matrixformats are > > > > row divisions > > > > > anyway. > > > > > > > > > > Thanks, > > > > > > > > > > Matt > > > > > Thanks in advance, > > > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------- > > > > > > > > > > On Sun, 3/22/15, > > > > Barry Smith <[email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Subject: Re: [petsc-users] Unequal > > > > sparse matrix row > > > > > distribution for MPI > > > > MatMult > > > > > > > > > > To: > > > > "Steena M" <[email protected]> > > > > > > > > > > Cc: [email protected] > > > > > > > > > > Date: Sunday, > > > > March 22, 2015, 3:58 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steena, > > > > > > > > > > > > > > > > > > > > I am > > > > > > > > > > > > > > a little unsure of your question. > > > > > > > > > > > > > > > > > > > > > > > > 1) > > > > you can create a MPIBAIJ > > > > > > > > > > matrix with any distribution of block > > > > rows per process > > > > > you > > > > > > > > > > want, just set the > > > > local row size for each process to > > > > > > > > > be > > > > > > > > > > what you > > > > like. Use MatCreateVecs() to get > > > > > > > > > correspondingly > > > > > > > > > > > > > > laid out vectors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > or 2) if you have a MPIBAIJ > > > > > > > > > > matrix with > > > > "equal" row layout and you want a > > > > > new > > > > > > > > > > one with uneven row layout you can > > > > simply use > > > > > > > > > > > > > > MatGetSubMatrix() to create that new matrix. > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > Unless you have > > > > another reason to have the > > > > > > > > > > matrix with an equal number row layout I > > > > would just > > > > > generate > > > > > > > > > > the matrix with > > > > the layout you want. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 22, 2015, at 5:50 PM, Steena > > > > M > > > > > > > > > > <[email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > I need to > > > > distribute > > > > > > > > > > a > > > > sparse matrix such that each proc owns an unequal > > > > > number > > > > > > > > > > of blocked rows before I proceed with > > > > MPI MatMult. My > > > > > > > > > > > > > > initial thoughts on doing this: > > > > > > > > > > > > > > > > > > > > > > 1) Use MatGetSubMatrices() on the > > > > test > > > > > > > > > > MATMPIBAIJ > > > > matrix to produce a new matrix where each > > > > > proc > > > > > > > > > > has an unequal number of rows. > > > > > > > > > > > > > > > > > > > > > > 2) Provide > > > > scatter context for vector X > > > > > > > > > > (for MatMult )using IS iscol from > > > > MatGetSubMatrices() > > > > > while > > > > > > > > > > creating the > > > > vector X. > > > > > > > > > > > > > > > > > > > > > > > > > > 3) > > > > Call MatMult() > > > > > > > > > > > > > > > > > > > > > > > > > > Will > > > > MatMult_MPIBAIJ continue to scatter > > > > > > > > > > this matrix and vector such that each > > > > proc will own an > > > > > equal > > > > > > > > > > number of matrix > > > > rows and corresponding diagonal vector > > > > > > > > > > > > > > elements? Should I write my own > > > > MPIMatMult function to > > > > > > > > > > retain my redistribution of the matrix > > > > and vector? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in > > > > > > > > > > > > > > advance, > > > > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > What most > > > > experimenters > > > > > take for granted before > > > > they begin their experiments is > > > > > > > > > infinitely more interesting than any results to which > > > > their > > > > > experiments lead. > > > > > -- Norbert > > > > > Wiener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > > > <trefethen.mtx><trefethen.dat><petsc-unequalrows-mpibaij.c> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
