Marius: Current PETSc interface supports sequential sparse multiple right-hand side, but not distributed. It turns out that mumps does not support distributed sparse multiple right-hand sides at the moment (see attached email).
Jean-Yves invites you to communicate with him directly. Let me know what we can help on this matter, e.g., add support for parallel implementation of sparse multiple right-hand side with centralized rhs input? Hong ---------------------- Jean-Yves LExcellent <[email protected]> 5:14 AM (3 hours ago) to Hong, mumps-dev Hello, We do not support distributed sparse multiple right-hand sides at the moment. From the feedback we have from applications, the right-hand sides are often very sparse, and having them distributed did not seem critical. Since we are specifying a distributed right-hand sides feature at the moment, could you let us know in more detail the need regarding distributed sparse right-hand side (e.g., do all columns have the same nonzero structure in that case) or put us in contact with the user who needs this? Thanks, Jean-Yves and Patrick Thanks a lot guys, very helpful. > > > > > I see MUMPS http://mumps.enseeiht.fr/ > > Sparse multiple right-hand side, distributed solution; Exploitation of > sparsity in the right-hand sidesPETSc interface computes mumps distributed > solution as default (this is not new) (ICNTL(21) = 1) > > I will add support for Sparse multiple right-hand side. > > Hong > > On Thu, May 31, 2018 at 11:25 AM, Smith, Barry F. <[email protected] > [mailto:[email protected]]> wrote: > Hong, > > Can you see about adding support for distributed right hand side? > > Thanks > > Barry > > > On May 31, 2018, at 2:37 AM, Marius Buerkle <[email protected][mailto: > [email protected]]> wrote: > > > > The fix for MAT_NEW_NONZERO_LOCATIONS, thanks again. > > > > I have yet another question, sorry. The recent version of MUMPS supports > distributed and sparse RHS is there any chance that this will be supported > in PETSc in the near future? > > > > > > > > > >> On May 30, 2018, at 6:55 PM, Marius Buerkle <[email protected][mailto: > [email protected]]> wrote: > >> > >> Thanks for the quick fix, I will test it and report back. > >> I have another maybe related question, if MAT_NEW_NONZERO_LOCATIONS is > true and let's say 1 new nonzero position is created it does not allocated > 1 but several new nonzeros but only use 1. > > > > Correct > > > >> I think that is normal, right? > > > > Yes > > > >> But, at least as far as I understand the manual, a subsequent call of > mat assemble with > >> MAT_FINAL_ASSEMBLY should compress out the unused allocations and > release the memory, is this correct? > > > > It "compresses it out" (by shifting all the nonzero entries to the > beginning of the internal i, j, and a arrays), but does NOT release any > memory. Since the values are stored in one big contiguous array (obtained > with a single malloc) it cannot just free part of the array, so the extra > locations just sit harmlessly at the end if the array unused. > > > >> If so, this did not work for me, even after doing > >> MAT_FINAL_ASSEMBLY the unused nonzero allocations remain. Is this > normal? > > > > Yes, > > > > Barry > > > >> > >>> > >>> Fixed in the branch barry/fix-mat-new-nonzero-locations/maint > >>> > >>> Once this passes testing it will go into the maint branch and then the > next patch release but you can use it now in the branch > barry/fix-mat-new-nonzero-locations/maint > >>> > >>> Thanks for the report and reproducible example > >>> > >>> Barry > >>> > >>> > >>>> On May 29, 2018, at 7:51 PM, Marius Buerkle <[email protected][mailto: > [email protected]]> wrote: > >>>> > >>>> Sure, I made a small reproducer, it is Fortran though I hope that is > ok. If MAT_NEW_NONZERO_LOCATIONS is set to false I get an error, if it is > set to true the new nonzero element is inserted, if > MAT_NEW_NONZERO_LOCATIONS is false and either MAT_NEW_NONZERO_LOCATION_ERR > or MAT_NEW_NONZERO_ALLOCATION_ERR is set to false afterwards then the new > nonzero is also created without an error, but if MAT_NEW_NONZERO_LOCATIONS > is set to false after > MAT_NEW_NONZERO_LOCATION_ERR/MAT_NEW_NONZERO_ALLOCATION_ERR > have been set to false I get an error again. > >>>> > >>>> > >>>> program newnonzero > >>>> #include <petsc/finclude/petscmat.h> > >>>> use petscmat > >>>> implicit none > >>>> > >>>> Mat :: A > >>>> PetscInt :: dnnz,onnz,n,m,idxm(1),idxn(1),nl1,nl2 > >>>> PetscScalar :: v(1) > >>>> PetscReal :: info(MAT_INFO_SIZE) > >>>> PetscErrorCode :: ierr > >>>> > >>>> integer :: nproc,iproc,i > >>>> > >>>> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >>>> > >>>> call MPI_COMM_SIZE(PETSC_COMM_WORLD, nproc,ierr) > >>>> > >>>> call MPI_Comm_rank( PETSC_COMM_WORLD, iproc, ierr ) > >>>> > >>>> n=3 > >>>> m=n > >>>> call MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,m, > 1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,A,ierr) > >>>> > >>>> > >>>> call MatGetOwnershipRange(A,nl1,nl2,ierr) > >>>> do i=nl1,nl2-1 > >>>> idxn(1)=i > >>>> idxm(1)=i > >>>> v(1)=1d0 > >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) > >>>> end do > >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> > >>>> call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR > ,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) > >>>> > >>>> > >>>> idxn(1)=0 > >>>> idxm(1)=n-1 > >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then > >>>> v(1)=2d0 > >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) > >>>> end if > >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> > >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then > >>>> v(1)=2d0 > >>>> call MatGetValues(A,1,idxn,1,idxm, v,ierr) > >>>> write(6,*) v > >>>> end if > >>>> > >>>> call PetscFinalize(ierr) > >>>> > >>>> end program newnonzero > >>>> > >>>> > >>>> > >>>> $ mpiexec.hydra -n 3 ./a.out > >>>> [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >>>> [0]PETSC ERROR: Argument out of range > >>>> [0]PETSC ERROR: Inserting a new nonzero at global row/column (0, 2) > into matrix > >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > documentation/faq.html[http://www.mcs.anl.gov/petsc/ > documentation/faq.html] for trouble shooting. > >>>> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018 > >>>> [0]PETSC ERROR: ./a.out on a named tono-hpc1 by marius Wed May 30 > 09:42:40 2018 > >>>> [0]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/3.9.2 > --download-elemental=yes --download-metis=yes --download-parmetis=yes > --download-mumps=yes --with-scalapack-lib="/home/ > marius/intel/compilers_and_libraries_2018.2.199/linux/ > mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > -Wl,--end-group -lpthread -lm -ldl" --FC=mpiifort --CC=mpicc --CXX=mpicxx > --with-scalar-type=complex --with-mpi-dir= --with-blaslapack-lib="/home/ > marius/intel/compilers_and_libraries_2018.2.199/linux/ > mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > -Wl,--end-group -lpthread -lm -ldl" --with-cxx-dialect=C++11 > --download-superlu_dist=yes --download-ptscotch=yes --with-x > --with-debugging=1 --download-superlu=yes --with-mkl_cpardiso=1 > --with-mkl_pardiso=1 --with-scalapack=1 > >>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 607 in > /home/marius/prog/petsc/petsc-3.9.2/src/mat/impls/aij/mpi/mpiaij.c > >>>> [0]PETSC ERROR: #2 MatSetValues() line 1312 in > /home/marius/prog/petsc/petsc-3.9.2/src/mat/interface/matrix.c > >>>> (0.000000000000000E+000,0.000000000000000E+000) > >>>> > >>>> > >>>> > >>>> Please send complete error message; type of matrix used etc. Ideally > code that demonstrates the problem. > >>>> > >>>> Barry > >>>> > >>>> > >>>>> On May 29, 2018, at 3:31 AM, Marius Buerkle <[email protected][mailto: > [email protected]]> wrote: > >>>>> > >>>>> > >>>>> Hi, > >>>>> > >>>>> I tried to set MAT_NEW_NONZERO_LOCATIONS to false, as far as I > understood MatSetValues should simply ignore entries which would give rise > to new nonzero values not creating a new entry and not cause an error, but > I get "[1]PETSC ERROR: Inserting a new nonzero at global row/column". Is > this option supposed to work or not? > >>>> > >>> > >>> > > > >
