Hung, I did not see any problem calling MatView on 8 nodes of XT4 (32 cores). I was able to save a 900Kx900K (40 nonzeros per row) sparse matrix within a few seconds.
Thanks, ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix.dat",FILE_MODE_WRITE,&view); CHKERRQ(ierr); ierr = MatView(A,view);CHKERRQ(ierr); ierr = PetscViewerDestroy(view);CHKERRQ(ierr); ================================ ?Keita Teranishi ?Scientific Library Group ?Cray, Inc. ?keita at cray.com ================================ -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:[email protected]] On Behalf Of Nguyen, Hung V ERDC-ITL-MS Sent: Friday, September 04, 2009 3:59 PM To: PETSc users list Subject: RE: Problem with saving the binary matrix via Matview Hello, Yes, It is a CrayXT4. I just rerun with setting the variables below. I will let you know if it helps. hvnguyen:jade20% setenv MPICH_PTL_SEND_CREDITS -1 hvnguyen:jade20% setenv MPICH_MAX_SHORT_MSG_SIZE 64000 hvnguyen:jade20% setenv MPICH_UNEX_BUFFER_SIZE 240M hvnguyen:jade20% setenv MPICH_PTL_UNEX_EVENTS 60000 >The problems will be worse if you don't have the latest Cray software on the system. What do you mean exactly about the latest Cray software? >Note we generally frown up saving huge matrices to disk except for debugging testing purposes. But as Matt notes this is not a particularly huge matrix. With any reasonable configuration it should take very little time to write the file. We need to dump out some matrices required large number of iterations for testing. As for small application case (nrows-12856,nnzeros=675744), it took a minute to write binary matrix to a file on CrayXT4 system so I don't know why it has a deadlock in this case. Thank you. -hung -----Original Message----- From: [email protected] [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Friday, September 04, 2009 3:24 PM To: PETSc users list Subject: Re: Problem with saving the binary matrix via Matview Is this a Cray? You may need to set the environmental variables MPI UNEX BUFFER SIZE and/or MPICH PTL MATCH OFF and/or MPICH PTL OTHER EVENTS and/or MPICH MSGS PER PROC and/or MPICH PTL SEND CREDITS you will likely need to hunt through Cray documentation to find the meaning of all this stuff. The problems will be worse if you don't have the latest Cray software on the system. Note we generally frown up saving huge matrices to disk except for debugging testing purposes. But as Matt notes this is not a particularly huge matrix. With any reasonable configuration it should take very little time to write the file. Good luck, Barry On Sep 4, 2009, at 2:25 PM, Nguyen, Hung V ERDC-ITL-MS wrote: > > Hello, > > I have a problem with saving a matrix with the binary viewer > PetscBinaryViewerOpen() for the matrix with nrow=853564, > nnzeros=47191472. > The application ran with 32 pes for more than 3 hours without writing > any into file. > However, the same executable ran for small size of application > (nrows-12856,nnzeros=675744) and were able to write successfully > binary files. > > Note: using Totalview I found that it hang in the line of > Matview(matrix->petsc, fd); > > Thank you, > > -hung > > --- code: > petsc_analyst_mat(matrix->petsc); > sprintf(file[LOAD_MATRIX],"Matrix.at%f",t_prev); > PetscViewerBinaryOpen(PETSC_COMM_WORLD, > file[LOAD_MATRIX],FILE_MODE_WRITE,&fd); > MatView(matrix->petsc, fd); > PetscViewerDestroy(fd); > > ---- Matrix info using salsa/AnaMod module: > > Computed <simple:trace> as <9.362113e+03> Computed <simple:trace-abs> > as <9.362623e+03> Computed <simple:norm1> as <2.800000e+01> Computed > <simple:normInf> as <2.800000e+01> Computed <simple:normF> as > <2.404511e+02> Computed <simple:diagonal-dominance> as <-3.064463e-02> > Could not compute <simple:symmetry-snorm> Could not compute > <simple:symmetry-anorm> Could not compute <simple:symmetry-fsnorm> > Could not compute <simple:symmetry-fanorm> Could not compute > <structure:n-struct-unsymm> Computed <structure:nrows> as <853564> > Could not compute <structure:symmetry> Computed <structure:nnzeros> as > <47191472> Computed <structure:max-nnzeros-per-row> as <112> Computed > <structure:min-nnzeros-per-row> as <16>
