Jinquan: Has this problem or similar problem been solved using scalapack? Your problem is virtually a dense problem, why choose petsc library? What type of computer, number of cores, to be used for this problem?
Hong First question:**** > > Are your matrices sparse or dense?**** > > **? **A is a dense matrix at the order of 225,000 of complex*16; > **** > > ** ** > > How large? **** > > **? **800GB on disk**** > > ** ** > > What type of math equations, e.g., solve Ax=b or eigenvalue problems?**** > > **? **A?x=b, where A?=[A^-1 U B]= sparse matrix at the order of 2 > million. B = sparse matrix to be mixed and mingled with A^-1.**** > > ** ** > > Hong**** > > ** ** > > Jinquan**** > > On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong <jzhong at scsolutions.com> > wrote:**** > > Thanks, Prof. Zhang.**** > > **** > > I could have used PETSc directly for this application. However, I have to > use this procedure for QA (quality assurance) purposes since this > parallelizing process for a program I am working on is a standard program > widely used in industry. That is, QA purposes have driven us to use > ScaLAPACK for inversion first then PETSc for solution. I wish I could a > big move now to eliminate ScaLAPACK.**** > > **** > > Jinquan**** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Hong Zhang > *Sent:* Wednesday, August 08, 2012 3:29 PM**** > > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > **** > > Jinquan:**** > > **** > > You need understand some basic concepts in numerical computations, for > example:**** > > - dense matrix vs. sparse matrix and their data structures**** > > - algorithms for solve Ax=b**** > > - available software packages, their targeted applications**** > > **** > > See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references.**** > > **** > > Hong**** > > **** > > What I am trying to say is that **** > > ? allocated 1D array A in PETSc**** > > ? passed 1D A into ScaLAPACK**** > > ? obtained the indices of A^-1 computed from ScaLAPACK and stored > them in a 1D array**** > > ? indexed A^-1 as a 1D array in PETSc using the indices obtained > from ScaLAPACK**** > > ? constructed a sparse matrix, including A^-1 and others, using a > new set of indices for A^-1. **** > > I can have the last step done in using **** > > **** > > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > **** > > ierr = > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);**** > > ierr = MatSetFromOptions(A);CHKERRQ(ierr);**** > > ierr = MatSetUp(A);CHKERRQ(ierr);**** > > **** > > ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);**** > > **** > > It just took too long. This is the performance issue I am having. **** > > **** > > To improve the performance, it is necessary for preallocation. The > dilemma is: *MatCreateAIJ* and *MatXCreateAIJ* complained the setting of > *d_nz, > d_nnz,o_nz, o_nnz* since there are no diagonal and off-diagonal parts > defined in A^-1.**** > > **** > > Jinquan**** > > **** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Wednesday, August 08, 2012 3:04 PM**** > > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > **** > > On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong <jzhong at scsolutions.com> > wrote:**** > > It is a 1D matrix in PETSc. I just passed it as a 2D matrix in > ScaLAPACK. It is still a 1D matrix when get back into PETSc.**** > > **** > > What are you trying to say?**** > > **** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120808/29ea0b98/attachment.html>
