First question: Are your matrices sparse or dense? * A is a dense matrix at the order of 225,000 of complex*16;
How large? * 800GB on disk What type of math equations, e.g., solve Ax=b or eigenvalue problems? * A'x=b, where A'=[A^-1 U B]= sparse matrix at the order of 2 million. B = sparse matrix to be mixed and mingled with A^-1. Hong Jinquan On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong <jzhong at scsolutions.com<mailto:jzhong at scsolutions.com>> wrote: Thanks, Prof. Zhang. I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could a big move now to eliminate ScaLAPACK. Jinquan From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [mailto:petsc-users-bounces at mcs.anl.gov<mailto:[email protected]>] On Behalf Of Hong Zhang Sent: Wednesday, August 08, 2012 3:29 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) Jinquan: You need understand some basic concepts in numerical computations, for example: - dense matrix vs. sparse matrix and their data structures - algorithms for solve Ax=b - available software packages, their targeted applications See www.mcs.anl.gov/~hzhang/teach/cs595<http://www.mcs.anl.gov/~hzhang/teach/cs595> for a list of references. Hong What I am trying to say is that * allocated 1D array A in PETSc * passed 1D A into ScaLAPACK * obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array * indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK * constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1. I can have the last step done in using ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetUp(A);CHKERRQ(ierr); ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr); It just took too long. This is the performance issue I am having. To improve the performance, it is necessary for preallocation. The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz since there are no diagonal and off-diagonal parts defined in A^-1. Jinquan From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [mailto:petsc-users-bounces at mcs.anl.gov<mailto:[email protected]>] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 3:04 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong <jzhong at scsolutions.com<mailto:jzhong at scsolutions.com>> wrote: It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc. What are you trying to say? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120808/de356d20/attachment-0001.html>
