Hello Jed, Thank you for your quick response. So I spent some time to dig deeper into my problem. I coded a shell script that passes through a bunch of ksp_type, pc_type and sub_pc_type. So please disregard the comment about the "does not converge properly for transpose". I had taken that conclusion from my own code (and not from the ex10 and extracted matrix), and a KSPSetFromOptions was missing. Apologies for that.
What remains is the performance issue. The MatSolveTranspose takes a very long time to converge. For a matrix of 3 million rows, MatSolveTranspose takes roughly 5 minutes on 64 cpus, whereas the MatSolve is almost instantaneous!. When I gdb my code, petsc seems to be stalled in the MatLUFactorNumeric_SeqAIJ_Inode () for a long time. I also did a top on the compute node to check the RAM usage. It was hovering over 2 gig, so memory usage does not seem to be an issue here. #0 0x00002afe8dfebd08 in MatLUFactorNumeric_SeqAIJ_Inode () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #1 0x00002afe8e07f15c in MatLUFactorNumeric () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #2 0x00002afe8e2afa99 in PCSetUp_ILU () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #3 0x00002afe8e337c0d in PCSetUp () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #4 0x00002afe8e39d643 in KSPSetUp () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #5 0x00002afe8e39e3ee in KSPSolveTranspose () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #6 0x00002afe8e300f8c in PCApplyTranspose_ASM () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #7 0x00002afe8e338c13 in PCApplyTranspose () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #8 0x00002afe8e3a8a84 in KSPInitialResidual () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #9 0x00002afe8e376c32 in KSPSolve_GMRES () from /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/libpetsc.so.3.5 #10 0x00002afe8e39e425 in KSPSolveTranspose () For that particular application, I was using: ksp_type: gmres pc_type: asm sub_pc_type: ilu adj_sub_pc_factor_levels 1 For small matrices, the MatSolveTranspose computing time is very similar to the simple MatSolve. And if I want to revert to a MatTranspose followed by the MatSolve, then the MatTranspose takes forever to finish... For a matrix of 3 million rows, MatTranspose takes 30 minutes on 64 cpus!! So thank you for implementing the transpose solve in superlu_dist. It would also be nice to have it with hypre. Let me know what you think and ideas on how to improve my computational time, Regards, Antoine -----Message d'origine----- De : Jed Brown [mailto:[email protected]] Envoyé : Thursday, August 28, 2014 5:01 PM À : Antoine De Blois; '[email protected]' Objet : Re: [petsc-users] superlu_dist and MatSolveTranspose Antoine De Blois <[email protected]> writes: > Hello everyone, > > I am trying to solve a A^T x = b system. For my applications, I had > realized that the MatSolveTranspose does not converge properly. What do you mean "does not converge properly"? Can you send a test case where the transpose solve should be equivalent, but is not? We have only a few tests for transpose solve and not all preconditioners support it, but where it is supported, we want to ensure that it is correct. > Therefore, I had implemented a MatTranspose followed by a MatSolve. > This proved to converge perfectly (which is strange since the > transposed matrix has the same eigenvalues as the untransposed...). > The problem is that for bigger matrices, the MatTranspose is very > costly and thus cannot be used. Costly in terms of memory? (I want you to be able to use KSPSolveTranspose, but I'm curious what you're experiencing.) > I tried using the superlu_dist package. Although it the package works > perfectly for the MatSolve, I get the an "No support for this operation for > this object type" error with MatSolveTransopse. I reproduced the error using > the MatView an ex10 tutorial. I can provide the matrix and rhs upon request. > My command line was: > > ex10 -f0 A_and_rhs.bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist -trans > > So it there an additional parameter I need to use for the transposed solve? > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Matrix type mpiaij This is easy to add. I'll do it now. > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown [0]PETSC ERROR: > /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/examples/t > utorials/ex10 on a ARGUS_impi_opt named hpc-user11 by ad007804 Thu Aug > 28 16:41:15 2014 [0]PETSC ERROR: Configure options --CFLAGS="-xHost > -axAVX" --download-hypre --download-metis --download-ml > --download-parmetis --download-scalapack --download-superlu_dist > --download-mumps --with-c2html=0 --with-cc=mpiicc --with-fc=mpiifort > --with-cxx=mpiicpc --with-debugging=yes > --prefix=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/petsc-3.5.1 > --with-cmake=/gpfs/fs1/aero/SOFTWARE/TOOLS/CMAKE/cmake-2.8.7/bin/cmake > --with-valgrind=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/valgrind-3.9 > .0/bin/valgrind --with-shared-libraries=0 [0]PETSC ERROR: #1 > MatSolveTranspose() line 3473 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/mat/interface/matr > ix.c [0]PETSC ERROR: #2 PCApplyTranspose_LU() line 214 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/impls/facto > r/lu/lu.c [0]PETSC ERROR: #3 PCApplyTranspose() line 573 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/interface/p > recon.c [0]PETSC ERROR: #4 KSP_PCApply() line 233 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/include/petsc-private/ > kspimpl.h [0]PETSC ERROR: #5 KSPInitialResidual() line 63 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interface/ > itres.c [0]PETSC ERROR: #6 KSPSolve_GMRES() line 234 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/impls/gmre > s/gmres.c [0]PETSC ERROR: #7 KSPSolveTranspose() line 704 in > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interface/ > itfunc.c [0]PETSC ERROR: #8 main() line 324 in > /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/examples/t > utorials/ex10.c > > FYI, the transpose solve is a typical application for adjoint optimization. > There should be a big adjoint community of developers that try to solve the > transposed matrix. > > Any help is much appreciated, > Best, > Antoine > > > Antoine DeBlois > Specialiste ingenierie, MDO lead / Engineering Specialist, MDO lead > Aéronautique / Aerospace 514-855-5001, x 50862 > [email protected]<mailto:[email protected] > dier.com> > > 2351 Blvd Alfred-Nobel > Montreal, Qc > H4S 1A9 > > [Description : Description : > http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg] > CONFIDENTIALITY NOTICE - This communication may contain privileged or > confidential information. > If you are not the intended recipient or received this communication > by error, please notify the sender and delete the message without > copying
