Antoine, I just find out that superlu_dist does not support MatSolveTransport yet (see Sherry's email below). Once superlu_dist provides this support, we can add it to the petsc/superlu_dist interface.
Thanks for your patience. Hong ------------------------------------------- Hong, Sorry, the transposed solve is not there yet; it's not as simple as serial version, because here, it requires to set up entirely different communication pattern. I will try to find time to do it. Sherry On Tue, Sep 23, 2014 at 8:11 AM, Hong <[email protected]> wrote: > > Sherry, > Can superlu_dist be used for solving A^T x = b? > > Using the option > options.Trans = TRANS; > with the existing petsc-superlu_dist interface, I cannot get correct solution. > > Hong On Mon, Sep 22, 2014 at 12:47 PM, Hong <[email protected]> wrote: > I'll add it. It would not take too long, just matter of priority. > I'll try to get it done in a day or two, then let you know when it works. > > Hong > > On Mon, Sep 22, 2014 at 12:11 PM, Antoine De Blois > <[email protected]> wrote: >> Dear all, >> >> Sorry for the delay on this topic. >> >> Thank you Gaetan for your suggestion. I had thought about doing that >> originally, but I had left it out since I thought that a rank owned the >> entire row of the matrix (and not only the sub-diagonal part). I will >> certainly give it a try. >> >> I still need the MatSolveTranspose since I need the ability to reuse the >> residual jacobian matrix from the flow (a 1st order approximation of it), >> which is assembled in a non-transposed format. This way the adjoint system >> is solved in a pseudo-time step manner, where the product of the exact >> jacobian matrix and the adjoint vector is used as a source term in the rhs. >> >> Hong, do you have an estimation of the time required to implement it in >> superlu_dist? >> >> Best, >> Antoine >> >> -----Message d'origine----- >> De : Hong [mailto:[email protected]] >> Envoyé : Friday, August 29, 2014 9:14 PM >> À : Gaetan Kenway >> Cc : Antoine De Blois; [email protected] >> Objet : Re: [petsc-users] superlu_dist and MatSolveTranspose >> >> We can add MatSolveTranspose() to the petsc interface with superlu_dist. >> >> Jed, >> Are you working on it? If not, I can work on it. >> >> Hong >> >> On Fri, Aug 29, 2014 at 6:14 PM, Gaetan Kenway <[email protected]> wrote: >>> Hi Antoine >>> >>> We are also using PETSc for solving adjoint systems resulting from >>> CFD. To get around the matSolveTranspose issue we just assemble the >>> transpose matrix directly and then call KSPSolve(). If this is >>> possible in your application I think it is probably the best approach >>> >>> Gaetan >>> >>> >>> On Fri, Aug 29, 2014 at 3:58 PM, Antoine De Blois >>> <[email protected]> wrote: >>>> >>>> Hello Jed, >>>> >>>> Thank you for your quick response. So I spent some time to dig deeper >>>> into my problem. I coded a shell script that passes through a bunch >>>> of ksp_type, pc_type and sub_pc_type. So please disregard the comment >>>> about the "does not converge properly for transpose". I had taken >>>> that conclusion from my own code (and not from the ex10 and extracted >>>> matrix), and a KSPSetFromOptions was missing. Apologies for that. >>>> >>>> What remains is the performance issue. The MatSolveTranspose takes a >>>> very long time to converge. For a matrix of 3 million rows, >>>> MatSolveTranspose takes roughly 5 minutes on 64 cpus, whereas the >>>> MatSolve is almost instantaneous!. When I gdb my code, petsc seems to >>>> be stalled in the MatLUFactorNumeric_SeqAIJ_Inode () for a long time. >>>> I also did a top on the compute node to check the RAM usage. It was >>>> hovering over 2 gig, so memory usage does not seem to be an issue here. >>>> >>>> #0 0x00002afe8dfebd08 in MatLUFactorNumeric_SeqAIJ_Inode () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #1 0x00002afe8e07f15c in MatLUFactorNumeric () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #2 0x00002afe8e2afa99 in PCSetUp_ILU () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #3 0x00002afe8e337c0d in PCSetUp () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #4 0x00002afe8e39d643 in KSPSetUp () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #5 0x00002afe8e39e3ee in KSPSolveTranspose () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #6 0x00002afe8e300f8c in PCApplyTranspose_ASM () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #7 0x00002afe8e338c13 in PCApplyTranspose () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #8 0x00002afe8e3a8a84 in KSPInitialResidual () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #9 0x00002afe8e376c32 in KSPSolve_GMRES () >>>> from >>>> /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.5.1/lib/li >>>> bpetsc.so.3.5 >>>> #10 0x00002afe8e39e425 in KSPSolveTranspose () >>>> >>>> For that particular application, I was using: >>>> ksp_type: gmres >>>> pc_type: asm >>>> sub_pc_type: ilu >>>> adj_sub_pc_factor_levels 1 >>>> >>>> For small matrices, the MatSolveTranspose computing time is very >>>> similar to the simple MatSolve. >>>> >>>> And if I want to revert to a MatTranspose followed by the MatSolve, >>>> then the MatTranspose takes forever to finish... For a matrix of 3 >>>> million rows, MatTranspose takes 30 minutes on 64 cpus!! >>>> >>>> So thank you for implementing the transpose solve in superlu_dist. It >>>> would also be nice to have it with hypre. >>>> Let me know what you think and ideas on how to improve my >>>> computational time, Regards, Antoine >>>> >>>> -----Message d'origine----- >>>> De : Jed Brown [mailto:[email protected]] Envoyé : Thursday, August >>>> 28, 2014 5:01 PM À : Antoine De Blois; '[email protected]' >>>> Objet : Re: [petsc-users] superlu_dist and MatSolveTranspose >>>> >>>> Antoine De Blois <[email protected]> writes: >>>> >>>> > Hello everyone, >>>> > >>>> > I am trying to solve a A^T x = b system. For my applications, I had >>>> > realized that the MatSolveTranspose does not converge properly. >>>> >>>> What do you mean "does not converge properly"? Can you send a test >>>> case where the transpose solve should be equivalent, but is not? We >>>> have only a few tests for transpose solve and not all preconditioners >>>> support it, but where it is supported, we want to ensure that it is >>>> correct. >>>> >>>> > Therefore, I had implemented a MatTranspose followed by a MatSolve. >>>> > This proved to converge perfectly (which is strange since the >>>> > transposed matrix has the same eigenvalues as the untransposed...). >>>> > The problem is that for bigger matrices, the MatTranspose is very >>>> > costly and thus cannot be used. >>>> >>>> Costly in terms of memory? (I want you to be able to use >>>> KSPSolveTranspose, but I'm curious what you're experiencing.) >>>> >>>> > I tried using the superlu_dist package. Although it the package >>>> > works perfectly for the MatSolve, I get the an "No support for this >>>> > operation for this object type" error with MatSolveTransopse. I >>>> > reproduced the error using the MatView an ex10 tutorial. I can >>>> > provide the matrix and rhs upon request. My command line was: >>>> > >>>> > ex10 -f0 A_and_rhs.bin -pc_type lu -pc_factor_mat_solver_package >>>> > superlu_dist -trans >>>> > >>>> > So it there an additional parameter I need to use for the >>>> > transposed solve? >>>> > >>>> > [0]PETSC ERROR: --------------------- Error Message >>>> > -------------------------------------------------------------- >>>> > [0]PETSC ERROR: No support for this operation for this object type >>>> > [0]PETSC ERROR: Matrix type mpiaij >>>> >>>> This is easy to add. I'll do it now. >>>> >>>> > [0]PETSC ERROR: See >>>> > http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> > for trouble shooting. >>>> > [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown [0]PETSC ERROR: >>>> > /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/example >>>> > s/t >>>> > utorials/ex10 on a ARGUS_impi_opt named hpc-user11 by ad007804 Thu >>>> > Aug >>>> > 28 16:41:15 2014 [0]PETSC ERROR: Configure options --CFLAGS="-xHost >>>> > -axAVX" --download-hypre --download-metis --download-ml >>>> > --download-parmetis --download-scalapack --download-superlu_dist >>>> > --download-mumps --with-c2html=0 --with-cc=mpiicc >>>> > --with-fc=mpiifort --with-cxx=mpiicpc --with-debugging=yes >>>> > --prefix=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/petsc-3.5. >>>> > 1 >>>> > --with-cmake=/gpfs/fs1/aero/SOFTWARE/TOOLS/CMAKE/cmake-2.8.7/bin/cm >>>> > ake >>>> > --with-valgrind=/gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/valgrind- >>>> > 3.9 .0/bin/valgrind --with-shared-libraries=0 [0]PETSC ERROR: #1 >>>> > MatSolveTranspose() line 3473 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/mat/interface/m >>>> > atr ix.c [0]PETSC ERROR: #2 PCApplyTranspose_LU() line 214 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/impls/fa >>>> > cto r/lu/lu.c [0]PETSC ERROR: #3 PCApplyTranspose() line 573 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/pc/interfac >>>> > e/p recon.c [0]PETSC ERROR: #4 KSP_PCApply() line 233 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/include/petsc-priva >>>> > te/ kspimpl.h [0]PETSC ERROR: #5 KSPInitialResidual() line 63 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interfa >>>> > ce/ itres.c [0]PETSC ERROR: #6 KSPSolve_GMRES() line 234 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/impls/g >>>> > mre s/gmres.c [0]PETSC ERROR: #7 KSPSolveTranspose() line 704 in >>>> > /gpfs/fs2/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/interfa >>>> > ce/ itfunc.c [0]PETSC ERROR: #8 main() line 324 in >>>> > /gpfs/fs1/aero/SOFTWARE/TOOLS/PROGRAMMING/petsc/src/ksp/ksp/example >>>> > s/t >>>> > utorials/ex10.c >>>> > >>>> > FYI, the transpose solve is a typical application for adjoint >>>> > optimization. There should be a big adjoint community of developers >>>> > that try to solve the transposed matrix. >>>> > >>>> > Any help is much appreciated, >>>> > Best, >>>> > Antoine >>>> > >>>> > >>>> > Antoine DeBlois >>>> > Specialiste ingenierie, MDO lead / Engineering Specialist, MDO lead >>>> > Aéronautique / Aerospace 514-855-5001, x 50862 >>>> > [email protected]<mailto:[email protected] >>>> > bar >>>> > dier.com> >>>> > >>>> > 2351 Blvd Alfred-Nobel >>>> > Montreal, Qc >>>> > H4S 1A9 >>>> > >>>> > [Description : Description : >>>> > http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg] >>>> > CONFIDENTIALITY NOTICE - This communication may contain privileged >>>> > or confidential information. >>>> > If you are not the intended recipient or received this >>>> > communication by error, please notify the sender and delete the >>>> > message without copying >>> >>>
