Varun, I created a branch hzhang/feature-mumps-mem-estimate, see https://gitlab.com/petsc/petsc/-/merge_requests/4727
You may give it a try and let me know if this is what you want. src/ksp/ksp/tutorials/ex52.c is an example. Hong ________________________________ From: Varun Hiremath <[email protected]> Sent: Monday, January 17, 2022 12:41 PM To: Zhang, Hong <[email protected]> Cc: Jose E. Roman <[email protected]>; Peder Jørgensgaard Olesen via petsc-users <[email protected]> Subject: Re: [petsc-users] PETSc MUMPS interface Hi Hong, Thanks for looking into this. Here is the workflow that I might use: MatLUFactorSymbolic(F,A,perm,iperm,&info); // get memory estimates from MUMPS e.g. INFO(3), INFOG(16), INFOG(17) // find available memory on the system e.g. RAM size if (estimated_memory > available_memory) { // inform and stop; or // switch MUMPS to out-of-core factorization ICNTL(22) = 1; } else { // set appropriate settings for in-core factorization } // Now we call the solve and inside if MatLUFactorSymbolic is already called then it should be skipped EPSSolve(eps); Thanks, Varun On Mon, Jan 17, 2022 at 9:18 AM Zhang, Hong <[email protected]<mailto:[email protected]>> wrote: Varun, I am trying to find a way to enable you to switch options after MatLUFactorSymbolic(). A hack is modifying the flag 'mumps->matstruc' inside MatLUFactorSymbolic_AIJMUMPS() and MatFactorNumeric_MUMPS(). My understanding of what you want is: // collect mumps memory info ... MatLUFactorSymbolic(F,A,perm,iperm,&info); printMumpsMemoryInfo(F); //--------- if (memory is available) { EPSSolve(eps); --> skip calling of MatLUFactorSymbolic() } else { //out-of-core (OOC) option in MUMPS } Am I correct? I'll let you know once I work out a solution. Hong ________________________________ From: Varun Hiremath <[email protected]<mailto:[email protected]>> Sent: Sunday, January 16, 2022 10:10 PM To: Zhang, Hong <[email protected]<mailto:[email protected]>> Cc: Jose E. Roman <[email protected]<mailto:[email protected]>>; Peder Jørgensgaard Olesen via petsc-users <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] PETSc MUMPS interface Hi Jose, Hong, Thanks for the explanation. I have verified using -log_view that MatLUFactorSymbolic is indeed getting called twice. Hong, we use MUMPS solver for other things, and we typically run the symbolic analysis first and get memory estimates to ensure that we have enough memory available to run the case. If the available memory is not enough, we can stop or switch to the out-of-core (OOC) option in MUMPS. We wanted to do the same when using MUMPS via SLEPc/PETSc. Please let me know if there are other ways of getting these memory stats and switching options during runtime with PETSc. Appreciate your help! Thanks, Varun On Sun, Jan 16, 2022 at 4:01 PM Zhang, Hong <[email protected]<mailto:[email protected]>> wrote: Varun, I believe Jose is correct. You may verify it by running your code with option '-log_view', then check the number of calls to MatLUFactorSym. I guess I can add a flag in PCSetUp() to check if user has already called MatLUFactorSymbolic() and wants to skip it. Normally, users simply allocate sufficient memory in the symbolic factorization. Why do you want to check it? Hong ________________________________ From: Jose E. Roman <[email protected]<mailto:[email protected]>> Sent: Sunday, January 16, 2022 5:11 AM To: Varun Hiremath <[email protected]<mailto:[email protected]>> Cc: Zhang, Hong <[email protected]<mailto:[email protected]>>; Peder Jørgensgaard Olesen via petsc-users <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] PETSc MUMPS interface Hong may give a better answer, but if you look at PCSetUp_LU() https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU you will see that MatLUFactorSymbolic() is called unconditionally during the first PCSetUp(). Currently there is no way to check if the user has already called MatLUFactorSymbolic(). Jose > El 16 ene 2022, a las 10:40, Varun Hiremath > <[email protected]<mailto:[email protected]>> escribió: > > Hi Hong, > > Thank you, this is very helpful! > > Using this method I am able to get the memory estimates before the actual > solve, however, I think my code may be causing the symbolic factorization to > be run twice. Attached is my code where I am using SLEPc to compute > eigenvalues, and I use MUMPS for factorization. I have commented above the > code that computes the memory estimates, could you please check and tell me > if this would cause the symbolic factor to be computed twice (a second time > inside EPSSolve?), as I am seeing a slight increase in the overall > computation time? > > Regards, > Varun > > On Wed, Jan 12, 2022 at 7:58 AM Zhang, Hong > <[email protected]<mailto:[email protected]>> wrote: > PCFactorSetMatSolverType(pc,MATSOLVERMUMPS); > PCFactorSetUpMatSolverType(pc); > PCFactorGetMatrix(pc,&F); > > MatLUFactorSymbolic(F,A,...) > You must provide row and column permutations etc, > petsc/src/mat/tests/ex125.c may give you a clue on how to get these inputs. > > Hong > > > From: petsc-users > <[email protected]<mailto:[email protected]>> on > behalf of Junchao Zhang > <[email protected]<mailto:[email protected]>> > Sent: Wednesday, January 12, 2022 9:03 AM > To: Varun Hiremath <[email protected]<mailto:[email protected]>> > Cc: Peder Jørgensgaard Olesen via petsc-users > <[email protected]<mailto:[email protected]>> > Subject: Re: [petsc-users] PETSc MUMPS interface > > Calling PCSetUp() before KSPSetUp()? > > --Junchao Zhang > > > On Wed, Jan 12, 2022 at 3:00 AM Varun Hiremath > <[email protected]<mailto:[email protected]>> wrote: > Hi All, > > I want to collect MUMPS memory estimates based on the initial symbolic > factorization analysis before the actual numerical factorization starts to > check if the estimated memory requirements fit the available memory. > > I am following the steps from > https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html > > PCFactorSetMatSolverType(pc,MATSOLVERMUMPS); > PCFactorSetUpMatSolverType(pc); > PCFactorGetMatrix(pc,&F); > > KSPSetUp(ksp); > MatMumpsGetInfog(F,...) > > But it appears KSPSetUp calls both symbolic and numerical factorization. So > is there some other way to get these statistics before the actual > factorization starts? > > Thanks, > Varun > <slepc_eps_mumps_test.cpp>
