Sorry, I overlooked your attachment, which gives '-log_summary': 1.txt: MatSolve 2 1.0 9.7397e-02 1.0 0.00e+00 0.0 5.4e+02 5.5e+03 6.0e+00 0 0 34 10 10 0 0 34 10 11 0 MatLUFactorSym 1 1.0 1.2882e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.0e+00 6 0 0 0 12 6 0 0 0 12 0 MatLUFactorNum 1 1.0 1.8813e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 90 0 0 0 2 90 0 0 0 2 0
2.txt: MatSolve 2 1.0 1.0811e-01 1.0 0.00e+00 0.0 4.9e+02 6.1e+03 6.0e+00 0 0 31 10 10 0 0 31 10 11 0 MatLUFactorSym 1 1.0 1.8920e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.0e+00 8 0 0 0 12 8 0 0 0 12 0 MatLUFactorNum 1 1.0 2.1836e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 89 0 0 0 2 89 0 0 0 2 0 Again, only 1st solve calls LU factorization, which dominates as expected. MatLUFactorSym is ignorable, but matrix ordering makes noticable effect. I would stay with sequential MatLUFactorSym and experiment different matrix orderings using '-mat_mumps_icntl_7 <>'. Hong On Wed, Jan 29, 2014 at 11:33 AM, Hong Zhang <[email protected]> wrote: > Tabrez: > >> I am getting the opposite result, i.e., MUMPS becomes slower when using >> ParMETIS for parallel ordering. What did I mess up? Is the problem too >> small? >> > > I saw similar performance when adding parallel symbolic factorization into > petsc interface, thus I did not set > parallel symbolic factorization as default for petsc/mumps interface. > How large is your matrix? > > Can you send us output of '-log_summary' for these two runs? > > Hong > >> >> >> Case 1 took 24.731s >> >> $ rm -f *vtk; time mpiexec -n 16 ./defmod -f point.inp -pc_type lu >> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_4 1 -log_summary > >> 1.txt >> >> >> Case 2 with "-mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2" took 34.720s >> >> $ rm -f *vtk; time mpiexec -n 16 ./defmod -f point.inp -pc_type lu >> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_4 1 -log_summary >> -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 > 2.txt >> >> >> Both 1.txt and 2.txt are attached. >> >> Regards, >> >> Tabrez >> >> >> On 01/29/2014 09:18 AM, Hong Zhang wrote: >> >> MUMPS now supports parallel symbolic factorization. With petsc-3.4 >> interface, you can use runtime option >> >> -mat_mumps_icntl_28 <1>: ICNTL(28): use 1 for sequential analysis and >> ictnl(7) ordering, or 2 for parallel analysis and ictnl(29) ordering >> -mat_mumps_icntl_29 <0>: ICNTL(29): parallel ordering 1 = ptscotch 2 = >> parmetis >> >> e.g, '-mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2' activates parallel >> symbolic factorization with pametis for matrix ordering. >> Give it a try and let us know what you get. >> >> Hong >> >> >> On Tue, Jan 28, 2014 at 5:48 PM, Smith, Barry F. <[email protected]>wrote: >> >>> >>> On Jan 28, 2014, at 5:39 PM, Matthew Knepley <[email protected]> wrote: >>> >>> > On Tue, Jan 28, 2014 at 5:25 PM, Tabrez Ali <[email protected]> >>> wrote: >>> > Hello >>> > >>> > This is my observation as well (with MUMPS). The first solve (after >>> assembly which is super fast) takes a few mins (for ~1 million unknowns on >>> 12/24 cores) but from then on only a few seconds for each subsequent solve >>> for each time step. >>> > >>> > Perhaps symbolic factorization in MUMPS is all serial? >>> > >>> > Yes, it is. >>> >>> I missed this. I was just assuming a PETSc LU. Yes, I have no idea >>> of relative time of symbolic and numeric for those other packages. >>> >>> Barry >>> > >>> > Matt >>> > >>> > Like the OP I often do multiple runs on the same problem but I dont >>> know if MUMPS or any other direct solver can save the symbolic >>> factorization info to a file that perhaps can be utilized in subsequent >>> reruns to avoid the costly "first solves". >>> > >>> > Tabrez >>> > >>> > >>> > On 01/28/2014 04:04 PM, Barry Smith wrote: >>> > On Jan 28, 2014, at 1:36 PM, David Liu<[email protected]> wrote: >>> > >>> > Hi, I'm writing an application that solves a sparse matrix many times >>> using Pastix. I notice that the first solves takes a very long time, >>> > Is it the first "solve" or the first time you put values into that >>> matrix that "takes a long time"? If you are not properly preallocating the >>> matrix then the initial setting of values will be slow and waste memory. >>> See >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatXAIJSetPreallocation.html >>> > >>> > The symbolic factorization is usually much faster than a numeric >>> factorization so that is not the cause of the slow "first solve". >>> > >>> > Barry >>> > >>> > >>> > >>> > while the subsequent solves are very fast. I don't fully understand >>> what's going on behind the curtains, but I'm guessing it's because the very >>> first solve has to read in the non-zero structure for the LU factorization, >>> while the subsequent solves are faster because the nonzero structure >>> doesn't change. >>> > >>> > My question is, is there any way to save the information obtained from >>> the very first solve, so that the next time I run the application, the very >>> first solve can be fast too (provided that I still have the same nonzero >>> structure)? >>> > >>> > >>> > -- >>> > No one trusts a model except the one who wrote it; Everyone trusts an >>> observation except the one who made it- Harlow Shapley >>> > >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> >>> >> >> >> -- >> No one trusts a model except the one who wrote it; Everyone trusts an >> observation except the one who made it- Harlow Shapley >> >> >
