It sounds like you're right at the limit of what your computer can handle with a direct solver. Which direct solver are you using? I generally use MUMPS or SuperLU instead of PETSc's built in direct solver (e.g. -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package superlu_dist). I think SuperLU typically uses less RAM than MUMPS, so maybe you should try that?
Also, I know that with MUMPS you can try different fill-reduction algorithms, which can be controlled via PETSc's ictnl flags <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMumpsSetIcntl.html> . David On Sat, Feb 7, 2015 at 10:33 AM, Manav Bhatia <bhatiama...@gmail.com> wrote: > Hi, > > I am trying to understand the dof numbering a bit better. > > The simulation I have at hand is that of a solid mechanics problem with > thermal load. This has a total of ~770,000 dofs, and so far none of the > iterative solves in petsc has provided good convergence property. Hence, I > am stuck with using direct solvers. > > As a result, it is important for me to have a fill reducing dof > numbering for my problems. Some of my dofs are coupled only through the dof > constraints, not through a physical element connectivity. I have two > different sets of such coupling constraints. One set is applied > independently, and the second set is applied in addition to the first set. > > What I have observed is that the -ksp_type preonly -pc_type lu is able > to solve with only the first set of dof constraints applied (but takes a > lot of memory ~16 GB, which is all that my computer has). However, petsc > complains about insufficient memory when I apply the second set in addition > to the first. > > I suspect that renumbering the dofs using a fill-reducing criteria by > accounting for these dof constraints could address the problem. However, I > am not sure if the current code setup allows this to be done in libMesh. > What I see is that the partitioners are associated with the MeshBase class, > without any information from the DofMap. > > I would appreciate any insight in this matter. > > Thanks, > Manav > > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming. The Go Parallel Website, > sponsored by Intel and developed in partnership with Slashdot Media, is > your > hub for all things parallel software development, from weekly thought > leadership blogs to news, videos, case studies, tutorials and more. Take a > look and join the conversation now. http://goparallel.sourceforge.net/ > _______________________________________________ > Libmesh-users mailing list > Libmesh-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/libmesh-users > ------------------------------------------------------------------------------ Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Libmesh-users mailing list Libmesh-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libmesh-users
