Jed Brown writes: > On Fri, Dec 2, 2011 at 21:06, Dave Nystrom <Dave.Nystrom at > tachyonlogic.com>wrote: > > > It seems that whenever I am running my code with petsc and run top on linux > > it reports that my code is running something like 24-25 GB of memory, even > > if I am running a smallish problem. Is this normal? I just had a problem > > terminate because it ran out of memory and I had no idea that I was pushing > > any memory boundaries. I actually don't think the problem should have run > > out of memory but this behavior with top seems to keep it from being a > > useful tool for getting a sense of how much memory my code is using. > > 1. Check -log_summary to see which object types the large allocations are > coming from.
The output of -log_summary never seems to agree with what I see in top. BTW, I'm running Centos 5.x on my system. It is probably about 5.5 or 5.6 based on the last time I did a yum update. > 2. What are you running? I'm running a variety of things. My code currently has 6 linear solves that are being called. Different methods work better for different linear solves. And I am doing lots of experimenting with different krylov solvers and preconditioners. So for krylov solvers, I have been using mostly cg and also trying minres, symmlq and gmres. For preconditioners, I have been trying jacobi, block jacobi, lu, cholesky, ilu, icc, sor, sacusp and ainvcusp. When I run my code using our native, home grown cg solver with a cholesky preconditioner, top reports for my current problem size memory usage of 567 MB. My current problem size is using a 2d grid that is 200x602 - not really all that big. And yet when I am running petsc solvers, top is reporting 24 to 26 GB whether or not I am using cuda. So do you guys not see this sort of behavior with top? I see this sort of behavior consistently, i.e. exclusively including with a petsc example problem. And according to dmesg, my previous job running with petsc ran out of physical memory and swap space. I have 12 GB of physical memory and about 14 GB of swap space. But this is the first time I have run out of memory on the cpu. I have scaled up a petsc example problem large enough to get a failed memory allocation for the gpu however.