Hi Derek, On Tue, Jun 24, 2008 at 1:30 PM, Derek Gaston <[EMAIL PROTECTED]> wrote: > > Here is how much each proc is using: > > #CPU : #MB/proc > 256 : 200-700 > 128 : 350-700 > 64 : 450-800
Thanks! These numbers are pretty interesting. I know there are some places in the code where we go ahead and build a serial std::vector of size global n_elem or n_nodes. I think this happens sometimes on all processors and sometimes on just one, so it's possible we will run out of memory on one CPU way before the others... > > First thing to note is that the #MB/proc is a _range_. This range is > taken from me watching "top" on one of the compute nodes. The memory > usage of each process _oscillates_ between the two numbers listed > about every 5 seconds. Based on watching using "xosview" I believe > that the high numbers occur during communication steps, while the low > numbers are during "assembly" (residual computation). At this point, > these are just guesses, but I've been watching these things for a few > weeks now and kind of have a feel for what's going on. I believe mpip (http://mpip.sourceforge.net/) can report more detailed memory usage, but implementing the profiling does require additional work. I'd be interested to see how uniform the memory usage is across all the procs. We have some people around here who know about mpip so if you find some strange things happening I might be able to take a closer look. Also, it sounds like now might be a good time to start thinking about our StructuredMesh implementation again...unless your eventual problem will be unstructured and you don't care. We could probably run some truly enormous matrix-free problems on structured grids.... -- John ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
