Hi Barry and Matt > What is N? Is that the number of processes? Yes
> What does the notation 5'912'016 mean? It means there were 5658160 bytes allocated. I introduced the 's for readability. > Are the numbers in your table from a particular process? Or are they summed > over all processes? Only process 0. > The intention is that the ASM is memory scalable so that if for example > you double the number >of processes and double the total number of nonzeros in the matrix (probably by doubling the total >number of rows and columns in the matrix) each process should require essentially the same amount >of memory. But what happens in practice for a particular problem will, to some degree, depend on >the amount of coupling between processes in the matrix (hence how much bigger the local overlapped >matrix is then the original matrix on that process) and depend on how the domain is sliced up. >But even with a "bad" slicing I would not expect the amount of local memory needed to double. >I think you need to determine more completely what all this memory is being used for. Doubling the total number of rows and nonzeros is what I think I'm doing. Every row has about 40 nonzeros in this example. The coupling / slicing should be fine since I am using pretty much the same system for another calculation where I compute interior eigenstates in a matrix with the same sparsity. There I do not use ASM and I can scale up to 80000 cores without memory problems anymore (after I have done a workaround for not using AOCreateMapping, see my report earlier this week). Also when I turn off ASM and use no preconditioning at all, or when I use the Jacobi preconditioner, then memory stays constant at about 30MB/core. But then the convergence deteriorates... Matt: > We have run ASM on 224,000 processors of the XT5 at ORNL, so > something else is going on here. The best thing to do here is send us > -log_summary. For attachments, we usually recommend > petsc-maint at mcs.anl.gov. My data also comes from the XT5, but it's important for me to know that there are cases where it scales to 224000 processors. I will post more complete profiling information to petsc-maint at mcs.anl.gov in a couple of minutes. Best Sebastian > > Barry > > > > > On Mar 11, 2011, at 9:52 AM, Sebastian Steiger wrote: > >> Hello PETSc developers >> >> I'm doing some scaling benchmarks and I found that the parallel asm >> preconditioner, my favorite preconditioner, has a limit in the number of >> cores it can handle. >> >> I am doing a numerical experiment where I scale up the size of my matrix >> by roughly the same factor as the number of CPUs employed. When I look >> at which function used how much memory using PETSc's routine >> PetscMallocDumpLog, I see the following: >> >> Function name N=300 N=600 increase >> ====================================================================== >> MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 >> MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 >> MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 >> >> The matrix sizes are 6'899'904 and 14'224'896, respectively. Above >> N~5000 CPUs I am running out of memory. >> >> Here's my question now: Is the asm preconditioner limited from the >> algorithm point of view, or is it the implementation? I thought that >> 'only' the local matrices, plus some constant overlap with neighbors, >> are solved, so that memory consumption should stay constant when I scale >> up with a constant number of rows per process. >> >> Best >> Sebastian >> >
