On Mon, Nov 21, 2011 at 22:47, Andrej Mesaros <andrej.mesaros at bc.edu> wrote:
> Dear all, > > I need guidance in finding the memory needed for matrix assembly. > > The job that fails when I reserve 3.5GB memory per node gives me the error > output below. The job was run on 96 nodes, each storing its own part of a > matrix (around 60k rows each, ~100M non-zero complex entries). > > The error occurs during assembly (similar numbers for every node): > > [25]PETSC ERROR: Out of memory. This could be due to allocating > [25]PETSC ERROR: too large an object or bleeding by not properly > [25]PETSC ERROR: destroying unneeded objects. > [25]PETSC ERROR: Memory allocated 4565256864 Memory used by process > 3658739712 > [25]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [25]PETSC ERROR: Memory requested 980025524! > [25]PETSC ERROR: > ------------------------------**------------------------------** > ------------ > [25]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [25]PETSC ERROR: See docs/changes/index.html for recent updates. > [25]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [25]PETSC ERROR: See docs/index.html for manual pages. > [25]PETSC ERROR: > ------------------------------**------------------------------** > ------------ > [25]PETSC ERROR: Unknown Name on a linux-gnu named compute-5-54.local by > mesaros Wed Oct 12 22:27:13 2011 > [25]PETSC ERROR: Libraries linked from > /home/mesaros/code/petsc-3.1-**p8/linux-gnu-mpi-debug-**complex/lib > [25]PETSC ERROR: Configure run at Thu Jun 30 12:30:13 2011 > [25]PETSC ERROR: Configure options --with-scalar-type=complex > --with-64-bit-indices=1 --download-f-blas-lapack=yes --download-mpich=1 > --with-mpi-exec=/usr/publ$ > [25]PETSC ERROR: > ------------------------------**------------------------------** > ------------ > [25]PETSC ERROR: PetscMallocAlign() line 49 in src/sys/memory/mal.c > [25]PETSC ERROR: PetscTrMallocDefault() line 192 in src/sys/memory/mtr.c > [25]PETSC ERROR: PetscPostIrecvInt() line 250 in src/sys/utils/mpimesg.c > Looks like you are trying to half a billion (--with-64-bit-indices) or a billion entries. How are you computing the nonzeros? Is it possible that many processes are computing entries that need to go to one process? > [25]PETSC ERROR: MatStashScatterBegin_Private() line 498 in > src/mat/utils/matstash.c > [25]PETSC ERROR: MatAssemblyBegin_MPIAIJ() line 474 in > src/mat/impls/aij/mpi/mpiaij.c > [25]PETSC ERROR: MatAssemblyBegin() line 4564 in src/mat/interface/matrix.c > > > Now, how much memory would I need per node for this assembly to work? Is > it "Memory allocated" + "Memory requested", which is around 5.5GB? And did > it fail when "Memory used by process" reached ~3.5GB, which was the limit > for the job? Usually, breaking the limit on memory per node kills the job, > and PETSc then doesn't give the above "Out of memory" output. > > Additionally, can I simply estimate the additional memory needed for SLEPc > to find ~100 lowest eigenvalues? > Start with what is typically needed by PETSc (the matrix, the setup cost is for your preconditioner, the vectors for the Krylov method) and add 100*n*sizeof(PetscScalar). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111121/d6de761b/attachment.htm>
