On Tue, Oct 27, 2009 at 10:56 AM, francois pacull <fpacull at fluorem.com>wrote:
> Thank you Matt for your quick reply. > > I will try to use MUMPS soon. About UMFPACK, the OS used is a 64-bit Linux > one. Also, we do have 64-bit pointers from what I understand; the following > lines are from the PETSc configure.log file: > > #ifndef PETSC_SIZEOF_VOID_P > #define PETSC_SIZEOF_VOID_P 8 > #endif > Yes, then it appears that UMFPACK cannot be used for large memory. Did you try MUMPS? Matt > And here is the umfpack_UMF_report_info from the subdomain that crashed: > > > UMFPACK V5.4.0 (May 20, 2009), Info: > matrix entry defined as: double > Int (generic integer) defined as: int > BLAS library used: Fortran BLAS. size of BLAS integer: 4 > MATLAB: no. > CPU timer: POSIX times ( ) routine. > number of rows in matrix A: 79002 > number of columns in matrix A: 79002 > entries in matrix A: 12030970 > memory usage reported in: 8-byte Units > size of int: 4 bytes > size of UF_long: 8 bytes > size of pointer: 8 bytes > size of numerical entry: 8 bytes > > strategy used: unsymmetric > ordering used: colamd on A > modify Q during factorization: yes > prefer diagonal pivoting: no > pivots with zero Markowitz cost: 0 > submatrix S after removing zero-cost pivots: > number of "dense" rows: 0 > number of "dense" columns: 0 > number of empty rows: 0 > number of empty columns 0 > submatrix S square and diagonal preserved > symbolic factorization defragmentations: 0 > symbolic memory usage (Units): 27435792 > symbolic memory usage (MBytes): 209.3 > Symbolic size (Units): 177636 > Symbolic size (MBytes): 1 > symbolic factorization CPU time (sec): 4.95 > symbolic factorization wallclock time(sec): 4.95 > > symbolic/numeric factorization: upper bound actual > % > variable-sized part of Numeric object: > initial size (Units) 31236744 - > - > peak size (Units) 597607658 - > - > final size (Units) 550474688 - > - > Numeric final size (Units) 550988250 - > - > Numeric final size (MBytes) 4203.7 - > - > peak memory usage (Units) 598718594 - > - > peak memory usage (MBytes) 4567.9 - > - > numeric factorization flops 1.63141e+12 - > - > nz in L (incl diagonal) 171352664 - > - > nz in U (incl diagonal) 346187947 - > - > nz in L+U (incl diagonal) 517461609 - > - > largest front (# entries) 15783705 - > - > largest # rows in front 2815 - > - > largest # columns in front 5607 - > - > > > UMFPACK V5.4.0 (May 20, 2009): ERROR: out of memory > > Thanks again, > francois. > > > > Matthew Knepley a ?crit : > > On Tue, Oct 27, 2009 at 10:12 AM, francois pacull <fpacull at > fluorem.com<mailto: >> fpacull at fluorem.com>> wrote: >> >> Dear PETSc team, >> >> I have a few questions... During the resolution of a linear system >> in parallel, I am trying to apply a local LU solver to each of the >> SEQaij diagonal blocks of a MPIaij matrix partitioned with PARMETIS. >> >> - I started with MUMPS but it seems that it only works with >> unpartitioned aij matrices, is it really the case? Or could we use >> MUMPS to build an additive Schwarz preconditioner for example? >> >> >> You can use MUMPS for the subproblem solver. >> >> - Then I tried UMFPACK. This works fine when the diagonal blocks >> (and the memory required to store the factors) are small but >> crashes when they are a little bit larger. For example with a >> "numeric final size" of 4203.7 MBytes, I got the following >> message "ERROR: out of memory" while there was plenty of memory >> left in the computer. I tried either with the UMFPACK version 5.2, >> downloaded by PETSc, or with a manually installed version 5.4, >> linked to PETSc. Is this a behavior from UMFPACK that you already >> experienced? >> >> >> Send all the error output. However, in oder to address more than 4G, you >> will need 64-bit pointers. >> >> - Since UMFPACK seemed to have a memory limit around 4096 MB, I >> tried to install a PETSc version with the option >> "--with-64-bit-indices", however none of the partitioning packages >> could be compiled with this option >> (parmetis,chaco,jostle,party,scotch). Is there a way to compile >> PETSc with 64 bit indices AND a partitioning package? >> >> >> Not that I know of. >> >> - Finally, I tried to modify the PETSc source code umfpack.c so >> that it would deal with 64 bit indices, but I only ended up so far >> with a segmentation violation message at the execution... Is it >> the only way I could use UMPACK with large sparse matrices? >> >> >> Why not just upgrade to a 64-bit OS if you want to address so much memory >> on a single machine? >> >> Matt >> >> Thank you, >> Regards, >> francois pacull. >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20091027/d9a011e3/attachment.htm>
