Jed, That seems to have worked (ridiculously well). It's now 55MB, and it's happening in the call to MatSetPreallocationCOO.
Thank you, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Jed Brown <[email protected]> Sent: Thursday, December 14, 2023 16:27 To: Fackler, Philip <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Subject: [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior I had a one-character typo in the diff above. This MR to release should work now. https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074&e= Jed Brown <[email protected]> writes: > 17 GB for a 1D DMDA, wow. :-) > > Could you try applying this diff to make it work for DMDA (it's currently > handled by DMPlex)? > > diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c > index cad4d926504..bd2a3bda635 100644 > --- i/src/dm/impls/da/fdda.c > +++ w/src/dm/impls/da/fdda.c > @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J) > specialized setting routines depend only on the particular preallocation > details of the matrix, not the type itself. > */ > - PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPIAIJSetPreallocation_C", &aij)); > - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqAIJSetPreallocation_C", &aij)); > - if (!aij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPIBAIJSetPreallocation_C", &baij)); > - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqBAIJSetPreallocation_C", &baij)); > - if (!baij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPISBAIJSetPreallocation_C", &sbaij)); > - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqSBAIJSetPreallocation_C", &sbaij)); > - if (!sbaij) { > - PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPISELLSetPreallocation_C", &sell)); > - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqSELLSetPreallocation_C", &sell)); > + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use > MatSetPreallocationCOO() > + PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPIAIJSetPreallocation_C", &aij)); > + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqAIJSetPreallocation_C", &aij)); > + if (!aij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPIBAIJSetPreallocation_C", &baij)); > + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqBAIJSetPreallocation_C", &baij)); > + if (!baij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPISBAIJSetPreallocation_C", &sbaij)); > + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqSBAIJSetPreallocation_C", &sbaij)); > + if (!sbaij) { > + PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatMPISELLSetPreallocation_C", &sell)); > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqSELLSetPreallocation_C", &sell)); > + } > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatISSetPreallocation_C", &is)); > } > - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatISSetPreallocation_C", &is)); > } > } > if (aij) { > > > "Fackler, Philip via petsc-users" <[email protected]> writes: > >> I'm using the following sequence of functions related to the Jacobian matrix: >> >> DMDACreate1d(..., &da); >> DMSetFromOptions(da); >> DMSetUp(da); >> DMSetMatType(da, MATAIJKOKKOS); >> DMSetMatrixPreallocateSkip(da, PETSC_TRUE); >> Mat J; >> DMCreateMatrix(da, &J); >> MatSetPreallocationCOO(J, ...); >> >> I recently added the call to DMSetMatrixPreallocateSkip, hoping the >> allocation would be delayed to MatSetPreallocationCOO, and that it would >> require less memory. The >> documentation<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_manualpages_DM_DMSetMatrixPreallocateSkip_&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=IMLBs0ydxDPvuXeD6jmsq1BN_8oieHVSyG6VA9c0DyM&e= >> > says that the data structures will not be preallocated. The following >> data from heaptrack shows that the allocation is still happening in the call >> to DMCreateMatrix. >> >> [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13] >> >> Can someone help me understand this? >> >> Thanks, >> >> Philip Fackler >> Research Software Engineer, Application Engineering Group >> Advanced Computing Systems Research Section >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory
