Re: [petsc-users] [EXTERNAL] Re: Call to DMSetMatrixPreallocateSkip not changing allocation behavior

[Jed Brown]

Great, thanks for letting us know. It'll merge to release shortly and thus be 
in petsc >= 3.20.3.

"Fackler, Philip"  writes:

> Jed,
>
> That seems to have worked (ridiculously well). It's now 55MB, and it's 
> happening in the call to MatSetPreallocationCOO.
>
> Thank you,
>
> Philip Fackler
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> Oak Ridge National Laboratory
> 
> From: Jed Brown 
> Sent: Thursday, December 14, 2023 16:27
> To: Fackler, Philip ; petsc-users@mcs.anl.gov 
> ; xolotl-psi-developm...@lists.sourceforge.net 
> 
> Subject: [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not 
> changing allocation behavior
>
> I had a one-character typo in the diff above. This MR to release should work 
> now.
>
> https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120=DwIBAg=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074=
>
> Jed Brown  writes:
>
>> 17 GB for a 1D DMDA, wow. :-)
>>
>> Could you try applying this diff to make it work for DMDA (it's currently 
>> handled by DMPlex)?
>>
>> diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c
>> index cad4d926504..bd2a3bda635 100644
>> --- i/src/dm/impls/da/fdda.c
>> +++ w/src/dm/impls/da/fdda.c
>> @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J)
>> specialized setting routines depend only on the particular preallocation
>> details of the matrix, not the type itself.
>>*/
>> -  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPIAIJSetPreallocation_C", ));
>> -  if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqAIJSetPreallocation_C", ));
>> -  if (!aij) {
>> -PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPIBAIJSetPreallocation_C", ));
>> -if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqBAIJSetPreallocation_C", ));
>> -if (!baij) {
>> -  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPISBAIJSetPreallocation_C", ));
>> -  if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqSBAIJSetPreallocation_C", ));
>> -  if (!sbaij) {
>> -PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPISELLSetPreallocation_C", ));
>> -if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqSELLSetPreallocation_C", ));
>> +  if (!dm->prealloc_skip) { // Flag is likely set when user intends to use 
>> MatSetPreallocationCOO()
>> +PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPIAIJSetPreallocation_C", ));
>> +if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqAIJSetPreallocation_C", ));
>> +if (!aij) {
>> +  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPIBAIJSetPreallocation_C", ));
>> +  if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqBAIJSetPreallocation_C", ));
>> +  if (!baij) {
>> +PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPISBAIJSetPreallocation_C", ));
>> +if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqSBAIJSetPreallocation_C", ));
>> +if (!sbaij) {
>> +  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatMPISELLSetPreallocation_C", ));
>> +  if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatSeqSELLSetPreallocation_C", ));
>> +}
>> +if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatISSetPreallocation_C", ));
>>}
>> -  if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
>> "MatISSetPreallocation_C", ));
>>  }
>>}
>>if (aij) {
>>
>>
>> "Fackler, Philip via petsc-users"  writes:
>>
>>> I'm using the following sequence of functions related to the Jacobian 
>>> matrix:
>>>
>>> DMDACreate1d(..., );
>>> DMSetFromOptions(da);
>>> DMSetUp(da);
>>> DMSetMatType(da, MATAIJKOKKOS);
>>> DMSetMatrixPreallocateSkip(da, P

Re: [petsc-users] [EXTERNAL] Re: Call to DMSetMatrixPreallocateSkip not changing allocation behavior

[Fackler, Philip via petsc-users]

Jed,

That seems to have worked (ridiculously well). It's now 55MB, and it's 
happening in the call to MatSetPreallocationCOO.

Thank you,

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory

From: Jed Brown 
Sent: Thursday, December 14, 2023 16:27
To: Fackler, Philip ; petsc-users@mcs.anl.gov 
; xolotl-psi-developm...@lists.sourceforge.net 

Subject: [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not 
changing allocation behavior

I had a one-character typo in the diff above. This MR to release should work 
now.

https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120=DwIBAg=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074=

Jed Brown  writes:

> 17 GB for a 1D DMDA, wow. :-)
>
> Could you try applying this diff to make it work for DMDA (it's currently 
> handled by DMPlex)?
>
> diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c
> index cad4d926504..bd2a3bda635 100644
> --- i/src/dm/impls/da/fdda.c
> +++ w/src/dm/impls/da/fdda.c
> @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J)
> specialized setting routines depend only on the particular preallocation
> details of the matrix, not the type itself.
>*/
> -  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPIAIJSetPreallocation_C", ));
> -  if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqAIJSetPreallocation_C", ));
> -  if (!aij) {
> -PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPIBAIJSetPreallocation_C", ));
> -if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqBAIJSetPreallocation_C", ));
> -if (!baij) {
> -  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPISBAIJSetPreallocation_C", ));
> -  if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqSBAIJSetPreallocation_C", ));
> -  if (!sbaij) {
> -PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPISELLSetPreallocation_C", ));
> -if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqSELLSetPreallocation_C", ));
> +  if (!dm->prealloc_skip) { // Flag is likely set when user intends to use 
> MatSetPreallocationCOO()
> +PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPIAIJSetPreallocation_C", ));
> +if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqAIJSetPreallocation_C", ));
> +if (!aij) {
> +  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPIBAIJSetPreallocation_C", ));
> +  if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqBAIJSetPreallocation_C", ));
> +  if (!baij) {
> +PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPISBAIJSetPreallocation_C", ));
> +if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqSBAIJSetPreallocation_C", ));
> +if (!sbaij) {
> +  PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatMPISELLSetPreallocation_C", ));
> +  if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatSeqSELLSetPreallocation_C", ));
> +}
> +if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatISSetPreallocation_C", ));
>}
> -  if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, 
> "MatISSetPreallocation_C", ));
>  }
>}
>if (aij) {
>
>
> "Fackler, Philip via petsc-users"  writes:
>
>> I'm using the following sequence of functions related to the Jacobian matrix:
>>
>> DMDACreate1d(..., );
>> DMSetFromOptions(da);
>> DMSetUp(da);
>> DMSetMatType(da, MATAIJKOKKOS);
>> DMSetMatrixPreallocateSkip(da, PETSC_TRUE);
>> Mat J;
>> DMCreateMatrix(da, );
>> MatSetPreallocationCOO(J, ...);
>>
>> I recently added the call to DMSetMatrixPreallocateSkip, hoping the 
>> allocation would be delayed to MatSetPreallocationCOO, and that it would 
>> require less memory. The 
>> documentation<https://urldefense.us/v2/url?u=https-3A__petsc.org_release_manualpages_DM_DMSetMatrixPreallocateSkip_=DwIBAg=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO