On Tue, Jan 27, 2026 at 11:40 PM neil liu <[email protected]> wrote:
> Thanks a lot, Matt.
>
> I assemble a matrix using MatSetValuesLocal(), where the cell-to-local
> mapping comes from PetscSectionGetOffset().
>
> With 2 MPI ranks, Valgrind (rank 0) reports uninitialized data being
> passed to MPI_Isend during matrix assembly (inside matstash.c).
>
> Two observations:
>
> -
>
> If I hard-code the local mapping to always be 1, the Valgrind warning
> disappears.
> -
>
> The warning occurs with PETSc *3.23.3*, but disappears when switching
> to *3.24.3* under the same setup.
>
> MPICH plays games with memory for efficiency reasons. We often see
Valgrind warnings. Could you check if the default MPICH version changed
between 3.23.3 and 3.24.3. That would be my first guess.
Thanks,
Matt
> This suggests either uninitialized local indices/values on my side, or a
> change in PETSc between these versions affecting stash packing or index
> handling.
>
> Does this type of Valgrind warning usually point to user-provided local
> indices being uninitialized, and were there relevant changes in this area
> between 3.23.3 and 3.24.x?
>
> Thanks,
> Xiaodong
>
> ==683913== Syscall param write(buf) points to uninitialised byte(s)
> ==683913== at 0xCAD15A8: write (in /usr/lib64/libc-2.28.so)
> ==683913== by 0xBB9B0A9: MPIDI_CH3I_Sock_write (sock.c:2622)
> ==683913== by 0xBBA12DF: MPIDI_CH3_iStartMsg (ch3_istartmsg.c:68)
> ==683913== by 0xBB55F82: MPIDI_CH3_RndvSend (ch3u_rndv.c:48)
> ==683913== by 0xBB70905: MPID_Isend (mpid_isend.c:159)
> ==683913== by 0xB83EDB7: internal_Isend_c (isend.c:273)
> ==683913== by 0xB83EFE9: PMPI_Isend_c (isend.c:363)
> ==683913== by 0x82CAC1A: MatStashBTSSend_Private (matstash.c:793)
> ==683913== by 0x6AA2BF8: PetscCommBuildTwoSidedFReq_Reference
> (mpits.c:311)
> ==683913== by 0x6AA9DAD: PetscCommBuildTwoSidedFReq (mpits.c:515)
> ==683913== by 0x82CEB46: MatStashScatterBegin_BTS (matstash.c:920)
> ==683913== by 0x82C009D: MatStashScatterBegin_Private (matstash.c:440)
> ==683913== by 0x73E77BF: MatAssemblyBegin_MPIAIJ (mpiaij.c:768)
> ==683913== by 0x81F0777: MatAssemblyBegin (matrix.c:5807)
>
> On Tue, Jan 27, 2026 at 6:13 PM Matthew Knepley <[email protected]> wrote:
>
>> On Tue, Jan 27, 2026 at 3:34 PM neil liu <[email protected]> wrote:
>>
>>> Dear Petsc users and developers,
>>>
>>> I am exploring the mapping between local, partition and global dofs.
>>> The following is my pseudo code, dof2Partitionmapping denotes the
>>> mapping between the local dof (20 local dofs each tetrahedra) and partition
>>> dof.
>>>
>>
>> We usually say cell, local, and global dofs.
>>
>>
>>> Is this mapping determined by Petsc itself under the hood
>>> (PetscSectionGetOffset)?
>>>
>>
>> Plex just iterates over the points in the canonical numbering (cells,
>> vertices, faces, edges). You can change the iteration order using
>>
>>
>> https://urldefense.us/v3/__https://petsc.org/main/manualpages/PetscSection/PetscSectionSetPermutation/__;!!G_uCfscf7eWS!axRnfOFmHcGL3iKKS0tSlsif0QKDXBdDwXWrPhWj9HPh6Td-Xx10T5eHPX4YORD_ITemssQl-KasmwS3V3kG$
>>
>>
>> You can use that, for example, to place all ghost dofs at the end.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> For now, I am coding this mapping (local to partition) myself just based
>>> on the edge and face number in the partition. It seems the results are
>>> reasonable. But with this kind of self-defined mapping, the owned dofs and
>>> ghost dofs are interleaved. Will this bring bad results for the
>>> communication of MatStash?
>>>
>>> Thanks,
>>> Xiaodong
>>>
>>> *1. set 2 dofs for each edge and 2 dofs for each edge face respectively*
>>> PetscSectionSetChart(s, faceStart, edgeEnd);
>>> PetscSectionSetDof(s, faceIndex, 2);
>>> PetscSectionSetFieldDof(s, faceIndex, 0, 1);
>>> PetscSectionSetDof(s, edgeIndex, 2);
>>> PetscSectionSetFieldDof(s, edgeIndex, 0, 1);
>>> PetscSectionsetup(s)
>>>
>>> *2. Create matrix based on DMPlex*
>>> DMSetMatType(dm, MATAIJ);
>>> DMCreateMatrix(dm, &A);
>>>
>>> *3. loop over elements to set local values for Matrix*
>>> MatSetValuesLocal(A, dof2Partitionmapping.size(),
>>> dof2Partitionmapping.data(), dof2Partitionmapping.size(),
>>> dof2Partitionmapping.data(), Matrix_Local.data(), ADD_VALUES);
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!axRnfOFmHcGL3iKKS0tSlsif0QKDXBdDwXWrPhWj9HPh6Td-Xx10T5eHPX4YORD_ITemssQl-KasmzDTmp34$
>>
>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!axRnfOFmHcGL3iKKS0tSlsif0QKDXBdDwXWrPhWj9HPh6Td-Xx10T5eHPX4YORD_ITemssQl-Kasm4WP45iV$
>> >
>>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!axRnfOFmHcGL3iKKS0tSlsif0QKDXBdDwXWrPhWj9HPh6Td-Xx10T5eHPX4YORD_ITemssQl-KasmzDTmp34$
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!axRnfOFmHcGL3iKKS0tSlsif0QKDXBdDwXWrPhWj9HPh6Td-Xx10T5eHPX4YORD_ITemssQl-Kasm4WP45iV$
>
