> > Perhaps you are calling ISColoringGetIS() and not calling > ISColoringRestoreIS()? >
I have matching ISColoringGet/Restore here, and it's only used prior to the first iteration so at least it doesn't seem to be growing. At the bottom I pasted the malloc_view and malloc_debug output from running 1 time step. I'm sort of thinking this might be a red herring -- is it possible the rank 0 process is chewing up dramatically more memory than others, like with logging or something? Like I mentioned earlier the total memory usage is well under the machine limits. I'll spring in some PetscMemoryGetMaximumUsage logging at every time step and try to get a big job going again. Are you using Fortran? > C++ [ 0]1408 bytes PetscSplitReductionCreate() line 63 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c [ 0]80 bytes PetscSplitReductionCreate() line 57 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c [ 0]16 bytes PetscCommBuildTwoSided_Allreduce() line 169 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/mpits.c [ 0]16 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]272 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]880 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]960 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]976 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]1024 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]1024 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]1040 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c [ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c [ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c [ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c [ 0]64 bytes ISColoringGetIS() line 266 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/utils/iscoloring.c [ 0]32 bytes PetscCommDuplicate() line 129 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/objects/tagm.c [0] Maximum memory PetscMalloc()ed 610153776 maximum size of entire process 719073280 [0] Memory usage sorted by function [0] 6 192 DMCoarsenHookAdd() [0] 2 9984 DMCreate() [0] 2 128 DMCreate_Shell() [0] 2 64 DMDSEnlarge_Static() [0] 1 672 DMKSPCreate() [0] 3 96 DMRefineHookAdd() [0] 3 2064 DMSNESCreate() [0] 4 128 DMSubDomainHookAdd() [0] 1 768 DMTSCreate() [0] 2 96 ISColoringCreate() [0] 8 12608 ISColoringGetIS() [0] 1 307200 ISConcatenate() [0] 29 25984 ISCreate() [0] 25 400 ISCreate_General() [0] 4 64 ISCreate_Stride() [0] 20 338016 ISGeneralSetIndices_General() [0] 3 921600 ISGetIndices_Stride() [0] 2 307232 ISGlobalToLocalMappingSetUp_Basic() [0] 1 6144 ISInvertPermutation_General() [0] 3 308576 ISLocalToGlobalMappingCreate() [0] 2 32 KSPConvergedDefaultCreate() [0] 2 2816 KSPCreate() [0] 1 224 KSPCreate_FGMRES() [0] 1 8016 KSPGMRESClassicalGramSchmidtOrthogonalization() [0] 2 16032 KSPSetUp_FGMRES() [0] 4 16084160 KSPSetUp_GMRES() [0] 2 36864 MatColoringApply_SL() [0] 1 656 MatColoringCreate() [0] 6 17088 MatCreate() [0] 1 16 MatCreateMFFD_WP() [0] 1 16 MatCreateSubMatrices_SeqBAIJ() [0] 1 12288 MatCreateSubMatrix_SeqBAIJ() [0] 3 32320 MatCreateSubMatrix_SeqBAIJ_Private() [0] 2 1472 MatCreate_MFFD() [0] 1 416 MatCreate_SeqAIJ() [0] 3 864 MatCreate_SeqBAIJ() [0] 2 416 MatCreate_Shell() [0] 1 784 MatFDColoringCreate() [0] 2 12288 MatFDColoringDegreeSequence_Minpack() [0] 6 30859392 MatFDColoringSetUp_SeqXAIJ() [0] 3 42512 MatGetColumnIJ_SeqAIJ() [0] 4 72720 MatGetColumnIJ_SeqBAIJ_Color() [0] 1 6144 MatGetOrdering_Natural() [0] 2 36384 MatGetRowIJ_SeqAIJ() [0] 7 210626000 MatILUFactorSymbolic_SeqBAIJ() [0] 2 313376 MatIncreaseOverlap_SeqBAIJ() [0] 2 30740608 MatLUFactorNumeric_SeqBAIJ_N() [0] 1 6144 MatMarkDiagonal_SeqAIJ() [0] 1 6144 MatMarkDiagonal_SeqBAIJ() [0] 8 256 MatRegisterRootName() [0] 1 6160 MatSeqAIJCheckInode() [0] 4 115216 MatSeqAIJSetPreallocation_SeqAIJ() [0] 4 302779424 MatSeqBAIJSetPreallocation_SeqBAIJ() [0] 13 576 MatSolverTypeRegister() [0] 1 16 PCASMCreateSubdomains() [0] 2 1664 PCCreate() [0] 1 160 PCCreate_ASM() [0] 1 192 PCCreate_ILU() [0] 5 307264 PCSetUp_ASM() [0] 2 416 PetscBTCreate() [0] 2 3216 PetscClassPerfLogCreate() [0] 2 1616 PetscClassRegLogCreate() [0] 2 32 PetscCommBuildTwoSided_Allreduce() [0] 2 64 PetscCommDuplicate() [0] 2 1888 PetscDSCreate() [0] 2 26416 PetscEventPerfLogCreate() [0] 2 158400 PetscEventPerfLogEnsureSize() [0] 2 1616 PetscEventRegLogCreate() [0] 2 9600 PetscEventRegLogRegister() [0] 8 102400 PetscFreeSpaceGet() [0] 474 15168 PetscFunctionListAdd_Private() [0] 2 528 PetscIntStackCreate() [0] 142 11360 PetscLayoutCreate() [0] 56 896 PetscLayoutSetUp() [0] 59 9440 PetscObjectComposedDataIncreaseReal() [0] 2 576 PetscObjectListAdd() [0] 33 768 PetscOptionsGetEList() [0] 1 16 PetscOptionsHelpPrintedCreate() [0] 1 32 PetscPushSignalHandler() [0] 7 6944 PetscSFCreate() [0] 3 432 PetscSFCreate_Basic() [0] 2 1472 PetscSFLinkCreate() [0] 11 1229040 PetscSFSetUpRanks() [0] 7 614512 PetscSFSetUp_Basic() [0] 4 20096 PetscSegBufferCreate() [0] 2 1488 PetscSplitReductionCreate() [0] 2 3008 PetscStageLogCreate() [0] 1148 23872 PetscStrallocpy() [0] 6 13056 PetscStrreplace() [0] 9 3456 PetscTableCreate() [0] 1 16 PetscViewerASCIIOpen() [0] 6 96 PetscViewerAndFormatCreate() [0] 1 752 PetscViewerCreate() [0] 1 96 PetscViewerCreate_ASCII() [0] 2 1424 SNESCreate() [0] 1 16 SNESCreate_NEWTONLS() [0] 1 1008 SNESLineSearchCreate() [0] 1 16 SNESLineSearchCreate_BT() [0] 16 1824 SNESMSRegister() [0] 46 9056 TSARKIMEXRegister() [0] 1 1264 TSAdaptCreate() [0] 8 384 TSBasicSymplecticRegister() [0] 1 2160 TSCreate() [0] 1 224 TSCreate_Theta() [0] 48 5968 TSGLEERegister() [0] 41 7728 TSRKRegister() [0] 89 14736 TSRosWRegister() [0] 71 110192 VecCreate() [0] 1 307200 VecCreateGhostWithArray() [0] 123 36874080 VecCreate_MPI_Private() [0] 7 4300800 VecCreate_Seq() [0] 8 256 VecCreate_Seq_Private() [0] 6 400 VecDuplicateVecs_Default() [0] 3 2352 VecScatterCreate() [0] 7 1843296 VecScatterSetUp_SF() [0] 126 2016 VecStashCreate_Private() [0] 1 3072 mapBlockColoringToJacobian() On Wed, Aug 12, 2020 at 4:22 PM Barry Smith <[email protected]> wrote: > > Yes, there are some PETSc objects or arrays that you are not freeing so > they are printed at the end of the run. For small runs this harmless but if > new objects/memory is allocated at each iteration and not suitably freed it > will eventually add up. > > Run with -malloc_view (small problem with say 2 iterations) it will > print everything allocated and might be helpful. > > Perhaps you are calling ISColoringGetIS() and not calling > ISColoringRestoreIS()? > > It is also possible it is a leak in PETSc, but that is unlikely since > we test for them. > > Are you using Fortran? > > Barry > > > On Aug 12, 2020, at 1:29 PM, Mark Lohry <[email protected]> wrote: > > Thanks Matt and Barry. At Matt's suggestion I ran a smaller representative > case with valgrind and didn't see anything alarming (apart from a small > leak in an older boost version I was using: > https://github.com/boostorg/serialization/issues/104 although I don't > think this was causing the issue). > > -malloc_debug dumps quite a lot, this is supposed to be empty right? > Output pasted below. It looks like the same sequence of calls is repeated 8 > times, which is how many nonlinear solves occurred in this particular run. > Thoughts? > > > > [ 0]1408 bytes PetscSplitReductionCreate() line 63 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c > [ 0]80 bytes PetscSplitReductionCreate() line 57 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c > [ 0]16 bytes PetscCommBuildTwoSided_Allreduce() line 169 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/mpits.c > [ 0]16 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]272 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]880 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]960 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]976 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]1024 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]1024 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]1040 bytes ISGeneralSetIndices_General() line 578 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]16 bytes PetscLayoutSetUp() line 269 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]80 bytes PetscLayoutCreate() line 55 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c > [ 0]16 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]32 bytes PetscStrallocpy() line 187 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c > [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c > [ 0]16 bytes ISCreate_General() line 647 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c > [ 0]896 bytes ISCreate() line 37 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c > [ 0]64 bytes ISColoringGetIS() line 266 in > /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/utils/iscoloring.c > [ 0]32 bytes PetscCommDuplicate() line 129 in > /home/mlohry/dev/cmake-build/external/petsc/src/sys/objects/tagm.c > > > > On Wed, Aug 12, 2020 at 1:46 PM Barry Smith <[email protected]> wrote: > >> >> Mark. >> >> When valgrind is not feasible (like on many centrally controlled >> batch systems) you can run PETSc with an extra flag to do some memory error >> checks >> -malloc_debug >> >> this >> >> 1) fills all malloced memory with Nan so if the code is using >> uninitialized memory it may be detected and >> 2) checks the beginning and end of each alloced memory region for >> out-of-bounds writes at each malloc and free. >> >> it will slow the code down a little bit but generally not a huge amount. >> >> It is no where near as good as valgrind or other memory corruption tools >> but it has the advantage you can run it anywhere on any size job. >> >> >> Barry >> >> >> >> >> >> On Aug 12, 2020, at 7:46 AM, Matthew Knepley <[email protected]> wrote: >> >> On Wed, Aug 12, 2020 at 7:53 AM Mark Lohry <[email protected]> wrote: >> >>> I'm getting seemingly random failures of late: >>> Caught signal number 7 BUS: Bus Error, possibly illegal memory access >>> >> >> The first thing I would do is run valgrind on as wide an array of tests >> as you can. This will find problems >> on things that run completely fine. >> >> Thanks, >> >> Matt >> >> >>> Symptoms: >>> 1) Seems to only happen (so far) on larger cases, 400-2000 cores >>> 2) It doesn't happen right away -- this was running happily for several >>> hours over several hundred time steps with no indication of bad health in >>> the numerics >>> 3) At least the total memory consumption seems to be within bounds, >>> though I'm not sure about individual processes. e.g. slurm here reported >>> Memory Efficiency: 75.23% of 1.76 TB (180.00 GB/node) >>> 4) running the same setup twice it fails at different points >>> >>> Any suggestions on what to look for? This is a bit painful to work on as >>> I can only reproduce it on large runs and then it's seemingly random. >>> >>> >>> Thanks, >>> Mark >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >> >> >
