Ok - I just ran this example with valgrind (and parmetis) on my laptop - and got no valgrind errors.
Will have a build tonight of this branch in next-tmp - and then look at merging it tomorrow. BTW: if the destination for this branch is maint - then its best to use the name 'hzhang/fix-submat_samerowdist/maint' to make it more obvious. Satish On Fri, 10 Nov 2017, Hong wrote: > Satish : > > > I don't see any changes hzhang/fix-submat_samerowdist in past few > > days. So how did this error get fixed? > > > > I've never seen valgrind error for ex56 from nightly tests. > Mark told me about Valgrind error on his mac, and I reproduced it on our > linux machine with my own configure, > which directs to lapack. > > > > > please do not merge yet. > > > Sure. > > Hong > > > > > On Fri, 10 Nov 2017, Mark Adams wrote: > > > > > I don't see a problem with this. As far as what branch to merge it into, > > > that is a Barry/Satish/etc decision. > > > > > > On Fri, Nov 10, 2017 at 5:17 PM, Hong <hzh...@mcs.anl.gov> wrote: > > > > > > > Mark: > > > > > > > >> I would add: > > > >> > > > >> '--download-fblaslapack=1', > > > >> > > > >> This is what I have on my Linux machine (cg at ANL) and it runs clean. > > > >> > > > > I do not see any error report from nightly tests for ex56. Should I > > merge > > > > this branch to maint? > > > > Hong > > > > > > > >> > > > >> On Fri, Nov 10, 2017 at 11:03 AM, Hong <hzh...@mcs.anl.gov> wrote: > > > >> > > > >>> I use > > > >>> Using configure Options: --download-metis --download-mpich > > > >>> --download-mumps --download-parmetis --download-scalapack > > > >>> --download-superlu --download-superlu_dist --download-suitesparse > > > >>> --download-hypre --download-ptscotch --download-chaco --with-ctable=1 > > > >>> --download-cmake --with-cc=gcc --with-cxx=g++ --with-debugging=1 > > > >>> --with-visibility=0 --with-fc=gfortran > > > >>> Hong > > > >>> > > > >>> On Fri, Nov 10, 2017 at 9:59 AM, Mark Adams <mfad...@lbl.gov> wrote: > > > >>> > > > >>>> This must be a configure issue. I don't see these warning: > > > >>>> > > > >>>> #!/usr/bin/python > > > >>>> if __name__ == '__main__': > > > >>>> import sys > > > >>>> import os > > > >>>> sys.path.insert(0, os.path.abspath('config')) > > > >>>> import configure > > > >>>> configure_options = [ > > > >>>> '--with-cc=clang', > > > >>>> '--with-cc++=clang++', > > > >>>> '--download-mpich=1', > > > >>>> '--download-metis=1', > > > >>>> '--download-superlu=1', > > > >>>> '--download-superlu_dist=1', > > > >>>> '--download-parmetis=1', > > > >>>> '--download-fblaslapack=1', > > > >>>> '--download-p4est=1', > > > >>>> '--with-debugging=1', > > > >>>> '--with-batch=0', > > > >>>> 'PETSC_ARCH=arch-linux2-c-dbg32', > > > >>>> '--with-openmp=0', > > > >>>> '--download-p4est=0' > > > >>>> ] > > > >>>> configure.petsc_configure(configure_options) > > > >>>> > > > >>>> ~ > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Fri, Nov 10, 2017 at 10:56 AM, Mark Adams <mfad...@lbl.gov> > > wrote: > > > >>>> > > > >>>>> This is comming from blas. How did you configure blas? > > > >>>>> > > > >>>>> On Fri, Nov 10, 2017 at 10:38 AM, Hong <hzh...@mcs.anl.gov> wrote: > > > >>>>> > > > >>>>>> Using petsc machine, I get > > > >>>>>> hzhang@petsc /sandbox/hzhang/petsc/src/snes/examples/tutorials > > > >>>>>> (hzhang/fix-submat_samerowdist) > > > >>>>>> $ mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3 > > > >>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg > > -ksp_rtol > > > >>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type > > gamg > > > >>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > > -pc_gamg_coarse_eq_limit 10 > > > >>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 > > > >>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 > > -snes_converged_reason > > > >>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 > > -mg_levels_ksp_type > > > >>>>>> chebyshev -mg_levels_esteig_ksp_type cg > > -mg_levels_esteig_ksp_max_it 10 > > > >>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type > > > >>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 > > -run_type 1 > > > >>>>>> > > > >>>>>> ==28811== Memcheck, a memory error detector > > > >>>>>> ==28811== Copyright (C) 2002-2013, and GNU GPL'd, by Julian > > Seward et > > > >>>>>> al. > > > >>>>>> ==28811== Using Valgrind-3.10.1 and LibVEX; rerun with -h for > > > >>>>>> copyright info > > > >>>>>> ==28811== Command: ./ex56 -cells 2,2,1 -max_conv_its 3 > > > >>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg > > -ksp_rtol > > > >>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type > > gamg > > > >>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > > -pc_gamg_coarse_eq_limit 10 > > > >>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 > > > >>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 > > -snes_converged_reason > > > >>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 > > -mg_levels_ksp_type > > > >>>>>> chebyshev -mg_levels_esteig_ksp_type cg > > -mg_levels_esteig_ksp_max_it 10 > > > >>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type > > > >>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3 > > -run_type 1 > > > >>>>>> ... > > > >>>>>> ==28811== Invalid read of size 16 > > > >>>>>> ==28811== at 0x8550946: dswap_k_NEHALEM (in > > > >>>>>> /usr/lib/openblas-base/libblas.so.3) > > > >>>>>> ==28811== by 0x7C6797F: dswap_ (in > > /usr/lib/openblas-base/libblas > > > >>>>>> .so.3) > > > >>>>>> ==28811== by 0x75B33B2: dgetri_ (in > > /usr/lib/lapack/liblapack.so.3 > > > >>>>>> .0) > > > >>>>>> ==28811== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012) > > > >>>>>> ==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274) > > > >>>>>> ==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749) > > > >>>>>> ==28811== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28811== Address 0xdc650d0 is 52,480 bytes inside a block of > > size > > > >>>>>> 52,488 alloc'd > > > >>>>>> ==28811== at 0x4C2D110: memalign (in > > /usr/lib/valgrind/vgpreload_me > > > >>>>>> mcheck-amd64-linux.so) > > > >>>>>> ==28811== by 0x51590F6: PetscMallocAlign (mal.c:39) > > > >>>>>> ==28811== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983) > > > >>>>>> ==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274) > > > >>>>>> ==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749) > > > >>>>>> ==28811== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28811== > > > >>>>>> ==28812== Invalid read of size 16 > > > >>>>>> ==28812== at 0x8550946: dswap_k_NEHALEM (in > > > >>>>>> /usr/lib/openblas-base/libblas.so.3) > > > >>>>>> ==28812== by 0x7C6797F: dswap_ (in > > /usr/lib/openblas-base/libblas > > > >>>>>> .so.3) > > > >>>>>> ==28812== by 0x75B33B2: dgetri_ (in > > /usr/lib/lapack/liblapack.so.3 > > > >>>>>> .0) > > > >>>>>> ==28812== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012) > > > >>>>>> ==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274) > > > >>>>>> ==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749) > > > >>>>>> ==28812== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28812== Address 0xd9c7600 is 52,480 bytes inside a block of > > size > > > >>>>>> 52,488 alloc'd > > > >>>>>> ==28812== at 0x4C2D110: memalign (in > > /usr/lib/valgrind/vgpreload_me > > > >>>>>> mcheck-amd64-linux.so) > > > >>>>>> ==28812== by 0x51590F6: PetscMallocAlign (mal.c:39) > > > >>>>>> ==28812== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983) > > > >>>>>> ==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274) > > > >>>>>> ==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749) > > > >>>>>> ==28812== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28812== > > > >>>>>> ==28811== Invalid read of size 16 > > > >>>>>> ==28811== at 0x8550A55: dswap_k_NEHALEM (in > > > >>>>>> /usr/lib/openblas-base/libblas.so.3) > > > >>>>>> ==28811== by 0x7C6797F: dswap_ (in > > /usr/lib/openblas-base/libblas > > > >>>>>> .so.3) > > > >>>>>> ==28811== by 0x7675179: dsteqr_ (in > > /usr/lib/lapack/liblapack.so.3 > > > >>>>>> .0) > > > >>>>>> ==28811== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508) > > > >>>>>> ==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582) > > > >>>>>> ==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763) > > > >>>>>> ==28811== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28811== Address 0xd99cbe0 is 64 bytes inside a block of size 72 > > > >>>>>> alloc'd > > > >>>>>> ==28811== at 0x4C2D110: memalign (in > > /usr/lib/valgrind/vgpreload_me > > > >>>>>> mcheck-amd64-linux.so) > > > >>>>>> ==28811== by 0x51590F6: PetscMallocAlign (mal.c:39) > > > >>>>>> ==28811== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504) > > > >>>>>> ==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582) > > > >>>>>> ==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763) > > > >>>>>> ==28811== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28811== > > > >>>>>> ==28812== Invalid read of size 16 > > > >>>>>> ==28812== at 0x8550A55: dswap_k_NEHALEM (in > > > >>>>>> /usr/lib/openblas-base/libblas.so.3) > > > >>>>>> ==28812== by 0x7C6797F: dswap_ (in > > /usr/lib/openblas-base/libblas > > > >>>>>> .so.3) > > > >>>>>> ==28812== by 0x7675179: dsteqr_ (in > > /usr/lib/lapack/liblapack.so.3 > > > >>>>>> .0) > > > >>>>>> ==28812== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508) > > > >>>>>> ==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582) > > > >>>>>> ==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763) > > > >>>>>> ==28812== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28812== Address 0xdc11f30 is 64 bytes inside a block of size 72 > > > >>>>>> alloc'd > > > >>>>>> ==28812== at 0x4C2D110: memalign (in > > /usr/lib/valgrind/vgpreload_me > > > >>>>>> mcheck-amd64-linux.so) > > > >>>>>> ==28812== by 0x51590F6: PetscMallocAlign (mal.c:39) > > > >>>>>> ==28812== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504) > > > >>>>>> ==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582) > > > >>>>>> ==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763) > > > >>>>>> ==28812== by 0x41056E: main (ex56.c:395) > > > >>>>>> ==28812== > > > >>>>>> [0] 27 global equations, 9 vertices > > > >>>>>> [0] 27 equations in vector, 9 vertices > > > >>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > > iterations 1 > > > >>>>>> [0] 441 global equations, 147 vertices > > > >>>>>> [0] 441 equations in vector, 147 vertices > > > >>>>>> > > > >>>>>> hangs here ... > > > >>>>>> > > > >>>>>> Hong > > > >>>>>> > > > >>>>>> On Fri, Nov 10, 2017 at 8:57 AM, Mark Adams <mfad...@lbl.gov> > > wrote: > > > >>>>>> > > > >>>>>>> This printed a little funny in gmail, snes/ex56 is running clean > > in > > > >>>>>>> the first few loops (appended), but the last one is the one with > > a reduced > > > >>>>>>> processor set. Still waiting. This is with 32 bit integers. I'm > > running > > > >>>>>>> another with 64 bit integers. > > > >>>>>>> > > > >>>>>>> ... > > > >>>>>>> [0] 27 global equations, 9 vertices > > > >>>>>>> [0] 27 equations in vector, 9 vertices > > > >>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > > iterations > > > >>>>>>> 1 > > > >>>>>>> [0] 441 global equations, 147 vertices > > > >>>>>>> [0] 441 equations in vector, 147 vertices > > > >>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > > iterations > > > >>>>>>> 1 > > > >>>>>>> [0] 4725 global equations, 1575 vertices > > > >>>>>>> [0] 4725 equations in vector, 1575 vertices > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On Fri, Nov 10, 2017 at 9:06 AM, Mark Adams <mfad...@lbl.gov> > > wrote: > > > >>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Thu, Nov 9, 2017 at 1:56 PM, Hong <hzh...@mcs.anl.gov> > > wrote: > > > >>>>>>>> > > > >>>>>>>>> Mark: > > > >>>>>>>>> > > > >>>>>>>>>> OK, well, just go with the Linux machine for the regression > > test. > > > >>>>>>>>>> I will keep trying to reproduce this on my Mac with an O > > build. > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> Valgrind error occurs on linux machines with g-build. I cannot > > > >>>>>>>>> merge this branch to maint until the bug is fixed. > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> Valgrind is failing on this run on my Mac. Moving to cg, like > > you I > > > >>>>>>>> suppose. This takes forever. This is what I have so far. Did > > you get this > > > >>>>>>>> far? > > > >>>>>>>> > > > >>>>>>>> 07:48 hzhang/fix-submat_samerowdist *= > > > >>>>>>>> /sandbox/adams/petsc/src/snes/examples/tutorials$ make > > > >>>>>>>> PETSC_DIR=/sandbox/adams/petsc PETSC_ARCH=arch-linux2-c-dbg32 > > val > > > >>>>>>>> /sandbox/adams/petsc/arch-linux2-c-dbg32/bin/mpiexec -n 2 > > valgrind > > > >>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 > > -snes_max_it 2 > > > >>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type > > > >>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type > > agg > > > >>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 > > > >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 > > > >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 > > -snes_converged_reason > > > >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 > > -mg_levels_ksp_type > > > >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg > > -mg_levels_esteig_ksp_max_it 10 > > > >>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 > > -mg_levels_pc_type > > > >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size > > 3 -run_type 1 > > > >>>>>>>> ==12414== Memcheck, a memory error detector > > > >>>>>>>> ==12414== Copyright (C) 2002-2013, and GNU GPL'd, by Julian > > Seward > > > >>>>>>>> et al. > > > >>>>>>>> ==12414== Using Valgrind-3.10.1 and LibVEX; rerun with -h for > > > >>>>>>>> copyright info > > > >>>>>>>> ==12415== Memcheck, a memory error detector > > > >>>>>>>> ==12415== Copyright (C) 2002-2013, and GNU GPL'd, by Julian > > Seward > > > >>>>>>>> et al. > > > >>>>>>>> ==12415== Using Valgrind-3.10.1 and LibVEX; rerun with -h for > > > >>>>>>>> copyright info > > > >>>>>>>> ==12415== Command: ./ex56 -cells 2,2,1 -max_conv_its 3 > > > >>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg > > -ksp_rtol > > > >>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 > > -pc_type gamg > > > >>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > > -pc_gamg_coarse_eq_limit 10 > > > >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 > > > >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 > > -snes_converged_reason > > > >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 > > -mg_levels_ksp_type > > > >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg > > -mg_levels_esteig_ksp_max_it 10 > > > >>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 > > -mg_levels_pc_type > > > >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size > > 3 -run_type 1 > > > >>>>>>>> ==12415== > > > >>>>>>>> ==12414== Command: ./ex56 -cells 2,2,1 -max_conv_its 3 > > > >>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg > > -ksp_rtol > > > >>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 > > -pc_type gamg > > > >>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > > -pc_gamg_coarse_eq_limit 10 > > > >>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 > > > >>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 > > -snes_converged_reason > > > >>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 > > -mg_levels_ksp_type > > > >>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg > > -mg_levels_esteig_ksp_max_it 10 > > > >>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 > > -mg_levels_pc_type > > > >>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size > > 3 -run_type 1 > > > >>>>>>>> ==12414== > > > >>>>>>>> [0] 27 global equations, 9 vertices > > > >>>>>>>> [0] 27 equations in vector, 9 vertices > > > >>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > > > >>>>>>>> iterations 1 > > > >>>>>>>> [0] 441 global equations, 147 vertices > > > >>>>>>>> [0] 441 equations in vector, 147 vertices > > > >>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > > > >>>>>>>> iterations 1 > > > >>>>>>>> [0] 4725 global equations, 1575 vertices > > > >>>>>>>> [0] 4725 equations in vector, 1575 vertices > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > >