I don't see a problem with this. As far as what branch to merge it into,
that is a Barry/Satish/etc decision.
On Fri, Nov 10, 2017 at 5:17 PM, Hong <hzh...@mcs.anl.gov> wrote:
> Mark:
>
>> I would add:
>>
>> '--download-fblaslapack=1',
>>
>> This is what I have on my Linux machine (cg at ANL) and it runs clean.
>>
> I do not see any error report from nightly tests for ex56. Should I merge
> this branch to maint?
> Hong
>
>>
>> On Fri, Nov 10, 2017 at 11:03 AM, Hong <hzh...@mcs.anl.gov> wrote:
>>
>>> I use
>>> Using configure Options: --download-metis --download-mpich
>>> --download-mumps --download-parmetis --download-scalapack
>>> --download-superlu --download-superlu_dist --download-suitesparse
>>> --download-hypre --download-ptscotch --download-chaco --with-ctable=1
>>> --download-cmake --with-cc=gcc --with-cxx=g++ --with-debugging=1
>>> --with-visibility=0 --with-fc=gfortran
>>> Hong
>>>
>>> On Fri, Nov 10, 2017 at 9:59 AM, Mark Adams <mfad...@lbl.gov> wrote:
>>>
>>>> This must be a configure issue. I don't see these warning:
>>>>
>>>> #!/usr/bin/python
>>>> if __name__ == '__main__':
>>>> import sys
>>>> import os
>>>> sys.path.insert(0, os.path.abspath('config'))
>>>> import configure
>>>> configure_options = [
>>>> '--with-cc=clang',
>>>> '--with-cc++=clang++',
>>>> '--download-mpich=1',
>>>> '--download-metis=1',
>>>> '--download-superlu=1',
>>>> '--download-superlu_dist=1',
>>>> '--download-parmetis=1',
>>>> '--download-fblaslapack=1',
>>>> '--download-p4est=1',
>>>> '--with-debugging=1',
>>>> '--with-batch=0',
>>>> 'PETSC_ARCH=arch-linux2-c-dbg32',
>>>> '--with-openmp=0',
>>>> '--download-p4est=0'
>>>> ]
>>>> configure.petsc_configure(configure_options)
>>>>
>>>> ~
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Nov 10, 2017 at 10:56 AM, Mark Adams <mfad...@lbl.gov> wrote:
>>>>
>>>>> This is comming from blas. How did you configure blas?
>>>>>
>>>>> On Fri, Nov 10, 2017 at 10:38 AM, Hong <hzh...@mcs.anl.gov> wrote:
>>>>>
>>>>>> Using petsc machine, I get
>>>>>> hzhang@petsc /sandbox/hzhang/petsc/src/snes/examples/tutorials
>>>>>> (hzhang/fix-submat_samerowdist)
>>>>>> $ mpiexec -n 2 valgrind ./ex56 -cells 2,2,1 -max_conv_its 3
>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol
>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg
>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>>>>>> -snes_converged_reason
>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>>>>>> -run_type 1
>>>>>>
>>>>>> ==28811== Memcheck, a memory error detector
>>>>>> ==28811== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et
>>>>>> al.
>>>>>> ==28811== Using Valgrind-3.10.1 and LibVEX; rerun with -h for
>>>>>> copyright info
>>>>>> ==28811== Command: ./ex56 -cells 2,2,1 -max_conv_its 3
>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol
>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg
>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>>>>>> -snes_converged_reason
>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>>>>>> -run_type 1
>>>>>> ...
>>>>>> ==28811== Invalid read of size 16
>>>>>> ==28811== at 0x8550946: dswap_k_NEHALEM (in
>>>>>> /usr/lib/openblas-base/libblas.so.3)
>>>>>> ==28811== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas
>>>>>> .so.3)
>>>>>> ==28811== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.3
>>>>>> .0)
>>>>>> ==28811== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)
>>>>>> ==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>>>>>> ==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>>>>>> ==28811== by 0x41056E: main (ex56.c:395)
>>>>>> ==28811== Address 0xdc650d0 is 52,480 bytes inside a block of size
>>>>>> 52,488 alloc'd
>>>>>> ==28811== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me
>>>>>> mcheck-amd64-linux.so)
>>>>>> ==28811== by 0x51590F6: PetscMallocAlign (mal.c:39)
>>>>>> ==28811== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)
>>>>>> ==28811== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>>>>>> ==28811== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>>>>>> ==28811== by 0x41056E: main (ex56.c:395)
>>>>>> ==28811==
>>>>>> ==28812== Invalid read of size 16
>>>>>> ==28812== at 0x8550946: dswap_k_NEHALEM (in
>>>>>> /usr/lib/openblas-base/libblas.so.3)
>>>>>> ==28812== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas
>>>>>> .so.3)
>>>>>> ==28812== by 0x75B33B2: dgetri_ (in /usr/lib/lapack/liblapack.so.3
>>>>>> .0)
>>>>>> ==28812== by 0x5E3CA5C: PetscFESetUp_Basic (dtfe.c:4012)
>>>>>> ==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>>>>>> ==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>>>>>> ==28812== by 0x41056E: main (ex56.c:395)
>>>>>> ==28812== Address 0xd9c7600 is 52,480 bytes inside a block of size
>>>>>> 52,488 alloc'd
>>>>>> ==28812== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me
>>>>>> mcheck-amd64-linux.so)
>>>>>> ==28812== by 0x51590F6: PetscMallocAlign (mal.c:39)
>>>>>> ==28812== by 0x5E3C169: PetscFESetUp_Basic (dtfe.c:3983)
>>>>>> ==28812== by 0x5E320C9: PetscFESetUp (dtfe.c:3274)
>>>>>> ==28812== by 0x5E5786F: PetscFECreateDefault (dtfe.c:6749)
>>>>>> ==28812== by 0x41056E: main (ex56.c:395)
>>>>>> ==28812==
>>>>>> ==28811== Invalid read of size 16
>>>>>> ==28811== at 0x8550A55: dswap_k_NEHALEM (in
>>>>>> /usr/lib/openblas-base/libblas.so.3)
>>>>>> ==28811== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas
>>>>>> .so.3)
>>>>>> ==28811== by 0x7675179: dsteqr_ (in /usr/lib/lapack/liblapack.so.3
>>>>>> .0)
>>>>>> ==28811== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)
>>>>>> ==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)
>>>>>> ==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>>>>>> ==28811== by 0x41056E: main (ex56.c:395)
>>>>>> ==28811== Address 0xd99cbe0 is 64 bytes inside a block of size 72
>>>>>> alloc'd
>>>>>> ==28811== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me
>>>>>> mcheck-amd64-linux.so)
>>>>>> ==28811== by 0x51590F6: PetscMallocAlign (mal.c:39)
>>>>>> ==28811== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)
>>>>>> ==28811== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)
>>>>>> ==28811== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>>>>>> ==28811== by 0x41056E: main (ex56.c:395)
>>>>>> ==28811==
>>>>>> ==28812== Invalid read of size 16
>>>>>> ==28812== at 0x8550A55: dswap_k_NEHALEM (in
>>>>>> /usr/lib/openblas-base/libblas.so.3)
>>>>>> ==28812== by 0x7C6797F: dswap_ (in /usr/lib/openblas-base/libblas
>>>>>> .so.3)
>>>>>> ==28812== by 0x7675179: dsteqr_ (in /usr/lib/lapack/liblapack.so.3
>>>>>> .0)
>>>>>> ==28812== by 0x5DFFA22: PetscDTGaussQuadrature (dt.c:508)
>>>>>> ==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)
>>>>>> ==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>>>>>> ==28812== by 0x41056E: main (ex56.c:395)
>>>>>> ==28812== Address 0xdc11f30 is 64 bytes inside a block of size 72
>>>>>> alloc'd
>>>>>> ==28812== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_me
>>>>>> mcheck-amd64-linux.so)
>>>>>> ==28812== by 0x51590F6: PetscMallocAlign (mal.c:39)
>>>>>> ==28812== by 0x5DFF766: PetscDTGaussQuadrature (dt.c:504)
>>>>>> ==28812== by 0x5E00BD8: PetscDTGaussTensorQuadrature (dt.c:582)
>>>>>> ==28812== by 0x5E57D7A: PetscFECreateDefault (dtfe.c:6763)
>>>>>> ==28812== by 0x41056E: main (ex56.c:395)
>>>>>> ==28812==
>>>>>> [0] 27 global equations, 9 vertices
>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 1
>>>>>> [0] 441 global equations, 147 vertices
>>>>>> [0] 441 equations in vector, 147 vertices
>>>>>>
>>>>>> hangs here ...
>>>>>>
>>>>>> Hong
>>>>>>
>>>>>> On Fri, Nov 10, 2017 at 8:57 AM, Mark Adams <mfad...@lbl.gov> wrote:
>>>>>>
>>>>>>> This printed a little funny in gmail, snes/ex56 is running clean in
>>>>>>> the first few loops (appended), but the last one is the one with a
>>>>>>> reduced
>>>>>>> processor set. Still waiting. This is with 32 bit integers. I'm running
>>>>>>> another with 64 bit integers.
>>>>>>>
>>>>>>> ...
>>>>>>> [0] 27 global equations, 9 vertices
>>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations
>>>>>>> 1
>>>>>>> [0] 441 global equations, 147 vertices
>>>>>>> [0] 441 equations in vector, 147 vertices
>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations
>>>>>>> 1
>>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 10, 2017 at 9:06 AM, Mark Adams <mfad...@lbl.gov> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 9, 2017 at 1:56 PM, Hong <hzh...@mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Mark:
>>>>>>>>>
>>>>>>>>>> OK, well, just go with the Linux machine for the regression test.
>>>>>>>>>> I will keep trying to reproduce this on my Mac with an O build.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Valgrind error occurs on linux machines with g-build. I cannot
>>>>>>>>> merge this branch to maint until the bug is fixed.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Valgrind is failing on this run on my Mac. Moving to cg, like you I
>>>>>>>> suppose. This takes forever. This is what I have so far. Did you get
>>>>>>>> this
>>>>>>>> far?
>>>>>>>>
>>>>>>>> 07:48 hzhang/fix-submat_samerowdist *=
>>>>>>>> /sandbox/adams/petsc/src/snes/examples/tutorials$ make
>>>>>>>> PETSC_DIR=/sandbox/adams/petsc PETSC_ARCH=arch-linux2-c-dbg32 val
>>>>>>>> /sandbox/adams/petsc/arch-linux2-c-dbg32/bin/mpiexec -n 2 valgrind
>>>>>>>> ./ex56 -cells 2,2,1 -max_conv_its 3 -petscspace_order 2 -snes_max_it 2
>>>>>>>> -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type
>>>>>>>> unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg
>>>>>>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>>>>>>>> -snes_converged_reason
>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>>>>>>>> -run_type 1
>>>>>>>> ==12414== Memcheck, a memory error detector
>>>>>>>> ==12414== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward
>>>>>>>> et al.
>>>>>>>> ==12414== Using Valgrind-3.10.1 and LibVEX; rerun with -h for
>>>>>>>> copyright info
>>>>>>>> ==12415== Memcheck, a memory error detector
>>>>>>>> ==12415== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward
>>>>>>>> et al.
>>>>>>>> ==12415== Using Valgrind-3.10.1 and LibVEX; rerun with -h for
>>>>>>>> copyright info
>>>>>>>> ==12415== Command: ./ex56 -cells 2,2,1 -max_conv_its 3
>>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg
>>>>>>>> -ksp_rtol
>>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg
>>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>>>>>>>> -snes_converged_reason
>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>>>>>>>> -run_type 1
>>>>>>>> ==12415==
>>>>>>>> ==12414== Command: ./ex56 -cells 2,2,1 -max_conv_its 3
>>>>>>>> -petscspace_order 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg
>>>>>>>> -ksp_rtol
>>>>>>>> 1.e-11 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg
>>>>>>>> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10
>>>>>>>> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1
>>>>>>>> -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
>>>>>>>> -snes_converged_reason
>>>>>>>> -use_mat_nearnullspace true -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>>>>>>>> chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
>>>>>>>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type
>>>>>>>> jacobi -pc_gamg_mat_partitioning_type parmetis -mat_block_size 3
>>>>>>>> -run_type 1
>>>>>>>> ==12414==
>>>>>>>> [0] 27 global equations, 9 vertices
>>>>>>>> [0] 27 equations in vector, 9 vertices
>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>> iterations 1
>>>>>>>> [0] 441 global equations, 147 vertices
>>>>>>>> [0] 441 equations in vector, 147 vertices
>>>>>>>> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE
>>>>>>>> iterations 1
>>>>>>>> [0] 4725 global equations, 1575 vertices
>>>>>>>> [0] 4725 equations in vector, 1575 vertices
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
