Re: [petsc-dev] Having configure log what version of g++ the Intel compilers are utilizing?

[Pierre Jolivet]

> On 9 Jan 2024, at 6:08 PM, Barry Smith  wrote:
> 
> 
>PETSc developers
> 
>I cannot see in the configure.log information about what version of g++ 
> the Intel compilers are using. Could this be added to configure logging so we 
> know since outdated g++ versions are often a problem with Intel compilers? Or 
> is it in the configure.log but well hidden from my old eyes?

It’s already available: icpc version 19.1.1.217 (gcc version 9.4.0 
compatibility)

Thanks,
Pierre

> 
>> Begin forwarded message:
>> 
>> From: "Yesypenko, Anna" 
>> Subject: [petsc-maint] trouble configuring petsc with cuda on tacc lonestar6
>> Date: January 9, 2024 at 11:46:12 AM EST
>> To: "petsc-ma...@mcs.anl.gov" 
>> Message-Id: 
>> 
>> 
>> Dear Petsc developers,
>> 
>> I hope you can help me with an issue I've been having. I'm on a TACC machine 
>> attempting to install petsc with hypre and cuda.
>> When compiling and linking a CUDA object file during make​, I'm encountering 
>> an error: namespace "std" has no member "align_val_t".
>> 
>> I'm running the following configure command:
>> 
>> ./configure PETSC_ARCH=linux-cuda --download-hypre --with-cuda=1 
>> --cuda-dir=/opt/apps/cuda/11.4
>> 
>> The modules loaded are
>> 1) cuda/11.4 (g)   2) intel/19.1.1   3) nccl/2.11.4 (g)   4) 
>> mvapich2-gdr/2.3.7 where (g) built for GPU.
>> 
>> Attached are the configure and make log files.
>> Let me know if I can provide more information to troubleshoot this issue. 
>> Thank you!
>> 
>> Best,
>> Anna
> 
> 
> 

Re: [petsc-dev] [petsc-users] performance regression with GAMG

[Pierre Jolivet]

> On 11 Oct 2023, at 6:41 AM, Stephan Kramer  wrote:
> 
> On 07/10/2023 06:51, Pierre Jolivet wrote:
>> Hello Stephan,
>> Could you share the Amat/Pmat in binary format of the specific fieldsplit 
>> block, as well as all inputs needed to generate the same grid hierarchy 
>> (block size, options used, near kernel)?
>> Alternatively, have you been able to generate the same error in a plain 
>> PETSc example?
> I could but unfortunately, as Mark indicated, we only see this on on a very 
> large system, run on 1536 cores. The matrix dump appears to be 300G. If you 
> want I could try make it available but I imagine it's not the most practical 
> thing.

It should be OK on my end.

> We have tried the one line change you suggested below and it indeed prevents 
> the problem - i.e. on the  adams/gamg-fast-filter branch we get the 
> "inconsistent data" error with -pc_gamg_low_memory_filter True but not if we 
> change that line as suggested

OK, then that means the bug is indeed pretty localized.
Either MatEliminateZeros(), MatDuplicate(), or MatHeaderReplace().
Hong (Mr.), do you think there is something missing in 
MatEliminateZeros_MPIAIJ()? Maybe a call to MatDisAssemble_MPIAIJ() — I have no 
idea what this function does.

> Note that for our uses, we're happy to just not use the low memory filter (as 
> is now the default in main), but let us know if we can provide any further 
> help

I’m not happy with the same function being twice in the library, and having an 
“improved” version only available to a part of the library.
I’m also not happy with GAMG having tons of MatAIJ-specific code, which makes 
it unusable with other MatType, e.g., we can’t even use MatBAIJ or MatSBAIJ 
whereas PCHYPRE works even though it’s an external package (a good use case 
here would have been to ask you to use a MatBAIJ with bs = 1 to incriminate 
MatEliminateZeros_MPIAIJ() or not, but we can’t).
But that’s just my opinion.

Thanks,
Pierre

> Thanks
> Stephan
> 
>> 
>> I’m suspecting a bug in MatEliminateZeros(). If you have the chance to, 
>> could you please edit src/mat/impls/aij/mpi/mpiaij.c, change the line that 
>> looks like:
>> PetscCall(MatFilter(Gmat, filter, PETSC_TRUE, PETSC_TRUE));
>> Into:
>> PetscCall(MatFilter(Gmat, filter, PETSC_FALSE, PETSC_TRUE));
>> And give that a go? It will be extremely memory-inefficient, but this is 
>> just to confirm my intuition.
>> 
>> Thanks,
>> Pierre
>> 
>>> On 6 Oct 2023, at 1:22 AM, Stephan Kramer  
>>> <mailto:s.kra...@imperial.ac.uk> wrote:
>>> 
>>> Great, that seems to fix the issue indeed - i.e. on the branch with the low 
>>> memory filtering switched off (by default) we no longer see the 
>>> "inconsistent data" error or hangs, and going back to the square graph 
>>> aggressive coarsening brings us back the old performance. So we'd be keen 
>>> to have that branch merged indeed
>>> Many thanks for your assistance with this
>>> Stephan
>>> 
>>> On 05/10/2023 01:11, Mark Adams wrote:
>>>> Thanks Stephan,
>>>> 
>>>> It looks like the matrix is in a bad/incorrect state and parallel Mat-Mat
>>>> is waiting for messages that were not sent. A bug.
>>>> 
>>>> Can you try my branch, which is ready to merge, adams/gamg-fast-filter.
>>>> We added a new filtering method in main that uses low memory but I found it
>>>> was slow, so this branch brings back the old filter code, used by default,
>>>> and keeps the low memory version as an option.
>>>> It is possible this low memory filtering messed up the internals of the Mat
>>>> in some way.
>>>> I hope this is it, but if not we can continue.
>>>> 
>>>> This MR also makes square graph the default.
>>>> I have found it does create better aggregates and on GPUs, with Kokkos bug
>>>> fixes from Junchao, Mat-Mat is fast. (it might be slow on CPUs)
>>>> 
>>>> Mark
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Wed, Oct 4, 2023 at 12:30 AM Stephan Kramer  
>>>> <mailto:s.kra...@imperial.ac.uk>
>>>> wrote:
>>>> 
>>>>> Hi Mark
>>>>> 
>>>>> Thanks again for re-enabling the square graph aggressive coarsening
>>>>> option which seems to have restored performance for most of our cases.
>>>>> Unfortunately we do have a remaining issue, which only seems to occur
>>>>> for the larger mesh size ("lev

Re: [petsc-dev] [petsc-users] performance regression with GAMG

[Pierre Jolivet]

Hello Stephan,
Could you share the Amat/Pmat in binary format of the specific fieldsplit 
block, as well as all inputs needed to generate the same grid hierarchy (block 
size, options used, near kernel)?
Alternatively, have you been able to generate the same error in a plain PETSc 
example?

I’m suspecting a bug in MatEliminateZeros(). If you have the chance to, could 
you please edit src/mat/impls/aij/mpi/mpiaij.c, change the line that looks like:
PetscCall(MatFilter(Gmat, filter, PETSC_TRUE, PETSC_TRUE));
Into:
PetscCall(MatFilter(Gmat, filter, PETSC_FALSE, PETSC_TRUE));
And give that a go? It will be extremely memory-inefficient, but this is just 
to confirm my intuition.

Thanks,
Pierre

> On 6 Oct 2023, at 1:22 AM, Stephan Kramer  wrote:
> 
> Great, that seems to fix the issue indeed - i.e. on the branch with the low 
> memory filtering switched off (by default) we no longer see the "inconsistent 
> data" error or hangs, and going back to the square graph aggressive 
> coarsening brings us back the old performance. So we'd be keen to have that 
> branch merged indeed
> Many thanks for your assistance with this
> Stephan
> 
> On 05/10/2023 01:11, Mark Adams wrote:
>> Thanks Stephan,
>> 
>> It looks like the matrix is in a bad/incorrect state and parallel Mat-Mat
>> is waiting for messages that were not sent. A bug.
>> 
>> Can you try my branch, which is ready to merge, adams/gamg-fast-filter.
>> We added a new filtering method in main that uses low memory but I found it
>> was slow, so this branch brings back the old filter code, used by default,
>> and keeps the low memory version as an option.
>> It is possible this low memory filtering messed up the internals of the Mat
>> in some way.
>> I hope this is it, but if not we can continue.
>> 
>> This MR also makes square graph the default.
>> I have found it does create better aggregates and on GPUs, with Kokkos bug
>> fixes from Junchao, Mat-Mat is fast. (it might be slow on CPUs)
>> 
>> Mark
>> 
>> 
>> 
>> 
>> On Wed, Oct 4, 2023 at 12:30 AM Stephan Kramer 
>> wrote:
>> 
>>> Hi Mark
>>> 
>>> Thanks again for re-enabling the square graph aggressive coarsening
>>> option which seems to have restored performance for most of our cases.
>>> Unfortunately we do have a remaining issue, which only seems to occur
>>> for the larger mesh size ("level 7" which has 6,389,890 vertices and we
>>> normally run on 1536 cpus): we either get a "Petsc has generated
>>> inconsistent data" error, or a hang - both when constructing the square
>>> graph matrix. So this is with the new
>>> -pc_gamg_aggressive_square_graph=true option, without the option there's
>>> no error but of course we would get back to the worse performance.
>>> 
>>> Backtrace for the "inconsistent data" error. Note this is actually just
>>> petsc main from 17 Sep, git 9a75acf6e50cfe213617e - so after your merge
>>> of adams/gamg-add-old-coarsening into main - with one unrelated commit
>>> from firedrake
>>> 
>>> [0]PETSC ERROR: - Error Message
>>> --
>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>> [0]PETSC ERROR: j 8 not equal to expected number of sends 9
>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.4.2-43104-ga3b76b71a1  GIT Date: 2023-09-18 10:26:04 +0100
>>> [0]PETSC ERROR: stokes_cubed_sphere_7e3_A3_TS1.py on a  named
>>> gadi-cpu-clx-0241.gadi.nci.org.au by sck551 Wed Oct  4 14:30:41 2023
>>> [0]PETSC ERROR: Configure options --prefix=/tmp/firedrake-prefix
>>> --with-make-np=4 --with-debugging=0 --with-shared-libraries=1
>>> --with-fortran-bindings=0 --with-zlib --with-c2html=0
>>> --with-mpiexec=mpiexec --with-cc=mpicc --with-cxx=mpicxx
>>> --with-fc=mpifort --download-hdf5 --download-hypre
>>> --download-superlu_dist --download-ptscotch --download-suitesparse
>>> --download-pastix --download-hwloc --download-metis --download-scalapack
>>> --download-mumps --download-chaco --download-ml
>>> CFLAGS=-diag-disable=10441 CXXFLAGS=-diag-disable=10441
>>> [0]PETSC ERROR: #1 PetscGatherMessageLengths2() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/sys/utils/mpimesg.c:270
>>> [0]PETSC ERROR: #2 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:1867
>>> [0]PETSC ERROR: #3 MatProductSymbolic_AtB_MPIAIJ_MPIAIJ() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:2071
>>> [0]PETSC ERROR: #4 MatProductSymbolic() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/mat/interface/matproduct.c:795
>>> [0]PETSC ERROR: #5 PCGAMGSquareGraph_GAMG() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/ksp/pc/impls/gamg/gamg.c:489
>>> [0]PETSC ERROR: #6 PCGAMGCoarsen_AGG() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/ksp/pc/impls/gamg/agg.c:969
>>> [0]PETSC ERROR: #7 PCSetUp_GAMG() at
>>> /jobfs/95504034.gadi-pbs/petsc/src/ksp/pc/impls/gamg/gamg.c:645
>>> [0]PETSC ERROR: #8 PCSetUp() at
>>> 

Re: [petsc-dev] [petsc-users] performance regression with GAMG

[Pierre Jolivet]

On Oct 6, 2023, at 3:48 AM, Mark Adams  wrote:Pierre, (moved to dev)It looks like there is a subtle bug in the new MatFilter.My guess is that after the compression/filter the communication buffers and lists need to be recomputed because the graph has changed.Maybe an issue with MatHeaderReplace()?Do you have a reproducer?I use this routine for AIJ, BAIJ, and SBAIJ and never ran into this (though the subsequent Mat is not involved in the same kind of operations as in GAMG).Thanks,PierreAnd, the Mat-Mat Mults failed or hung because the communication requirements, as seen in the graph, did not match the cached communication lists.The old way just created a whole new matrix, which took care of that.MarkOn Thu, Oct 5, 2023 at 8:51 PM Mark Adams  wrote:Fantastic, it will get merged soon.Thank you for your diligence and patience.This would have been a time bomb waiting to explode.Mark On Thu, Oct 5, 2023 at 7:23 PM Stephan Kramer  wrote:Great, that seems to fix the issue indeed - i.e. on the branch with the 
low memory filtering switched off (by default) we no longer see the 
"inconsistent data" error or hangs, and going back to the square graph 
aggressive coarsening brings us back the old performance. So we'd be 
keen to have that branch merged indeed
Many thanks for your assistance with this
Stephan

On 05/10/2023 01:11, Mark Adams wrote:
> Thanks Stephan,
>
> It looks like the matrix is in a bad/incorrect state and parallel Mat-Mat
> is waiting for messages that were not sent. A bug.
>
> Can you try my branch, which is ready to merge, adams/gamg-fast-filter.
> We added a new filtering method in main that uses low memory but I found it
> was slow, so this branch brings back the old filter code, used by default,
> and keeps the low memory version as an option.
> It is possible this low memory filtering messed up the internals of the Mat
> in some way.
> I hope this is it, but if not we can continue.
>
> This MR also makes square graph the default.
> I have found it does create better aggregates and on GPUs, with Kokkos bug
> fixes from Junchao, Mat-Mat is fast. (it might be slow on CPUs)
>
> Mark
>
>
>
>
> On Wed, Oct 4, 2023 at 12:30 AM Stephan Kramer 
> wrote:
>
>> Hi Mark
>>
>> Thanks again for re-enabling the square graph aggressive coarsening
>> option which seems to have restored performance for most of our cases.
>> Unfortunately we do have a remaining issue, which only seems to occur
>> for the larger mesh size ("level 7" which has 6,389,890 vertices and we
>> normally run on 1536 cpus): we either get a "Petsc has generated
>> inconsistent data" error, or a hang - both when constructing the square
>> graph matrix. So this is with the new
>> -pc_gamg_aggressive_square_graph=true option, without the option there's
>> no error but of course we would get back to the worse performance.
>>
>> Backtrace for the "inconsistent data" error. Note this is actually just
>> petsc main from 17 Sep, git 9a75acf6e50cfe213617e - so after your merge
>> of adams/gamg-add-old-coarsening into main - with one unrelated commit
>> from firedrake
>>
>> [0]PETSC ERROR: - Error Message
>> --
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: j 8 not equal to expected number of sends 9
>> [0]PETSC ERROR: Petsc Development GIT revision:
>> v3.4.2-43104-ga3b76b71a1  GIT Date: 2023-09-18 10:26:04 +0100
>> [0]PETSC ERROR: stokes_cubed_sphere_7e3_A3_TS1.py on a  named
>> gadi-cpu-clx-0241.gadi.nci.org.au by sck551 Wed Oct  4 14:30:41 2023
>> [0]PETSC ERROR: Configure options --prefix=/tmp/firedrake-prefix
>> --with-make-np=4 --with-debugging=0 --with-shared-libraries=1
>> --with-fortran-bindings=0 --with-zlib --with-c2html=0
>> --with-mpiexec=mpiexec --with-cc=mpicc --with-cxx=mpicxx
>> --with-fc=mpifort --download-hdf5 --download-hypre
>> --download-superlu_dist --download-ptscotch --download-suitesparse
>> --download-pastix --download-hwloc --download-metis --download-scalapack
>> --download-mumps --download-chaco --download-ml
>> CFLAGS=-diag-disable=10441 CXXFLAGS=-diag-disable=10441
>> [0]PETSC ERROR: #1 PetscGatherMessageLengths2() at
>> /jobfs/95504034.gadi-pbs/petsc/src/sys/utils/mpimesg.c:270
>> [0]PETSC ERROR: #2 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() at
>> /jobfs/95504034.gadi-pbs/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:1867
>> [0]PETSC ERROR: #3 MatProductSymbolic_AtB_MPIAIJ_MPIAIJ() at
>> /jobfs/95504034.gadi-pbs/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:2071
>> [0]PETSC ERROR: #4 MatProductSymbolic() at
>> /jobfs/95504034.gadi-pbs/petsc/src/mat/interface/matproduct.c:795
>> [0]PETSC ERROR: #5 PCGAMGSquareGraph_GAMG() at
>> /jobfs/95504034.gadi-pbs/petsc/src/ksp/pc/impls/gamg/gamg.c:489
>> [0]PETSC ERROR: #6 PCGAMGCoarsen_AGG() at
>> /jobfs/95504034.gadi-pbs/petsc/src/ksp/pc/impls/gamg/agg.c:969
>> [0]PETSC ERROR: #7 PCSetUp_GAMG() at
>> 

Re: [petsc-dev] Is the petsc4py build broken?

[Pierre Jolivet]

petsc4py build is not broken, but maybe it is not future-proof and can’t handle 
Cython 3.0.0b2 (or there is a regression in Cython).
Could you downgrade your Cython (to an actual release and not a beta) and see 
if the error persists?

Thanks,
Pierre

> On 16 Apr 2023, at 12:25 AM, Satish Balay via petsc-dev 
>  wrote:
> 
> Works for me with latest main - so I'm not sure whats going on here..
> 
> Is this reproducible in a clean clone? Also - are you using mpi4py in this 
> build?
> https://gitlab.com/petsc/petsc/-/issues/1359
> [this issue looks different though..]
> 
> Satish
> 
> --
> 
> *** Building petsc4py ***
> running build
> running build_src
> using Cython 0.29.33
> cythonizing 'petsc4py/PETSc.pyx' -> 'petsc4py/PETSc.c'
> running build_py
> creating build
> creating build/lib.linux-x86_64-cpython-311
> creating build/lib.linux-x86_64-cpython-311/petsc4py
> copying src/petsc4py/PETSc.py -> build/lib.linux-x86_64-cpython-311/petsc4py
> copying src/petsc4py/__main__.py -> 
> build/lib.linux-x86_64-cpython-311/petsc4py
> copying src/petsc4py/__init__.py -> 
> build/lib.linux-x86_64-cpython-311/petsc4py
> creating build/lib.linux-x86_64-cpython-311/petsc4py/lib
> 
> 
> 
> On Sat, 15 Apr 2023, Matthew Knepley wrote:
> 
>> I get
>> 
>> *** Building petsc4py ***
>> running build
>> running build_src
>> removing Cython 0.29.30 from sys.modules
>> fetching build requirement 'Cython >= 0.29.32'
>> Searching for Cython>=0.29.32
>> Best match: Cython 3.0.0b2
>> Processing Cython-3.0.0b2-py3.8-macosx-10.9-x86_64.egg
>> 
>> Using
>> /PETSc3/petsc/petsc-dev/src/binding/petsc4py/.eggs/Cython-3.0.0b2-py3.8-macosx-10.9-x86_64.egg
>> 
>> using Cython 3.0.0b2
>> cythonizing 'petsc4py/PETSc.pyx' -> 'petsc4py/PETSc.c'
>> /PETSc3/petsc/petsc-dev/src/binding/petsc4py/conf/cythonize.py: No such
>> file or directory: 'petsc4py/PETSc.pyx'
>> error: Cython failure: 'petsc4py/PETSc.pyx' -> 'petsc4py/PETSc.c'
>> **ERROR*
>> Error building petsc4py.
>> 
>>  Thanks,
>> 
>> Matt
>> 
>> 
> 

[petsc-dev] PETSC_VIEWER_FAILED

[Pierre Jolivet]

Hello,
There is not much code around using PETSC_VIEWER_FAILED, and I don’t understand 
the following behavior.
diff --git a/src/ksp/ksp/tutorials/ex1.c b/src/ksp/ksp/tutorials/ex1.c
index 702467ebd51..708a2ff8c9a 100644
--- a/src/ksp/ksp/tutorials/ex1.c
+++ b/src/ksp/ksp/tutorials/ex1.c
@@ -147,3 +147,5 @@ int main(int argc, char **args)
   PetscCall(MatShift(A, 2.0));
+  
PetscCall(PetscViewerPushFormat(PETSC_VIEWER_STDOUT_(PetscObjectComm((PetscObject)ksp)),
 PETSC_VIEWER_FAILED));
   PetscCall(KSPSolve(ksp, b, x));
+  
PetscCall(PetscViewerPopFormat(PETSC_VIEWER_STDOUT_(PetscObjectComm((PetscObject)ksp;
 
$ ./ex1 -ksp_converged_reason
Linear solve converged due to CONVERGED_RTOL iterations 5
[…]
Linear solve converged due to CONVERGED_RTOL iterations 5

How do I get the second solve to remain silent _unless_ there is a failure?
Why is the code not doing what the documentation is stating, i.e., “use 
PETSC_VIEWER_FAILED to only display a reason if it fails” (maybe because the 
KSP has its own viewer but how do I retrieve it in a user-code)?

Thanks,
Piere

Re: [petsc-dev] Potential memory leak in PETSc - hypre interface when using Euclid

[Pierre Jolivet]

It appears this is a bug in the PETSc interface to hypre: 
https://github.com/hypre-space/hypre/issues/764

Thanks,
Pierre

> On 29 Oct 2022, at 5:56 AM, Pierre Jolivet  wrote:
> 
> 
> 
>> On 29 Oct 2022, at 12:55 AM, Jed Brown  wrote:
>> 
>> I think you mean 2.25 versus 2.26, and we can use 
>> --download-hypre-commit=origin/v2.26.0 to test if the Euclid issue is fixed 
>> (likely no because Euclid hasn't seen much activity this release cycle).
> 
> Yes, got the release numbers mixed up, sorry.
> I tried yesterday on my box with Valgrind and I got the same leak as you, so 
> your intuition is correct.
> 
> Thanks,
> Pierre
> 
>> Pierre Jolivet  writes:
>> 
>>> Please do note that we are currently stuck with version 2.15.0
>>> They broke something — using make instead of $(MAKE) — in 2.16.0 which is 
>>> making PETSc pipelines fail on FreeBSD boxes, cf. 
>>> https://github.com/hypre-space/hypre/issues/755#issuecomment-1279336057.
>>> We won’t be able to update hypre.py until this is resolved (or we would 
>>> have to remove --download-hypre from FreeBSD runners, or stick to 
>>> --download-hypre-commit=origin/v2.15.0).
>>> Satish, could you please try to ping Rob about this? Maybe you’ll have more 
>>> leverage, and maybe the leak is fixed in 2.16.0?
>>> 
>>> Thanks,
>>> Pierre
>>> 
>>>>> On 27 Oct 2022, at 11:43 PM, Zhang, Hong via petsc-dev 
>>>>>  wrote:
>>>> 
>>>> CCing Ruipeng. I think he can help with this.
>>>> 
>>>> Hong (Mr.)
>>>> 
>>>>> On Oct 27, 2022, at 3:53 PM, Barry Smith  wrote:
>>>>> 
>>>>> 
>>>>> My quick examination of hypre.c shows the only relevant code in PETSc is 
>>>>> 
>>>>> PetscCall(PetscOptionsEList("-pc_hypre_boomeramg_smooth_type", "Enable 
>>>>> more complex smoothers", "None", HYPREBoomerAMGSmoothType, 
>>>>> PETSC_STATIC_ARRAY_LENGTH(HYPREBoomerAMGSmoothType), 
>>>>> HYPREBoomerAMGSmoothType[0], , ));
>>>>> if (flg) {
>>>>> jac->smoothtype = indx;
>>>>> PetscCallExternal(HYPRE_BoomerAMGSetSmoothType, jac->hsolver, indx + 6);
>>>>> 
>>>>> In other words PETSc just sends this option off to hypre and does not 
>>>>> create any objects or allocate any memory based on this option.
>>>>> 
>>>>> Thus my conclusion is the memory leak is within hypre. Likely valgrind 
>>>>> would locate the exact position easily.
>>>>> 
>>>>>> On Oct 27, 2022, at 4:27 PM, Emil Constantinescu via petsc-dev 
>>>>>>  wrote:
>>>>>> 
>>>>>> Hi there,
>>>>>> 
>>>>>> Tang Qi (LANL) reported a potential memory leak when using hypre/Euclid. 
>>>>>> Upon rudimentary testing, I could reproduce it for many examples in 
>>>>>> PETSc TS. The symptom is memory usage (measured with top)  with the 
>>>>>> number of time steps. Without Euclid, memory use does not increase.
>>>>>> 
>>>>>> For instance, one can try ex15 under TS:
>>>>>> 
>>>>>> ex15  -da_grid_x 50 -da_grid_y 50 -boundary 0 -ts_max_steps 20 -Jtype 1 
>>>>>> -ts_monitor -pc_type hypre -pc_hypre_boomeramg_smooth_type Euclid
>>>>>> 
>>>>>> I am not sure if it's PETSc - hypre that causes the memory use or hypre 
>>>>>> itself.
>>>>>> 
>>>>>> Can someone with more sophisticated tools take a look at it?
>>>>>> 
>>>>>> Emil
>>>>> 
>>>> 

Re: [petsc-dev] Potential memory leak in PETSc - hypre interface when using Euclid

[Pierre Jolivet]

> On 29 Oct 2022, at 12:55 AM, Jed Brown  wrote:
> 
> I think you mean 2.25 versus 2.26, and we can use 
> --download-hypre-commit=origin/v2.26.0 to test if the Euclid issue is fixed 
> (likely no because Euclid hasn't seen much activity this release cycle).

Yes, got the release numbers mixed up, sorry.
I tried yesterday on my box with Valgrind and I got the same leak as you, so 
your intuition is correct.

Thanks,
Pierre

> Pierre Jolivet  writes:
> 
>> Please do note that we are currently stuck with version 2.15.0
>> They broke something — using make instead of $(MAKE) — in 2.16.0 which is 
>> making PETSc pipelines fail on FreeBSD boxes, cf. 
>> https://github.com/hypre-space/hypre/issues/755#issuecomment-1279336057.
>> We won’t be able to update hypre.py until this is resolved (or we would have 
>> to remove --download-hypre from FreeBSD runners, or stick to 
>> --download-hypre-commit=origin/v2.15.0).
>> Satish, could you please try to ping Rob about this? Maybe you’ll have more 
>> leverage, and maybe the leak is fixed in 2.16.0?
>> 
>> Thanks,
>> Pierre
>> 
>>>> On 27 Oct 2022, at 11:43 PM, Zhang, Hong via petsc-dev 
>>>>  wrote:
>>> 
>>> CCing Ruipeng. I think he can help with this.
>>> 
>>> Hong (Mr.)
>>> 
>>>> On Oct 27, 2022, at 3:53 PM, Barry Smith  wrote:
>>>> 
>>>> 
>>>> My quick examination of hypre.c shows the only relevant code in PETSc is 
>>>> 
>>>> PetscCall(PetscOptionsEList("-pc_hypre_boomeramg_smooth_type", "Enable 
>>>> more complex smoothers", "None", HYPREBoomerAMGSmoothType, 
>>>> PETSC_STATIC_ARRAY_LENGTH(HYPREBoomerAMGSmoothType), 
>>>> HYPREBoomerAMGSmoothType[0], , ));
>>>> if (flg) {
>>>>  jac->smoothtype = indx;
>>>>  PetscCallExternal(HYPRE_BoomerAMGSetSmoothType, jac->hsolver, indx + 6);
>>>> 
>>>> In other words PETSc just sends this option off to hypre and does not 
>>>> create any objects or allocate any memory based on this option.
>>>> 
>>>> Thus my conclusion is the memory leak is within hypre. Likely valgrind 
>>>> would locate the exact position easily.
>>>> 
>>>>> On Oct 27, 2022, at 4:27 PM, Emil Constantinescu via petsc-dev 
>>>>>  wrote:
>>>>> 
>>>>> Hi there,
>>>>> 
>>>>> Tang Qi (LANL) reported a potential memory leak when using hypre/Euclid. 
>>>>> Upon rudimentary testing, I could reproduce it for many examples in PETSc 
>>>>> TS. The symptom is memory usage (measured with top)  with the number of 
>>>>> time steps. Without Euclid, memory use does not increase.
>>>>> 
>>>>> For instance, one can try ex15 under TS:
>>>>> 
>>>>> ex15  -da_grid_x 50 -da_grid_y 50 -boundary 0 -ts_max_steps 20 -Jtype 1 
>>>>> -ts_monitor -pc_type hypre -pc_hypre_boomeramg_smooth_type Euclid
>>>>> 
>>>>> I am not sure if it's PETSc - hypre that causes the memory use or hypre 
>>>>> itself.
>>>>> 
>>>>> Can someone with more sophisticated tools take a look at it?
>>>>> 
>>>>> Emil
>>>> 
>>> 

Re: [petsc-dev] Potential memory leak in PETSc - hypre interface when using Euclid

[Pierre Jolivet]

Please do note that we are currently stuck with version 2.15.0
They broke something — using make instead of $(MAKE) — in 2.16.0 which is 
making PETSc pipelines fail on FreeBSD boxes, cf. 
https://github.com/hypre-space/hypre/issues/755#issuecomment-1279336057.
We won’t be able to update hypre.py until this is resolved (or we would have to 
remove --download-hypre from FreeBSD runners, or stick to 
--download-hypre-commit=origin/v2.15.0).
Satish, could you please try to ping Rob about this? Maybe you’ll have more 
leverage, and maybe the leak is fixed in 2.16.0?

Thanks,
Pierre

> On 27 Oct 2022, at 11:43 PM, Zhang, Hong via petsc-dev 
>  wrote:
> 
> CCing Ruipeng. I think he can help with this.
> 
> Hong (Mr.)
> 
>> On Oct 27, 2022, at 3:53 PM, Barry Smith  wrote:
>> 
>> 
>> My quick examination of hypre.c shows the only relevant code in PETSc is 
>> 
>> PetscCall(PetscOptionsEList("-pc_hypre_boomeramg_smooth_type", "Enable more 
>> complex smoothers", "None", HYPREBoomerAMGSmoothType, 
>> PETSC_STATIC_ARRAY_LENGTH(HYPREBoomerAMGSmoothType), 
>> HYPREBoomerAMGSmoothType[0], , ));
>> if (flg) {
>>   jac->smoothtype = indx;
>>   PetscCallExternal(HYPRE_BoomerAMGSetSmoothType, jac->hsolver, indx + 6);
>> 
>> In other words PETSc just sends this option off to hypre and does not create 
>> any objects or allocate any memory based on this option.
>> 
>> Thus my conclusion is the memory leak is within hypre. Likely valgrind would 
>> locate the exact position easily.
>> 
>>> On Oct 27, 2022, at 4:27 PM, Emil Constantinescu via petsc-dev 
>>>  wrote:
>>> 
>>> Hi there,
>>> 
>>> Tang Qi (LANL) reported a potential memory leak when using hypre/Euclid. 
>>> Upon rudimentary testing, I could reproduce it for many examples in PETSc 
>>> TS. The symptom is memory usage (measured with top)  with the number of 
>>> time steps. Without Euclid, memory use does not increase.
>>> 
>>> For instance, one can try ex15 under TS:
>>> 
>>> ex15  -da_grid_x 50 -da_grid_y 50 -boundary 0 -ts_max_steps 20 -Jtype 1 
>>> -ts_monitor -pc_type hypre -pc_hypre_boomeramg_smooth_type Euclid
>>> 
>>> I am not sure if it's PETSc - hypre that causes the memory use or hypre 
>>> itself.
>>> 
>>> Can someone with more sophisticated tools take a look at it?
>>> 
>>> Emil
>> 
> 

Re: [petsc-dev] MatProduct_AtB --with-scalar-type=complex

[Pierre Jolivet]

Barry,
MatTransposeMatMultSymbolic_SeqAIJ_SeqAIJ() is indeed called.
product->alg is default, square is PETSC_FALSE.

Hong,
I believe the issue comes from the fact that atb->updateAt is PETSC_FALSE in 
MatProductNumeric_AtB_SeqAIJ_SeqAIJ().
If the name of this variable is relevant to its purpose, I believe it should be 
set to PETSC_TRUE when calling MatProductReplaceMats() whenever A is changed.
I would prefer using MatProductReplaceMats() because I’m implementing the same 
MatConvert() as MatNormal for the Hermitian case and it’s the only way to reuse 
the symbolic product, cf. 
https://petsc.org/main/src/mat/impls/normal/normm.c.html#line315 
<https://petsc.org/main/src/mat/impls/normal/normm.c.html#line315> in the case 
where --with-scalar-type=real

Thanks,
Pierre

> On 15 Jul 2022, at 5:52 AM, Zhang, Hong  wrote:
> 
> Pierre,
> Our MatProductReplaceMats() is not well tested, which might be buggy. I 
> simplified your code without calling MatProductReplaceMats() and got correct 
> results in the cases
> ./ex -product_view ::ascii_matlab -convert false/true -correct false
> and
> ./ex -product_view ::ascii_matlab -convert false/true -correct true
> 
> My code is attached. I'll investigate MatProductReplaceMats().
> Hong
> 
> 
> 
> 
> From: petsc-dev  on behalf of Barry Smith 
> 
> Sent: Thursday, July 14, 2022 4:38 PM
> To: Pierre Jolivet 
> Cc: For users of the development version of PETSc 
> Subject: Re: [petsc-dev] MatProduct_AtB --with-scalar-type=complex
>  
> 
>   Can you confirm if MatTransposeMatMultSymbolic_SeqAIJ_SeqAIJ() ends up 
> being called for you and what path it takes inside that routine (depends) on 
> the algorithm it is using. 
> 
> 
> 
> > On Jul 14, 2022, at 4:30 PM, Pierre Jolivet  wrote:
> > 
> > Hello,
> > In the following example, the SeqAIJ implementation of MatProduct_AtB 
> > produce a different (and wrong) result, compared to the SeqDense 
> > implementation or MATLAB.
> > I want to compute B = A^H A (where ^H is the Hermitian transpose).
> > So I create a MatProduct with A and A.
> > Duplicate A into another Mat which I conjugate.
> > And I replace the first Mat of the product with this conjugate.
> > I expect to get the proper result, which I don’t.
> > Is the MatProduct_AtB implementation in the complex case not computing A^T 
> > B (where ^T is the transpose)? 
> > For reference, here is how to properly compute A^H A with current main: 
> > conj(A^H conj(A)) — so it requires an extra MatConjugate I’d like to avoid.
> > 
> > Thanks,
> > Pierre
> > 
> > 
> > 
> > $ ./ex -product_view ::ascii_matlab -A_view ::ascii_matlab -convert 
> > false   
> > %Mat Object: 1 MPI process
> > %  type: seqdense
> > % Size = 2 2 
> > Mat_0xc401_0 = zeros(2,2);
> > Mat_0xc401_0 = [
> > 7.2003197397953400e-01 + 6.1793966542126100e-02i 3.980919128602e-01 + 
> > 7.3036588248200474e-02i 
> > 1.0022337819588500e-02 + 1.4463931936456476e-01i 1.0386628927366459e-01 + 
> > 2.5078039364333193e-01i 
> > ];
> > %Mat Object: 1 MPI process
> > %  type: seqdense
> > % Size = 2 2 
> > Mat_0xc401_1 = zeros(2,2);
> > Mat_0xc401_1 = [
> > 5.4328551781548817e-01 + 0.e+00i 3.2823965013353340e-01 + 
> > 1.549814872689e-02i 
> > 3.2823965013353340e-01 + -1.549814872689e-02i 2.3724054059134142e-01 + 
> > 0.e+00i 
> > ];
> > 
> > $ ./ex -product_view ::ascii_matlab -convert true   
> > %Mat Object: 1 MPI process
> > %  type: seqaij
> > % Size = 2 2 
> > % Nonzeros = 4 
> > zzz = zeros(4,4);
> > zzz = [
> > 1 1  4.9380746380098023e-01 9.1886511660038694e-02
> > 1 2  2.4666779825931440e-01 9.4705502650537468e-02
> > 2 1  2.4666779825931440e-01 9.4705502650537468e-02
> > 2 2  1.0079024247365802e-01 1.1019992594899400e-01
> > ];
> > Mat_0xc401_0 = spconvert(zzz);
> > 
> > $ ./ex -product_view ::ascii_matlab -convert true -correct true
> > %Mat Object: 1 MPI process
> > %  type: seqaij
> > % Size = 2 2 
> > % Nonzeros = 4 
> > zzz = zeros(4,4);
> > zzz = [
> > 1 1  5.4328551781548828e-01 -0.e+00
> > 1 2  3.2823965013353340e-01 1.549814872696e-02
> > 2 1  3.2823965013353340e-01 -1.549814872696e-02
> > 2 2  2.3724054059134142e-01 -0.e+00
> > ];
> > Mat_0xc401_0 = spconvert(zzz);
> > 
> > 
> 
> 

[petsc-dev] MatProduct_AtB --with-scalar-type=complex

[Pierre Jolivet]

Hello,
In the following example, the SeqAIJ implementation of MatProduct_AtB produce a 
different (and wrong) result, compared to the SeqDense implementation or MATLAB.
I want to compute B = A^H A (where ^H is the Hermitian transpose).
So I create a MatProduct with A and A.
Duplicate A into another Mat which I conjugate.
And I replace the first Mat of the product with this conjugate.
I expect to get the proper result, which I don’t.
Is the MatProduct_AtB implementation in the complex case not computing A^T B 
(where ^T is the transpose)? 
For reference, here is how to properly compute A^H A with current main: 
conj(A^H conj(A)) — so it requires an extra MatConjugate I’d like to avoid.

Thanks,
Pierre



ex.c
Description: Binary data


$ ./ex -product_view ::ascii_matlab -A_view ::ascii_matlab -convert false   

%Mat Object: 1 MPI process
%  type: seqdense
% Size = 2 2 
Mat_0xc401_0 = zeros(2,2);
Mat_0xc401_0 = [
7.2003197397953400e-01 + 6.1793966542126100e-02i 3.980919128602e-01 + 
7.3036588248200474e-02i 
1.0022337819588500e-02 + 1.4463931936456476e-01i 1.0386628927366459e-01 + 
2.5078039364333193e-01i 
];
%Mat Object: 1 MPI process
%  type: seqdense
% Size = 2 2 
Mat_0xc401_1 = zeros(2,2);
Mat_0xc401_1 = [
5.4328551781548817e-01 + 0.e+00i 3.2823965013353340e-01 + 
1.549814872689e-02i 
3.2823965013353340e-01 + -1.549814872689e-02i 2.3724054059134142e-01 + 
0.e+00i 
];

$ ./ex -product_view ::ascii_matlab -convert true   
%Mat Object: 1 MPI process
%  type: seqaij
% Size = 2 2 
% Nonzeros = 4 
zzz = zeros(4,4);
zzz = [
1 1  4.9380746380098023e-01 9.1886511660038694e-02
1 2  2.4666779825931440e-01 9.4705502650537468e-02
2 1  2.4666779825931440e-01 9.4705502650537468e-02
2 2  1.0079024247365802e-01 1.1019992594899400e-01
];
 Mat_0xc401_0 = spconvert(zzz);

$ ./ex -product_view ::ascii_matlab -convert true -correct true
%Mat Object: 1 MPI process
%  type: seqaij
% Size = 2 2 
% Nonzeros = 4 
zzz = zeros(4,4);
zzz = [
1 1  5.4328551781548828e-01 -0.e+00
1 2  3.2823965013353340e-01 1.549814872696e-02
2 1  3.2823965013353340e-01 -1.549814872696e-02
2 2  2.3724054059134142e-01 -0.e+00
];
 Mat_0xc401_0 = spconvert(zzz);

Re: [petsc-dev] PCFIELDSPLIT with inner -ksp_reuse_preconditioner

[Pierre Jolivet]

> On 13 Jul 2022, at 6:01 PM, Barry Smith  wrote:
> 
> 
>   Some of the ugliness of SetFromOptions in some of the nested solvers :-(

Right, I figured it out slightly after sending the mail by pulling the sub KSP 
and explicitly calling KSPSetFromOptions on it, which also works.
I’ve provided a fix to my user by telling them to pull the sub KSP, so if you 
are not satisfied by your patch, rest assured that they can live with this 
“undocumented feature” for the time being.

Thanks,
Pierre

> * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
>   * frame #0: 0x0001132795f0 
> libpetsc.3.017.3.dylib`KSPSetFromOptions(ksp=0x7f899f149a70) at 
> itcl.c:324:19
> frame #1: 0x000112eddda4 
> libpetsc.3.017.3.dylib`PCSetUp_FieldSplit(pc=0x7f899f08d870) at 
> fieldsplit.c:1087:32
> frame #2: 0x000112dfb874 
> libpetsc.3.017.3.dylib`PCSetUp(pc=0x7f899f08d870) at precon.c:991:5
> frame #3: 0x000113285834 
> libpetsc.3.017.3.dylib`KSPSetUp(ksp=0x7f899e95f670) at itfunc.c:401:3
> frame #4: 0x00011328b5e9 
> libpetsc.3.017.3.dylib`KSPSolve_Private(ksp=0x7f899e95f670, 
> b=0x7f899f0dec70, x=0x7f899f0ce870) at itfunc.c:835:3
> frame #5: 0x00011328aa66 
> libpetsc.3.017.3.dylib`KSPSolve(ksp=0x7f899e95f670, b=0x7f899f0dec70, 
> x=0x7f899f0ce870) at itfunc.c:1066:3
> frame #6: 0x00010e7c94e8 ex`main(argc=12, 
> args=0x7ff7b1739328) at ex.c:33:3
> frame #7: 0x00011703751e dyld`start + 462
> 
> The SetFromOptions doesn't always traverse all the sub-methods suitably 
> calling SetFromOptions on them because they may not have been created by the 
> time that
> the SetFromOptions is first called. In this case, the PCSetUp_FieldSplit() 
> creates the sub solvers and calls the options database on them after the 
> initial call to PCSetFromOptions_FieldSplit() 
> 
> I've attached a patch that should fix this particular issue. 
> 
> I ran it with your test code, and it seemed to resolve the issue and ran the 
> test suite, and it did not break anything.
> 
> A redesigned/implemented PETSc would need some thought on how to improve the 
> SetFromOptions model to prevent this kind of difficulty.
> 
> Barry
> 
> 
> 
> 
>> On Jul 13, 2022, at 8:40 AM, Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> 
>> Hello,
>> A user(*) wants to switch back and forth with the option 
>> -fieldsplit_1_ksp_reuse_preconditioner, but it’s generating results that I 
>> don’t understand.
>> In the following example I solve three successive linear systems.
>> If I run
>> $ ./ex -pc_type fieldsplit -fieldsplit_pc_type lu -log_view -info 
>> -options_view -switch false
>> -fieldsplit_1_ksp_reuse_preconditioner is always false, and I properly get:
>> 1) convergence in a single iteration every time 
>> 2) [0] PCSetUp(): Setting up PC with same nonzero pattern for the last solve
>> 3) MatLUFactorNum 3 in -log_view
>> 
>> Now, if I run
>> $ ./ex -pc_type fieldsplit -fieldsplit_pc_type lu -log_view -info 
>> -options_view -switch true
>> -fieldsplit_1_ksp_reuse_preconditioner is set to true for the second solve 
>> and switched back to false for the last solve.
>> I (wrongfully, IMHO?) get:
>> 1) convergence in 2 iterations for the last solve (instead of 1?)
>> 2) [0] PCSetUp(): Leaving PC with identical preconditioner since reuse 
>> preconditioner is set (which should not be set for the last solve?)
>> 3) MatLUFactorNum 1 (instead of 2?)
>> 4) #PETSc Option Table entries:
>> -fieldsplit_1_ksp_reuse_preconditioner false (though from -info, cf. point 
>> #2 just above, this option value is true?)
>> 
>> Does it make sense or do you agree that something is off?
>> 
>> Thanks,
>> Pierre
>> 
>> 
>> 
>> (*) 
>> https://community.freefem.org/t/supply-petsc-numbering-for-shared-memory-block-matrix/486/18
>>  
>> <https://community.freefem.org/t/supply-petsc-numbering-for-shared-memory-block-matrix/486/18>

[petsc-dev] PCFIELDSPLIT with inner -ksp_reuse_preconditioner

[Pierre Jolivet]

Hello,
A user(*) wants to switch back and forth with the option 
-fieldsplit_1_ksp_reuse_preconditioner, but it’s generating results that I 
don’t understand.
In the following example I solve three successive linear systems.
If I run
$ ./ex -pc_type fieldsplit -fieldsplit_pc_type lu -log_view -info 
-options_view -switch false
-fieldsplit_1_ksp_reuse_preconditioner is always false, and I properly get:
1) convergence in a single iteration every time 
2) [0] PCSetUp(): Setting up PC with same nonzero pattern for the last solve
3) MatLUFactorNum 3 in -log_view

Now, if I run
$ ./ex -pc_type fieldsplit -fieldsplit_pc_type lu -log_view -info 
-options_view -switch true
-fieldsplit_1_ksp_reuse_preconditioner is set to true for the second solve and 
switched back to false for the last solve.
I (wrongfully, IMHO?) get:
1) convergence in 2 iterations for the last solve (instead of 1?)
2) [0] PCSetUp(): Leaving PC with identical preconditioner since reuse 
preconditioner is set (which should not be set for the last solve?)
3) MatLUFactorNum 1 (instead of 2?)
4) #PETSc Option Table entries:
-fieldsplit_1_ksp_reuse_preconditioner false (though from -info, cf. point #2 
just above, this option value is true?)

Does it make sense or do you agree that something is off?

Thanks,
Pierre



ex.c
Description: Binary data


(*) 
https://community.freefem.org/t/supply-petsc-numbering-for-shared-memory-block-matrix/486/18

Re: [petsc-dev] VecNest

[Pierre Jolivet]

> On 12 Jul 2022, at 2:50 PM, Matthew Knepley  wrote:
> 
> On Tue, Jul 12, 2022 at 7:41 AM Matthew Knepley  <mailto:knep...@gmail.com>> wrote:
> On Tue, Jul 12, 2022 at 7:39 AM Pierre Jolivet  <mailto:pie...@joliv.et>> wrote:
> 
> 
>> On 12 Jul 2022, at 2:32 PM, Matthew Knepley > <mailto:knep...@gmail.com>> wrote:
>> 
>> On Mon, Jul 11, 2022 at 1:17 PM Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> Hello,
>> Could anyone help me understand what is going on in the following example, 
>> please?
>> I have a VecNest.
>> I either: a) initialize all values to 0.0, then set a specific part of the 
>> vector to nonzero or b) initialize a part of the vector to 0.0 and set the 
>> other part to nonzero.
>> I don’t see why a) and b) produce different results.
>> 
>> $ ./ex -pc_type fieldsplit -ksp_monitor_true_residual 
>> -ksp_converged_reason -fieldsplit_pc_type jacobi -ksp_pc_side right 
>> -ksp_view_final_residual -nest_subvec true 
>>   0 KSP unpreconditioned resid norm 8.375635517980e-01 true resid norm 
>> 8.375635517980e-01 ||r(i)||/||b|| 1.e+00
>>   1 KSP unpreconditioned resid norm 4.748816884247e-01 true resid norm 
>> 4.748816884247e-01 ||r(i)||/||b|| 5.669798875623e-01
>>   2 KSP unpreconditioned resid norm 4.713006778679e-01 true resid norm 
>> 4.713006778679e-01 ||r(i)||/||b|| 5.627043784990e-01
>>   3 KSP unpreconditioned resid norm 7.092979927129e-02 true resid norm 
>> 7.092979927129e-02 ||r(i)||/||b|| 8.468587144106e-02
>>   4 KSP unpreconditioned resid norm 1.457836310255e-02 true resid norm 
>> 1.457836310255e-02 ||r(i)||/||b|| 1.740567992870e-02
>>   5 KSP unpreconditioned resid norm 1.625040500524e-14 true resid norm 
>> 1.633468028779e-14 ||r(i)||/||b|| 1.950261595401e-14
>> Linear solve converged due to CONVERGED_RTOL iterations 5
>> KSP final norm of residual 1.63347e-14
>> $ ./ex -pc_type fieldsplit -ksp_monitor_true_residual 
>> -ksp_converged_reason -fieldsplit_pc_type jacobi -ksp_pc_side right 
>> -ksp_view_final_residual -nest_subvec false
>>   0 KSP unpreconditioned resid norm 0.e+00 true resid norm 
>> 8.375635517980e-01 ||r(i)||/||b||inf
>> Linear solve converged due to CONVERGED_ATOL iterations 0
>> KSP final norm of residual 0.837564
>> 
>> I find if I assemble the vector, I get the same answers. Will try to figure 
>> out what assembly is doing.
> 
> It’s probably reseting all these values 
> https://petsc.org/main/src/vec/vec/interface/rvector.c.html#line511 
> <https://petsc.org/main/src/vec/vec/interface/rvector.c.html#line511>, which 
> I believe are being used in VecNorm() inside VecNormalize().
> I guess any call to VecNestSubVec() should invalidate those as well, 
> otherwise we get wrong cached norms.
> I will give this a go.
> 
> I believe the bug is the following:
> 
>   You change values in a subvector, which does StateIncrease on the 
> subvector, but not StateIncrease on the nest vector, so it has its cached norm
> 
> I am not sure what to do about this, since how can the parent know you pulled 
> out the subvector? Will think.
> 
> Okay, the error is more subtle. VecSet() automatically computes some norms. 
> VecNestGetSubVec() is broken with respect to caching.
> I think the right fix now is VecNestGetSubVec/s() should increase the state 
> no matter what. If we want to get more precise, we need
> to have Restore() for the subvecs, which checks the state of the subvector 
> when it is restored.
> 
> Do you want to change it?

https://gitlab.com/petsc/petsc/-/merge_requests/5420 
<https://gitlab.com/petsc/petsc/-/merge_requests/5420> gets the job done, and 
is hopefully not too far from what you had in mind.
If anyone is not confortable with the incurred cost (e.g., of recomputing some 
norms), please let me know.

Thanks,
Pierre

>   Thanks,
> 
>  Matt
>  
>Matt
>  
> Thanks,
> Pierre
> 
>>   Thanks,
>> 
>> Matt
>>  
>> Thanks,
>> Pierre
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments 
>> is infinitely more interesting than any results to which their experiments 
>> lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-dev] VecNest

[Pierre Jolivet]

> On 12 Jul 2022, at 2:32 PM, Matthew Knepley  wrote:
> 
> On Mon, Jul 11, 2022 at 1:17 PM Pierre Jolivet  <mailto:pie...@joliv.et>> wrote:
> Hello,
> Could anyone help me understand what is going on in the following example, 
> please?
> I have a VecNest.
> I either: a) initialize all values to 0.0, then set a specific part of the 
> vector to nonzero or b) initialize a part of the vector to 0.0 and set the 
> other part to nonzero.
> I don’t see why a) and b) produce different results.
> 
> $ ./ex -pc_type fieldsplit -ksp_monitor_true_residual 
> -ksp_converged_reason -fieldsplit_pc_type jacobi -ksp_pc_side right 
> -ksp_view_final_residual -nest_subvec true 
>   0 KSP unpreconditioned resid norm 8.375635517980e-01 true resid norm 
> 8.375635517980e-01 ||r(i)||/||b|| 1.e+00
>   1 KSP unpreconditioned resid norm 4.748816884247e-01 true resid norm 
> 4.748816884247e-01 ||r(i)||/||b|| 5.669798875623e-01
>   2 KSP unpreconditioned resid norm 4.713006778679e-01 true resid norm 
> 4.713006778679e-01 ||r(i)||/||b|| 5.627043784990e-01
>   3 KSP unpreconditioned resid norm 7.092979927129e-02 true resid norm 
> 7.092979927129e-02 ||r(i)||/||b|| 8.468587144106e-02
>   4 KSP unpreconditioned resid norm 1.457836310255e-02 true resid norm 
> 1.457836310255e-02 ||r(i)||/||b|| 1.740567992870e-02
>   5 KSP unpreconditioned resid norm 1.625040500524e-14 true resid norm 
> 1.633468028779e-14 ||r(i)||/||b|| 1.950261595401e-14
> Linear solve converged due to CONVERGED_RTOL iterations 5
> KSP final norm of residual 1.63347e-14
> $ ./ex -pc_type fieldsplit -ksp_monitor_true_residual 
> -ksp_converged_reason -fieldsplit_pc_type jacobi -ksp_pc_side right 
> -ksp_view_final_residual -nest_subvec false
>   0 KSP unpreconditioned resid norm 0.e+00 true resid norm 
> 8.375635517980e-01 ||r(i)||/||b||inf
> Linear solve converged due to CONVERGED_ATOL iterations 0
> KSP final norm of residual 0.837564
> 
> I find if I assemble the vector, I get the same answers. Will try to figure 
> out what assembly is doing.

It’s probably reseting all these values 
https://petsc.org/main/src/vec/vec/interface/rvector.c.html#line511 
<https://petsc.org/main/src/vec/vec/interface/rvector.c.html#line511>, which I 
believe are being used in VecNorm() inside VecNormalize().
I guess any call to VecNestSubVec() should invalidate those as well, otherwise 
we get wrong cached norms.
I will give this a go.

Thanks,
Pierre

>   Thanks,
> 
> Matt
>  
> Thanks,
> Pierre
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-dev] VecNest

[Pierre Jolivet]

> On 12 Jul 2022, at 2:13 PM, Pierre Jolivet  wrote:
> 
> 
> 
>> On 12 Jul 2022, at 2:04 PM, Mark Adams > <mailto:mfad...@lbl.gov>> wrote:
>> 
>> It looks like the RHS is zero in the 2nd case (0 KSP unpreconditioned resid 
>> norm 0.e+00), but the true residual is the same.
>> 
>> It looks like you added "nest_subvec" to our example.
> 
> It’s a new (dumb) example (ex).
> 
>> You can start by looking at the vectors with -vec_view (there is code that 
>> you can view vectors explicitly in your code to get more control if this is 
>> too noisy)
> 
> The Vec are exactly the same (as reported by -ksp_view_rhs) in both cases.
> 
>> You could send us the diffs in ex111. We may be able to see something wrong.
> 
> I believe there is something weird inside KSPGMRES + PCFIELDSPLIT + right 
> preconditioning.

More precisely, this is the faulty line: 
https://petsc.org/main/src/ksp/ksp/impls/gmres/gmres.c.html#line119 
<https://petsc.org/main/src/ksp/ksp/impls/gmres/gmres.c.html#line119>
Here is the value of res with -nest_subvec false: 3.162278
With -nest_subvec true: 2.387784
The input Vec being
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
0.720032
0.061794
0.0100223
0.144639
0.397778
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1.
1.
1.
1.
1.
One can see that either VecNormalize() or VecNest is indeed broken.

Thanks,
Pierre

> Because if I switch to KSPHPDDM implementation of GMRES, then all is good.
> In the below traces, I switched VecSet(part of RHS,0.0) to VecSet(part of 
> RHS,1.0) in ex.c to make the error more visible.
> 
> Thanks,
> Pierre
> 
> $ ./ex -pc_type fieldsplit -ksp_converged_reason -fieldsplit_pc_type lu 
> -ksp_pc_side right -ksp_view_final_residual -nest_subvec false -ksp_view_rhs 
> -ksp_view_solution  
> Linear solve did not converge due to DIVERGED_BREAKDOWN iterations 30 < 
> garbage
> KSP final norm of residual 1.75544 < garbage
> Vec Object: 1 MPI process
>   type: nest
>   VecNest, rows=2,  structure: 
>   (0) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 0.720032
> 0.061794
> 0.0100223
> 0.144639
> 0.397778
>   (1) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 1.
> 1.
> 1.
> 1.
> 1.
> Vec Object: 1 MPI process
>   type: nest
>   VecNest, rows=2,  structure: 
>   (0) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 1.73518 < garbage
> 2.07583e-16
> -8.13135e-16
> -4.24749e-17
> 8.18337e-16
>   (1) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> -1.73518 < garbage
> -1.73518 < garbage
> -1.73518 < garbage
> -1.73518 < garbage
> -1.73518 < garbage
> 
> $ ./ex -pc_type fieldsplit -ksp_converged_reason -fieldsplit_pc_type lu 
> -ksp_pc_side right -ksp_view_final_residual -nest_subvec false -ksp_view_rhs 
> -ksp_view_solution -ksp_type hpddm
> Linear solve converged due to CONVERGED_RTOL iterations 1 < correct
> KSP final norm of residual 7.89955e-16
> Vec Object: 1 MPI process
>   type: nest
>   VecNest, rows=2,  structure: 
>   (0) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 0.720032
> 0.061794
> 0.0100223
> 0.144639
> 0.397778
>   (1) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 1.
> 1.
> 1.
> 1.
> 1.
> Vec Object: 1 MPI process
>   type: nest
>   VecNest, rows=2,  structure: 
>   (0) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 1. < correct
> 1.79841e-18
> -5.45696e-17
> 8.83987e-18
> 2.82354e-17
>   (1) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> -1. < correct
> -1. < correct
> -1. < correct
> -1. < correct
> -1. < correct
> 
> $ ./ex -pc_type fieldsplit -ksp_converged_reason -fieldsplit_pc_type lu 
> -ksp_pc_side right -ksp_view_final_residual -nest_subvec true -ksp_view_rhs 
> -ksp_view_solution
> Linear solve converged due to CONVERGED_RTOL iterations 1 < correct
> KSP final norm of residual 2.74152e-16
> Vec Object: 1 MPI process
>   type: nest
>   VecNest, rows=2,  structure: 
>   (0) : type=seq, rows=5 
> Vec Object: 1 MPI process
>   type: seq
> 0.720032
> 0.061794
> 0.0100223
> 0.14

Re: [petsc-dev] VecNest

[Pierre Jolivet]

> On 12 Jul 2022, at 2:04 PM, Mark Adams  wrote:
> 
> It looks like the RHS is zero in the 2nd case (0 KSP unpreconditioned resid 
> norm 0.e+00), but the true residual is the same.
> 
> It looks like you added "nest_subvec" to our example.

It’s a new (dumb) example (ex).

> You can start by looking at the vectors with -vec_view (there is code that 
> you can view vectors explicitly in your code to get more control if this is 
> too noisy)

The Vec are exactly the same (as reported by -ksp_view_rhs) in both cases.

> You could send us the diffs in ex111. We may be able to see something wrong.

I believe there is something weird inside KSPGMRES + PCFIELDSPLIT + right 
preconditioning.
Because if I switch to KSPHPDDM implementation of GMRES, then all is good.
In the below traces, I switched VecSet(part of RHS,0.0) to VecSet(part of 
RHS,1.0) in ex.c to make the error more visible.

Thanks,
Pierre

$ ./ex -pc_type fieldsplit -ksp_converged_reason -fieldsplit_pc_type lu 
-ksp_pc_side right -ksp_view_final_residual -nest_subvec false -ksp_view_rhs 
-ksp_view_solution  
Linear solve did not converge due to DIVERGED_BREAKDOWN iterations 30 < garbage
KSP final norm of residual 1.75544 < garbage
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
0.720032
0.061794
0.0100223
0.144639
0.397778
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1.
1.
1.
1.
1.
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1.73518 < garbage
2.07583e-16
-8.13135e-16
-4.24749e-17
8.18337e-16
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
-1.73518 < garbage
-1.73518 < garbage
-1.73518 < garbage
-1.73518 < garbage
-1.73518 < garbage

$ ./ex -pc_type fieldsplit -ksp_converged_reason -fieldsplit_pc_type lu 
-ksp_pc_side right -ksp_view_final_residual -nest_subvec false -ksp_view_rhs 
-ksp_view_solution -ksp_type hpddm
Linear solve converged due to CONVERGED_RTOL iterations 1 < correct
KSP final norm of residual 7.89955e-16
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
0.720032
0.061794
0.0100223
0.144639
0.397778
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1.
1.
1.
1.
1.
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1. < correct
1.79841e-18
-5.45696e-17
8.83987e-18
2.82354e-17
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
-1. < correct
-1. < correct
-1. < correct
-1. < correct
-1. < correct

$ ./ex -pc_type fieldsplit -ksp_converged_reason -fieldsplit_pc_type lu 
-ksp_pc_side right -ksp_view_final_residual -nest_subvec true -ksp_view_rhs 
-ksp_view_solution
Linear solve converged due to CONVERGED_RTOL iterations 1 < correct
KSP final norm of residual 2.74152e-16
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
0.720032
0.061794
0.0100223
0.144639
0.397778
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1.
1.
1.
1.
1.
Vec Object: 1 MPI process
  type: nest
  VecNest, rows=2,  structure: 
  (0) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
1. < correct
-3.75571e-17
2.51808e-16
7.69081e-18
-2.65477e-16
  (1) : type=seq, rows=5 
Vec Object: 1 MPI process
  type: seq
-1. < correct
-1. < correct
    -1. < correct
-1. < correct
-1. < correct

> Mark
> 
> On Mon, Jul 11, 2022 at 2:17 PM Pierre Jolivet  <mailto:pie...@joliv.et>> wrote:
> Hello,
> Could anyone help me understand what is going on in the following example, 
> please?
> I have a VecNest.
> I either: a) initialize all values to 0.0, then set a specific part of the 
> vector to nonzero or b) initialize a part of the vector to 0.0 and set the 
> other part to nonzero.
> I don’t see why a) and b) produce different results.
> 
> $ ./ex -pc_type fieldsplit -ksp_monitor_true_residual 
> -ksp_converged_reason -fieldsplit_pc_type jacobi -ksp_pc_side right 
> -ksp_view_final_residual -nest_subvec true 
>   0 KSP unpreconditioned resid norm 8.375635517980e-01 true resid norm 
> 8.375635517980e-01 ||r(i)||/||b|| 1.e+00
>   1 KSP unpreconditioned resid norm 4.748816884247e-01 true resid norm 
> 4.748816884247e-01 

[petsc-dev] VecNest

[Pierre Jolivet]

Hello,
Could anyone help me understand what is going on in the following example, 
please?
I have a VecNest.
I either: a) initialize all values to 0.0, then set a specific part of the 
vector to nonzero or b) initialize a part of the vector to 0.0 and set the 
other part to nonzero.
I don’t see why a) and b) produce different results.

$ ./ex -pc_type fieldsplit -ksp_monitor_true_residual -ksp_converged_reason 
-fieldsplit_pc_type jacobi -ksp_pc_side right -ksp_view_final_residual 
-nest_subvec true 
  0 KSP unpreconditioned resid norm 8.375635517980e-01 true resid norm 
8.375635517980e-01 ||r(i)||/||b|| 1.e+00
  1 KSP unpreconditioned resid norm 4.748816884247e-01 true resid norm 
4.748816884247e-01 ||r(i)||/||b|| 5.669798875623e-01
  2 KSP unpreconditioned resid norm 4.713006778679e-01 true resid norm 
4.713006778679e-01 ||r(i)||/||b|| 5.627043784990e-01
  3 KSP unpreconditioned resid norm 7.092979927129e-02 true resid norm 
7.092979927129e-02 ||r(i)||/||b|| 8.468587144106e-02
  4 KSP unpreconditioned resid norm 1.457836310255e-02 true resid norm 
1.457836310255e-02 ||r(i)||/||b|| 1.740567992870e-02
  5 KSP unpreconditioned resid norm 1.625040500524e-14 true resid norm 
1.633468028779e-14 ||r(i)||/||b|| 1.950261595401e-14
Linear solve converged due to CONVERGED_RTOL iterations 5
KSP final norm of residual 1.63347e-14
$ ./ex -pc_type fieldsplit -ksp_monitor_true_residual -ksp_converged_reason 
-fieldsplit_pc_type jacobi -ksp_pc_side right -ksp_view_final_residual 
-nest_subvec false
  0 KSP unpreconditioned resid norm 0.e+00 true resid norm 
8.375635517980e-01 ||r(i)||/||b||inf
Linear solve converged due to CONVERGED_ATOL iterations 0
KSP final norm of residual 0.837564

Thanks,
Pierre



ex.c
Description: Binary data

[petsc-dev] Hiptmair--Xu as PCCOMPOSITE

[Pierre Jolivet]

Hello,
Is anyone aware of an openly-available PETSc implementation of Hiptmair—Xu 
preconditioner, especially for problems in H(curl)?
For some applications, I’m currently using AMS from hypre, but for some others, 
it’s not applicable, e.g., --with-scalar-type=complex where GAMG could be used 
instead of BoomerAMG for inner solves.

Thanks,
Pierre

Re: [petsc-dev] new format error

[Pierre Jolivet]

> On 9 Jun 2022, at 4:33 PM, Mark Adams  wrote:
> 
> I rebased over master and now this branch fails source code check:
> 
> SETERRQ() with trailing newline --
> 1346 
> src/ksp/pc/impls/amgx/amgx.cxx:169:
>   SETERRQ(amgx->comm, PETSC_ERR_LIB, "%s\n", msg); \
> 
> 
> Any idea how to best fix this?

That’s pretty self-explanatory: remove the trailing \n in your error message, 
i.e., "%s".

Thanks,
Pierre

> Thanks
> 
> // XXX Need to add call in AmgX API that gracefully destroys everything
> // without abort etc.
> #define PetscCallAmgX(rc) do { \
>   AMGX_RC err = (rc); \
>   char msg[4096]; \
>   switch (err) { \
> case AMGX_RC_OK: \
>   break; \
> default: \
>   AMGX_get_error_string(err, msg, 4096); \
>   SETERRQ(amgx->comm, PETSC_ERR_LIB, "%s\n", msg); \
>   } \
> } while (0)
> 
> 
> 

[petsc-dev] argsearch or query='args' not returning the correct tests

[Pierre Jolivet]

Hello,
I’m not sure I understand why I get a discrepancy with those three queries:
$ make -f gmakefile print-test argsearch='st_share_sub_ksp'
ksp_ksp_tutorials-ex72_hpddm_gen_non_hermitian 
ksp_ksp_tutorials-ex72_hpddm_gen_non_hermitian_baij
ksp_ksp_tutorials-ex76_geneo_share_cholesky 
ksp_ksp_tutorials-ex76_geneo_share_lu
$ make -f gmakefile print-test argsearch='1_st_share_sub_ksp'
[blank]
$ make -f gmakefile.test print-test query='args' queryval='*_st_share_sub_ksp*'
ksp_ksp_tutorials-ex72_hpddm_gen_non_hermitian 
ksp_ksp_tutorials-ex72_hpddm_gen_non_hermitian_baij

The exact argument in those tests is -pc_hpddm_levels_1_st_share_sub_ksp, so 
only the first query returns the appropriate list.

Thanks,
Pierre

Re: [petsc-dev] CI fail

[Pierre Jolivet]

You need to rebase your branch, you are missing 
https://gitlab.com/petsc/petsc/-/commit/16494f630f598d90bd39b2bdce6fb796c5fd5072
 


Thanks,
Pierre

> On 7 Nov 2021, at 1:54 PM, Mark Adams  wrote:
> 
> The pipeline seems to be failing with something in Slepc
> https://gitlab.com/petsc/petsc/-/jobs/1756604732 
> 

Re: [petsc-dev] Anyone run STREAMS on an Apple M1 system?

[Pierre Jolivet]

To address your original question in case you didn’t get an answer in private, 
Barry, see infra what I get on a Macmini9,1.
I can send you an update when I get my hands on the newer MBP if you don’t have 
new batch of results by then.

Thanks,
Pierre 

/Users/jolivet/petsc/arch-darwin-c-opt/bin/mpicc -o MPIVersion.o -c -Wall 
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector 
-fno-stack-check -Qunused-arguments -fvisibility=hidden -Ofast
-I/Users/jolivet/petsc/include -I/Users/jolivet/petsc/arch-darwin-c-opt/include 
   `pwd`/MPIVersion.c
Running streams with '/Users/jolivet/petsc/arch-darwin-c-opt/bin/mpiexec ' 
using 'NPMAX=8'
1  58465.2862   Rate (MB/s)
2  57883.6350   Rate (MB/s) 0.990051
3  57538.6275   Rate (MB/s) 0.98415
4  57343.4226   Rate (MB/s) 0.980811
5  53210.3840   Rate (MB/s) 0.910119
6  50956.9355   Rate (MB/s) 0.871576
7  51621.7521   Rate (MB/s) 0.882947
8  48377.2931   Rate (MB/s) 0.827453

Traceback (most recent call last):
  File "process.py", line 89, in 
process(sys.argv[1],len(sys.argv)-2)
  File "process.py", line 33, in process
speedups[i] = triads[i]/triads[0]
TypeError: 'dict_values' object is not subscriptable
make[2]: [mpistream] Error 1 (ignored)
Traceback (most recent call last):
  File "process.py", line 89, in 
process(sys.argv[1],len(sys.argv)-2)
  File "process.py", line 33, in process
speedups[i] = triads[i]/triads[0]
TypeError: 'dict_values' object is not subscriptable
make[2]: [mpistreams] Error 1 (ignored)

> On 25 Oct 2021, at 10:52 PM, Jed Brown  wrote:
> 
> This shows 240 GB/s using 10 cores (8 performance + 2 efficiency) and 224 
> GB/s with 8 cores (as you'd most likely run HPC apps). Good, but far from the 
> theoretical 400 GB/s headline.
> 
> https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2
> 
> Barry Smith  writes:
> 
>>  Thanks, presumably we'll see the new Mac's there in a few days. BTW: the 
>> old streams benchmark page should point to this site; google is worthless.
>> 
>>  I get 24 on my Intel MacBook Pro and it also saturates with 1 core.
>> 
>>  Barry
>> 
>> 
>>> On Oct 18, 2021, at 4:48 PM, Jed Brown  wrote:
>>> 
>>> I don't have one, but this suggests it gets about 40 GB/s and can be 
>>> saturated by a single core. I believe it uses two channels of LPDDR4X-4266, 
>>> which has a theoretical peak of 68 GB/s.
>>> 
>>> https://browser.geekbench.com/v3/cpu/8931693
>>> 
>>> The press release claims up to 400 GB/s on the Max using DDR5. I assume 
>>> that's calculated based on 8 channels of LPDDR5-6400, which seems like a 
>>> surprisingly big step and I'm skeptical of what will actually be realized.
>>> 
>>> Barry Smith  writes:
>>> 
  Can anyone who owns an Apple M1 system run the MPI streams benchmark? 
 Make sure the -O3 (or something) optimization flags are turned on.
 
 Thanks
 
   Barry

[petsc-dev] PETSCTEST_VALGRIND

[Pierre Jolivet]

Hello,
I think I’m asking too much of test harness, but is there a way to 
automatically deactivate a test when running with VALGRIND=1 without having to 
play around with PETSC_RUNNING_ON_VALGRIND in the code?
I see the variable PETSCTEST_VALGRIND, but it seems rather static and set 
depending on PETSC_ARCH.
So even if there is src/snes/tutorials/ex19.c: requires: 
defined(PETSC_USE_LOG) !defined(PETSCTEST_VALGRIND), 
Running make -f gmakefile test s='snes*logviewmemory' VALGRIND=1 on my machine 
(without valgrind in the PETSC_ARCH) gets the test going.

Thanks,
Pierre

Re: [petsc-dev] AMGx integration

[Pierre Jolivet]

> On 19 Oct 2021, at 6:03 PM, Mark Adams  wrote:
> 
> Matt, (cc'ing Matt M.).
> Thanks, that looks pretty straight forward.
> 
> Matt M: I'm not very familiar with the configuration system. It might be best 
> for you to ask questions directly instead of me hacking this up.
> 
> mmg.py (appended) and AMReX.py are in 
> petsc/config/BuildSystem/config/packages. I'm guessing it will be easiest to 
> prune and clone AMReX.py (150 LOC)

I’d advise _not_ to use AMReX.py, as it is one of those weird packages that 
have self.builtafterpetsc = 1, and thus require a lot of unneeded LOC that you 
shouldn’t need for AMGx.
mmg.py has probably one of the lightest configure script indeed, alongside 
eigen.py (you’ll see they look similar, there is just the bare minimum).

Thanks,
Pierre

> I have rebased the branch over main and am testing it now.
> 
> Thanks,
> Mark
> 
> import config.package
> import os
> 
> class Configure(config.package.CMakePackage):
>   def __init__(self, framework):
> config.package.CMakePackage.__init__(self, framework)
> self.gitcommit= '5fc1b2c76fe1d6787387f5c9600759d4cbf26a88' # 
> jolivet/feature-mmg-install-3.16.0 sep-29-2021
> self.download = ['git://https://github.com/prj-/mmg.git 
> ','https://github.com/prj-/mmg/archive/'+self.gitcommit+'.tar.gz
>  ']
> self.versionname  = 'MMG_VERSION_RELEASE'
> self.includes = ['mmg/libmmg.h']
> self.liblist  = [['libmmg.a','libmmg3d.a']]
> self.functions= ['MMG5_paramUsage1']
> self.precisions   = ['double']
> return
> 
>   def setupDependencies(self, framework):
> config.package.CMakePackage.setupDependencies(self, framework)
> self.externalpackagesdir = 
> framework.require('PETSc.options.externalpackagesdir',self)
> self.compilerFlags = framework.require('config.compilerFlags',self)
> self.mathlib   = framework.require('config.packages.mathlib',self)
> self.ptscotch  = framework.require('config.packages.PTScotch',self)
> self.deps  = [self.mathlib,self.ptscotch]
> return
> 
>   def formCMakeConfigureArgs(self):
> args = config.package.CMakePackage.formCMakeConfigureArgs(self)
> args.append('-DUSE_ELAS=OFF')
> args.append('-DUSE_VTK=OFF')
> args.append('-DUSE_POINTMAP=ON')
> args.append('-DSCOTCH_DIR:STRING="'+self.ptscotch.directory+'"')
> return args
> 
>   def configureLibrary(self):
> config.package.CMakePackage.configureLibrary(self)
> 
> 
> On Tue, Oct 19, 2021 at 9:34 AM Matthew Knepley  > wrote:
> On Tue, Oct 19, 2021 at 9:32 AM Mark Adams  > wrote:
> cmake
> 
> barf. Clone one of those, like mmg.py or AMReX.py
> 
>   Thanks,
> 
> Matt
>  
> On Tue, Oct 19, 2021 at 9:26 AM Matthew Knepley  > wrote:
> On Tue, Oct 19, 2021 at 8:17 AM Mark Adams  > wrote:
> We (Matt Martineau and I) are ready to work on the AMGx integration and I 
> want to start by getting the AMGx download working.
> 
> I'm looking for advice on how to proceed.
> 
> Should I look at a simple example and clone it?
> 
> What build system does it use?
> 
>   Thanks,
> 
> Matt
>  
> Thanks,
> Mark
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] make test check-test-errors returning $? = 0

[Pierre Jolivet]

> On 23 Sep 2021, at 2:14 PM, Matthew Knepley  wrote:
> 
> On Thu, Sep 23, 2021 at 5:07 AM Pierre Jolivet  <mailto:pie...@joliv.et>> wrote:
> Hello,
> When a tests fails because the timeout limit is reached, check-test-errors 
> still returns a 0 exit code.
> Is this intended? Any way to bypass this and return something different than 
> 0, as it could return false negative in CI tests?
> 
> This is the intended behavior. We ignore timeouts, but report them.

Indeed, I see the failed test in the summary at the end of make test.
That’s no big deal, GitHub workers are slow (or maybe I should stay away from 
${DATAFILESPATH}/matrices/arco6), I’ll increase my timeout value for now.

Thanks,
Pierre

> I would not have a problem with a flag to convert these to errors, but I 
> would leave this default.
> 
>   Thanks,
> 
>  Matt
>  
> Thanks,
> Pierre
> 
> $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 
> 200" check-test-errors 2>&1 > /dev/null; echo $?
> make: *** [check-test-errors] Error 1
> 2
> $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 
> 200" check-test-errors TIMEOUT=1 2>&1 > /dev/null; echo $?
> 0
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

[petsc-dev] make test check-test-errors returning $? = 0

[Pierre Jolivet]

Hello,
When a tests fails because the timeout limit is reached, check-test-errors 
still returns a 0 exit code.
Is this intended? Any way to bypass this and return something different than 0, 
as it could return false negative in CI tests?

Thanks,
Pierre

$ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 
200" check-test-errors 2>&1 > /dev/null; echo $?
make: *** [check-test-errors] Error 1
2
$ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 
200" check-test-errors TIMEOUT=1 2>&1 > /dev/null; echo $?
0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small

[Pierre Jolivet]

> On 12 Sep 2021, at 8:56 PM, Matthew Knepley  wrote:
> 
> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter 
> mailto:sagit...@fedoraproject.org>> wrote:
> Those attached are configure.log/make.log from a MPI build in Fedora 34 
> x86_64 where the error below occurred.
> 
> This is OpenMPI 4.1.0. Is that the only MPI you build? My first inclination 
> is that this is an MPI implementation bug.
> 
> Junchao, do we have an OpenMPI build in the CI?

config/examples/arch-ci-linux-cuda-double-64idx.py:'--download-openmpi=1',
config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py:  
'--download-openmpi=1',
config/examples/arch-ci-linux-pkgs-opt.py:  '--download-openmpi=1',

config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as well.
I’m not sure PETSc is to blame here Antonio. You may want to try to ditch the 
OpenMPI shipped by your packet manager and try --download-openmpi as well, just 
for a quick sanity check.

Thanks,
Pierre

>   Thanks,
> 
>  Matt
>  
> On 9/12/21 19:18, Antonio T. sagitter wrote:
> > Okay. I will try to set correctly the DATAFILESPATH options.
> > 
> > I see even this error:
> > 
> > not ok 
> > vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create # 
> > Error code: 68
> > 
> > #PetscSF Object: 4 MPI processes
> > 
> > #  type: window
> > 
> > #  [0] Number of roots=3, leaves=2, remote ranks=2
> > 
> > #  [0] 0 <- (3,1)
> > 
> > #  [0] 1 <- (1,0)
> > 
> > #  [1] Number of roots=2, leaves=3, remote ranks=2
> > 
> > #  [1] 0 <- (0,1)
> > 
> > #  [1] 1 <- (2,0)
> > 
> > #  [1] 2 <- (0,2)
> > 
> > #  [2] Number of roots=2, leaves=3, remote ranks=3
> > 
> > #  [2] 0 <- (1,1)
> > 
> > #  [2] 1 <- (3,0)
> > 
> > #  [2] 2 <- (0,2)
> > 
> > #  [3] Number of roots=2, leaves=3, remote ranks=2
> > 
> > #  [3] 0 <- (2,1)
> > 
> > #  [3] 1 <- (0,0)
> > 
> > #  [3] 2 <- (0,2)
> > 
> > #  [0] Roots referenced by my leaves, by rank
> > 
> > #  [0] 1: 1 edges
> > 
> > #  [0]1 <- 0
> > 
> > #  [0] 3: 1 edges
> > 
> > #  [0]0 <- 1
> > 
> > #  [1] Roots referenced by my leaves, by rank
> > 
> > #  [1] 0: 2 edges
> > 
> > #  [1]0 <- 1
> > 
> > #  [1]2 <- 2
> > 
> > #  [1] 2: 1 edges
> > 
> > #  [1]1 <- 0
> > 
> > #  [2] Roots referenced by my leaves, by rank
> > 
> > #  [2] 0: 1 edges
> > 
> > #  [2]2 <- 2
> > 
> > #  [2] 1: 1 edges
> > 
> > #  [2]0 <- 1
> > 
> > #  [2] 3: 1 edges
> > 
> > #  [2]1 <- 0
> > 
> > #  [3] Roots referenced by my leaves, by rank
> > 
> > #  [3] 0: 2 edges
> > 
> > #  [3]1 <- 0
> > 
> > #  [3]2 <- 2
> > 
> > #  [3] 2: 1 edges
> > 
> > #  [3]0 <- 1
> > 
> > #  current flavor=CREATE synchronization=FENCE MultiSF sort=rank-order
> > 
> > #current info=MPI_INFO_NULL
> > 
> > #[buildhw-x86-09:1135574] *** An error occurred in MPI_Accumulate
> > 
> > #[buildhw-x86-09:1135574] *** reported by process [3562602497,3]
> > 
> > #[buildhw-x86-09:1135574] *** on win rdma window 4
> > 
> > #[buildhw-x86-09:1135574] *** MPI_ERR_RMA_RANGE: invalid RMA address 
> > range
> > 
> > #[buildhw-x86-09:1135574] *** MPI_ERRORS_ARE_FATAL (processes in 
> > this win will now abort,
> > 
> > #[buildhw-x86-09:1135574] ***and potentially your MPI job)
> > 
> > #[buildhw-x86-09.iad2.fedoraproject.org:1135567 
> > ] 3 more processes 
> > have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
> > 
> > #[buildhw-x86-09.iad2.fedoraproject.org:1135567 
> > ] Set MCA parameter 
> > "orte_base_help_aggregate" to 0 to see all help / error messages
> > 
> > Looks like an error related to OpenMPI-4*:
> > https://github.com/open-mpi/ompi/issues/6374 
> > 
> >  
> 
> -- 
> ---
> Antonio Trande
> Fedora Project
> mailto: sagit...@fedoraproject.org 
> GPG key: 0x29FBC85D7A51CC2F
> GPG key server: https://keyserver1.pgp.com/ 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small

[Pierre Jolivet]

Could you please forward configure.log to petsc-ma...@mcs.anl.gov 
<mailto:petsc-ma...@mcs.anl.gov>?
It’ll help the SF people figure out if it’s a false positive or if something 
needs fixing.

Thanks,
Pierre

> On 12 Sep 2021, at 7:33 PM, Antonio T. sagitter  
> wrote:
> 
> On 9/12/21 19:29, Pierre Jolivet wrote:
>> In your configure.log, I see --with-mpi=0, so I’m surprised this is even 
>> running, and even more surprised that you are encountering OpenMPI errors.
>> Is this from a different build?
> 
> Yes, sorry. This is another build with MPI.
> 
>> Thanks,
>> Pierre
>>> On 12 Sep 2021, at 7:18 PM, Antonio T. sagitter 
>>>  wrote:
>>> 
>>> Okay. I will try to set correctly the DATAFILESPATH options.
>>> 
>>> I see even this error:
>>> 
>>> not ok 
>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create # 
>>> Error code: 68
>>> 
>>> #   PetscSF Object: 4 MPI processes
>>> 
>>> # type: window
>>> 
>>> # [0] Number of roots=3, leaves=2, remote ranks=2
>>> 
>>> # [0] 0 <- (3,1)
>>> 
>>> # [0] 1 <- (1,0)
>>> 
>>> # [1] Number of roots=2, leaves=3, remote ranks=2
>>> 
>>> # [1] 0 <- (0,1)
>>> 
>>> # [1] 1 <- (2,0)
>>> 
>>> # [1] 2 <- (0,2)
>>> 
>>> # [2] Number of roots=2, leaves=3, remote ranks=3
>>> 
>>> # [2] 0 <- (1,1)
>>> 
>>> # [2] 1 <- (3,0)
>>> 
>>> # [2] 2 <- (0,2)
>>> 
>>> # [3] Number of roots=2, leaves=3, remote ranks=2
>>> 
>>> # [3] 0 <- (2,1)
>>> 
>>> # [3] 1 <- (0,0)
>>> 
>>> # [3] 2 <- (0,2)
>>> 
>>> # [0] Roots referenced by my leaves, by rank
>>> 
>>> # [0] 1: 1 edges
>>> 
>>> # [0]1 <- 0
>>> 
>>> # [0] 3: 1 edges
>>> 
>>> # [0]0 <- 1
>>> 
>>> # [1] Roots referenced by my leaves, by rank
>>> 
>>> # [1] 0: 2 edges
>>> 
>>> # [1]0 <- 1
>>> 
>>> # [1]2 <- 2
>>> 
>>> # [1] 2: 1 edges
>>> 
>>> # [1]1 <- 0
>>> 
>>> # [2] Roots referenced by my leaves, by rank
>>> 
>>> # [2] 0: 1 edges
>>> 
>>> # [2]2 <- 2
>>> 
>>> # [2] 1: 1 edges
>>> 
>>> # [2]0 <- 1
>>> 
>>> # [2] 3: 1 edges
>>> 
>>> # [2]1 <- 0
>>> 
>>> # [3] Roots referenced by my leaves, by rank
>>> 
>>> # [3] 0: 2 edges
>>> 
>>> # [3]1 <- 0
>>> 
>>> # [3]2 <- 2
>>> 
>>> # [3] 2: 1 edges
>>> 
>>> # [3]0 <- 1
>>> 
>>> # current flavor=CREATE synchronization=FENCE MultiSF sort=rank-order
>>> 
>>> #   current info=MPI_INFO_NULL
>>> 
>>> #   [buildhw-x86-09:1135574] *** An error occurred in MPI_Accumulate
>>> 
>>> #   [buildhw-x86-09:1135574] *** reported by process [3562602497,3]
>>> 
>>> #   [buildhw-x86-09:1135574] *** on win rdma window 4
>>> 
>>> #   [buildhw-x86-09:1135574] *** MPI_ERR_RMA_RANGE: invalid RMA address 
>>> range
>>> 
>>> #   [buildhw-x86-09:1135574] *** MPI_ERRORS_ARE_FATAL (processes in this 
>>> win will now abort,
>>> 
>>> #   [buildhw-x86-09:1135574] ***and potentially your MPI job)
>>> 
>>> #   [buildhw-x86-09.iad2.fedoraproject.org:1135567] 3 more processes have 
>>> sent help message help-mpi-errors.txt / mpi_errors_are_fatal
>>> 
>>> #   [buildhw-x86-09.iad2.fedoraproject.org:1135567] Set MCA parameter 
>>> "orte_base_help_aggregate" to 0 to see all help / error messages
>>> 
>>> Looks like an error related to OpenMPI-4*:
>>> https://github.com/open-mpi/ompi/issues/6374
>>> 
>>> Thank you.
>>> 
>>> On 9/12/21 18:09, Pierre Jolivet wrote:
>>>> Hello,
>>>> Did you copy the files from https://gitlab.com/petsc/datafiles 
>>>> <https://gitlab.com/petsc/datafiles> before the make check?
>>>> cfd.1.10 is not part of https://gitlab.com/petsc/petsc 
>>>> <https://gitlab.com/petsc/petsc>, so you either need to copy the files 
>>>> from the datafiles repository, or remove the 
>>>> --DATAFILESPATH=/home/sagitter/rpmbuild/BUILD/petsc-3.15.4/petsc-3.15.4/share/petsc/datafiles
>>>>  configure option.
>>>> Thanks,
>>>> Pierre
>>>>> On 12 Sep 2021, at 6:02 PM, Antonio T. sagitter 
>>>>> mailto:sagit...@fedoraproject.org>> wrote:
>>>>> 
>>>>> Hi all.
>>>>> 
>>>>> This error is repeated when the tests are compiled in PETSc-3.15.4 
>>>>> compilation:
>>>>> https://paste.in/MhdNjf <https://paste.in/MhdNjf>
>>>>> 
>>>>> This happens in Fedora 34 x86_64 with gcc-11.2.1
>>>>> 
>>>>> make.log and configure.log are attached
>>>>> -- 
>>>>> ---
>>>>> Antonio Trande
>>>>> Fedora Project
>>>>> mailto: sagit...@fedoraproject.org
>>>>> GPG key: 0x29FBC85D7A51CC2F
>>>>> GPG key server: https://keyserver1.pgp.com/
>>>>> 
>>> 
>>> -- 
>>> ---
>>> Antonio Trande
>>> Fedora Project
>>> mailto: sagit...@fedoraproject.org
>>> GPG key: 0x29FBC85D7A51CC2F
>>> GPG key server: https://keyserver1.pgp.com/
> 
> -- 
> ---
> Antonio Trande
> Fedora Project
> mailto: sagit...@fedoraproject.org
> GPG key: 0x29FBC85D7A51CC2F
> GPG key server: https://keyserver1.pgp.com/

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small

[Pierre Jolivet]

In your configure.log, I see --with-mpi=0, so I’m surprised this is even 
running, and even more surprised that you are encountering OpenMPI errors.
Is this from a different build?

Thanks,
Pierre

> On 12 Sep 2021, at 7:18 PM, Antonio T. sagitter  
> wrote:
> 
> Okay. I will try to set correctly the DATAFILESPATH options.
> 
> I see even this error:
> 
> not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create 
> # Error code: 68
> 
> # PetscSF Object: 4 MPI processes
> 
> #   type: window
> 
> #   [0] Number of roots=3, leaves=2, remote ranks=2
> 
> #   [0] 0 <- (3,1)
> 
> #   [0] 1 <- (1,0)
> 
> #   [1] Number of roots=2, leaves=3, remote ranks=2
> 
> #   [1] 0 <- (0,1)
> 
> #   [1] 1 <- (2,0)
> 
> #   [1] 2 <- (0,2)
> 
> #   [2] Number of roots=2, leaves=3, remote ranks=3
> 
> #   [2] 0 <- (1,1)
> 
> #   [2] 1 <- (3,0)
> 
> #   [2] 2 <- (0,2)
> 
> #   [3] Number of roots=2, leaves=3, remote ranks=2
> 
> #   [3] 0 <- (2,1)
> 
> #   [3] 1 <- (0,0)
> 
> #   [3] 2 <- (0,2)
> 
> #   [0] Roots referenced by my leaves, by rank
> 
> #   [0] 1: 1 edges
> 
> #   [0]1 <- 0
> 
> #   [0] 3: 1 edges
> 
> #   [0]0 <- 1
> 
> #   [1] Roots referenced by my leaves, by rank
> 
> #   [1] 0: 2 edges
> 
> #   [1]0 <- 1
> 
> #   [1]2 <- 2
> 
> #   [1] 2: 1 edges
> 
> #   [1]1 <- 0
> 
> #   [2] Roots referenced by my leaves, by rank
> 
> #   [2] 0: 1 edges
> 
> #   [2]2 <- 2
> 
> #   [2] 1: 1 edges
> 
> #   [2]0 <- 1
> 
> #   [2] 3: 1 edges
> 
> #   [2]1 <- 0
> 
> #   [3] Roots referenced by my leaves, by rank
> 
> #   [3] 0: 2 edges
> 
> #   [3]1 <- 0
> 
> #   [3]2 <- 2
> 
> #   [3] 2: 1 edges
> 
> #   [3]0 <- 1
> 
> #   current flavor=CREATE synchronization=FENCE MultiSF sort=rank-order
> 
> # current info=MPI_INFO_NULL
> 
> # [buildhw-x86-09:1135574] *** An error occurred in MPI_Accumulate
> 
> # [buildhw-x86-09:1135574] *** reported by process [3562602497,3]
> 
> # [buildhw-x86-09:1135574] *** on win rdma window 4
> 
> # [buildhw-x86-09:1135574] *** MPI_ERR_RMA_RANGE: invalid RMA address 
> range
> 
> # [buildhw-x86-09:1135574] *** MPI_ERRORS_ARE_FATAL (processes in this 
> win will now abort,
> 
> # [buildhw-x86-09:1135574] ***and potentially your MPI job)
> 
> # [buildhw-x86-09.iad2.fedoraproject.org:1135567] 3 more processes have 
> sent help message help-mpi-errors.txt / mpi_errors_are_fatal
> 
> # [buildhw-x86-09.iad2.fedoraproject.org:1135567] Set MCA parameter 
> "orte_base_help_aggregate" to 0 to see all help / error messages
> 
> Looks like an error related to OpenMPI-4*:
> https://github.com/open-mpi/ompi/issues/6374
> 
> Thank you.
> 
> On 9/12/21 18:09, Pierre Jolivet wrote:
>> Hello,
>> Did you copy the files from https://gitlab.com/petsc/datafiles 
>> <https://gitlab.com/petsc/datafiles> before the make check?
>> cfd.1.10 is not part of https://gitlab.com/petsc/petsc 
>> <https://gitlab.com/petsc/petsc>, so you either need to copy the files from 
>> the datafiles repository, or remove the 
>> --DATAFILESPATH=/home/sagitter/rpmbuild/BUILD/petsc-3.15.4/petsc-3.15.4/share/petsc/datafiles
>>  configure option.
>> Thanks,
>> Pierre
>>> On 12 Sep 2021, at 6:02 PM, Antonio T. sagitter >> <mailto:sagit...@fedoraproject.org>> wrote:
>>> 
>>> Hi all.
>>> 
>>> This error is repeated when the tests are compiled in PETSc-3.15.4 
>>> compilation:
>>> https://paste.in/MhdNjf <https://paste.in/MhdNjf>
>>> 
>>> This happens in Fedora 34 x86_64 with gcc-11.2.1
>>> 
>>> make.log and configure.log are attached
>>> -- 
>>> ---
>>> Antonio Trande
>>> Fedora Project
>>> mailto: sagit...@fedoraproject.org
>>> GPG key: 0x29FBC85D7A51CC2F
>>> GPG key server: https://keyserver1.pgp.com/
>>> 
> 
> -- 
> ---
> Antonio Trande
> Fedora Project
> mailto: sagit...@fedoraproject.org
> GPG key: 0x29FBC85D7A51CC2F
> GPG key server: https://keyserver1.pgp.com/

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small

[Pierre Jolivet]

Hello,
Did you copy the files from https://gitlab.com/petsc/datafiles 
 before the make check?
cfd.1.10 is not part of https://gitlab.com/petsc/petsc 
, so you either need to copy the files from the 
datafiles repository, or remove the 
--DATAFILESPATH=/home/sagitter/rpmbuild/BUILD/petsc-3.15.4/petsc-3.15.4/share/petsc/datafiles
 configure option.

Thanks,
Pierre

> On 12 Sep 2021, at 6:02 PM, Antonio T. sagitter  
> wrote:
> 
> Hi all.
> 
> This error is repeated when the tests are compiled in PETSc-3.15.4 
> compilation:
> https://paste.in/MhdNjf
> 
> This happens in Fedora 34 x86_64 with gcc-11.2.1
> 
> make.log and configure.log are attached
> -- 
> ---
> Antonio Trande
> Fedora Project
> mailto: sagit...@fedoraproject.org
> GPG key: 0x29FBC85D7A51CC2F
> GPG key server: https://keyserver1.pgp.com/
> 

Re: [petsc-dev] Is this a bug in test generation?

[Pierre Jolivet]

That’s a long-standing issue that I personally bypass by setting the suffix in 
the test instead of at the testset scope.
But maybe that’s not working for you either?

Thanks,
Pierre

> On 20 Aug 2021, at 12:43 PM, Stefano Zampini  
> wrote:
> 
> Scott
> 
> This test is specified as
> 
>testset:
>   suffix: expl
>   nsize: {{1 2}}
>   filter: grep -v "MPI processes" | grep -v " type:" | grep -v "Mat 
> Object"
>   args: -ksp_converged_reason -view_explicit_mat -pc_type none -ksp_type 
> {{cg gmres}}
>   test:
> args: -mat_type aij
>   test:
> requires: hypre
> args: -mat_type hypre
> 
> It triggers an error here https://gitlab.com/petsc/petsc/-/jobs/1519558357 
> , with a  CI configuration 
> without hypre.
> Other testset instances works fine if any “require:" is both specified in the 
> testset body and in the specific test.
> Is this supposed to work?
> 
> Thanks
> Stefano

[petsc-dev] PETSC_COMM_WORLD not a PETSc communicator

[Pierre Jolivet]

Hello,
Is there a specific reason why PETSC_COMM_WORLD is not a PETSc communicator, 
i.e., has no Petsc_Counter_keyval attached?
ierr = PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,);CHKERRQ(ierr);
yields
[0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc communicator
[0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60
[0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340

Thanks,
Pierre

[petsc-dev] -ksp_error_if_not_converged

[Pierre Jolivet]

Hello,
I’d like to trigger an error if a linear solve does not converge due to 
DIVERGED_ITS.
Since this MR https://gitlab.com/petsc/petsc/-/merge_requests/1294 
, no error is triggered 
anymore if ksp->reason == KSP_DIVERGED_ITS.
This seems to be inconsistent with -snes_error_if_not_converged.
Is there a workaround? Shouldn’t this be clarified in the manual page?

Thank you,
Pierre

$ src/ksp/ksp/tutorials/ex1 -ksp_max_it 1 -ksp_converged_reason 
-ksp_error_if_not_converged && echo "No error"
Linear solve did not converge due to DIVERGED_ITS iterations 1
KSP Object: 1 MPI processes
  type: gmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization 
with no iterative refinement
happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: jacobi
type DIAGONAL
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
type: seqaij
rows=10, cols=10
total: nonzeros=28, allocated nonzeros=50
total number of mallocs used during MatSetValues calls=0
  not using I-node routines
Norm of error 2.95296, Iterations 1
No error

Re: [petsc-dev] Broken snes_tutorials-ex20_2d_p1_gmg_vcycle_cr and CRSetup_Private()

[Pierre Jolivet]

I’ve flagged the test as broken on top of my feature branch 
(https://gitlab.com/petsc/petsc/-/commit/b02fd907789ed54b7deef6e9395f48df00996858?merge_request_iid=4182
 
<https://gitlab.com/petsc/petsc/-/commit/b02fd907789ed54b7deef6e9395f48df00996858?merge_request_iid=4182>).
Satish will have to make the executive decision of merging this or not, as the 
pipeline is now successful.

Thanks,
Pierre

> On 4 Aug 2021, at 2:50 PM, Stefano Zampini  wrote:
> 
> Matt,
> 
> I’m fine flagging the test as broken for now
> Actually, this test has always been broken….
> As Pierre said, if we make these matrices inherit from MatShell, we get shift 
> and scale for free
> 
> 
>> On Aug 4, 2021, at 10:13 AM, Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> 
>> Hello,
>> Up until very recently, MatGetOwnershipRange() was returning only zeros for 
>> any MATNORMAL (thanks Stefano for the help fixing that 
>> https://gitlab.com/petsc/petsc/-/merge_requests/4182#note_640848528 
>> <https://gitlab.com/petsc/petsc/-/merge_requests/4182#note_640848528>).
>> In CRSetup_Private(), there is a call to MatCreateNormal() followed by 
>> MatShift().
>> Since ops->shift = NULL for MATNORMAL, code goes into MatShift_Basic().
>> There, a for loop goes from “start” to “end”, but the values were previously 
>> only 0.
>> Now that MatGetOwnershipRange() has been fixed for MATNORMAL, the bounds are 
>> nonzero, and snes_tutorials-ex20_2d_p1_gmg_vcycle_cr fails because MATNORMAL 
>> has no ops->setvalues
>> What is the proper fix or how can I bypass the issue? Would it be OK to 
>> (temporarily) mark this test as broken?
>> Previous code was obviously not doing what was intended in theory, but yet 
>> I’m guessing numerically everything was OK?
>> I’m thinking https://gitlab.com/petsc/petsc/-/issues/972 
>> <https://gitlab.com/petsc/petsc/-/issues/972> should do the trick, but this 
>> is somehow orthogonal to my developments, which are now blocked because of 
>> failing pipelines in snes_tutorials-ex20_2d_p1_gmg_vcycle_cr 
>> (https://gitlab.com/petsc/petsc/-/jobs/1475595231 
>> <https://gitlab.com/petsc/petsc/-/jobs/1475595231>).
>> 
>> Thanks,
>> Pierre
>> 
>> $ make -f gmakefile test query='name' 
>> queryval="*snes_tutorials-ex20_2d_p1_gmg_vcycle_cr*"
>> Using MAKEFLAGS: queryval=*snes_tutorials-ex20_2d_p1_gmg_vcycle_cr* 
>> query=name
>> TEST 
>> arch-darwin-c-debug-real/tests/counts/snes_tutorials-ex20_2d_p1_gmg_vcycle_cr.counts
>> not ok snes_tutorials-ex20_2d_p1_gmg_vcycle_cr # Error code: 56
>> #[0]PETSC ERROR: - Error Message 
>> --
>> #[0]PETSC ERROR: No support for this operation for this object type
>> #[0]PETSC ERROR: Mat type normal
>> #[0]PETSC ERROR: See 
>> https://www.mcs.anl.gov/petsc/documentation/faq.html 
>> <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>> #[0]PETSC ERROR: Petsc Development GIT revision: v3.15.2-633-gdc9bbe6e6a 
>>  GIT Date: 2021-08-03 20:53:25 +0200
>> #[0]PETSC ERROR: ../ex20 on a arch-darwin-c-debug-real named 
>> aardvark.local by jolivet Wed Aug  4 10:04:59 2021
>> #[0]PETSC ERROR: Configure options --download-c2html 
>> --download-elemental --download-hpddm --download-hpddm-commit=origin/omp 
>> --download-htool --download-htool-commit=origin/main --download-hypre 
>> --download-metis --download-suitesparse --download-mmg --download-mumps 
>> --download-hwloc --download-p4est --download-parmetis --download-parmmg 
>> --download-ptscotch --download-slepc --download-slepc-commit=origin/main 
>> --download-slepc-configure-arguments="--download-arpack=https://github.com/prj-/arpack-ng/archive/b64dccb.tar.gz
>>  <https://github.com/prj-/arpack-ng/archive/b64dccb.tar.gz>" 
>> --download-tetgen --download-triangle --download-zlib 
>> --with-blaslapack-include=/opt/intel/oneapi/mkl/latest/include 
>> --with-blaslapack-lib="[/opt/intel/oneapi/compiler/latest/mac/compiler/lib/libiomp5.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_thread.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.dylib]"
>>  --with-fortran-bindings=1 
>> --with-mkl_cpardiso-include=/opt/intel/oneapi/mkl/latest/include 
>> --with-mkl_cpardiso=1 
>> --with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include 
>> --with-mkl_pardiso=1 --with-opengl=1 --with-petsc4py 
>> --with-scalapack-include=/opt/intel/oneapi/mkl/latest/includ

[petsc-dev] Broken snes_tutorials-ex20_2d_p1_gmg_vcycle_cr and CRSetup_Private()

[Pierre Jolivet]

Hello,
Up until very recently, MatGetOwnershipRange() was returning only zeros for any 
MATNORMAL (thanks Stefano for the help fixing that 
https://gitlab.com/petsc/petsc/-/merge_requests/4182#note_640848528 
).
In CRSetup_Private(), there is a call to MatCreateNormal() followed by 
MatShift().
Since ops->shift = NULL for MATNORMAL, code goes into MatShift_Basic().
There, a for loop goes from “start” to “end”, but the values were previously 
only 0.
Now that MatGetOwnershipRange() has been fixed for MATNORMAL, the bounds are 
nonzero, and snes_tutorials-ex20_2d_p1_gmg_vcycle_cr fails because MATNORMAL 
has no ops->setvalues
What is the proper fix or how can I bypass the issue? Would it be OK to 
(temporarily) mark this test as broken?
Previous code was obviously not doing what was intended in theory, but yet I’m 
guessing numerically everything was OK?
I’m thinking https://gitlab.com/petsc/petsc/-/issues/972 
 should do the trick, but this is 
somehow orthogonal to my developments, which are now blocked because of failing 
pipelines in snes_tutorials-ex20_2d_p1_gmg_vcycle_cr 
(https://gitlab.com/petsc/petsc/-/jobs/1475595231 
).

Thanks,
Pierre

$ make -f gmakefile test query='name' 
queryval="*snes_tutorials-ex20_2d_p1_gmg_vcycle_cr*"
Using MAKEFLAGS: queryval=*snes_tutorials-ex20_2d_p1_gmg_vcycle_cr* query=name
TEST 
arch-darwin-c-debug-real/tests/counts/snes_tutorials-ex20_2d_p1_gmg_vcycle_cr.counts
not ok snes_tutorials-ex20_2d_p1_gmg_vcycle_cr # Error code: 56
#   [0]PETSC ERROR: - Error Message 
--
#   [0]PETSC ERROR: No support for this operation for this object type
#   [0]PETSC ERROR: Mat type normal
#   [0]PETSC ERROR: See 
https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#   [0]PETSC ERROR: Petsc Development GIT revision: v3.15.2-633-gdc9bbe6e6a 
 GIT Date: 2021-08-03 20:53:25 +0200
#   [0]PETSC ERROR: ../ex20 on a arch-darwin-c-debug-real named 
aardvark.local by jolivet Wed Aug  4 10:04:59 2021
#   [0]PETSC ERROR: Configure options --download-c2html 
--download-elemental --download-hpddm --download-hpddm-commit=origin/omp 
--download-htool --download-htool-commit=origin/main --download-hypre 
--download-metis --download-suitesparse --download-mmg --download-mumps 
--download-hwloc --download-p4est --download-parmetis --download-parmmg 
--download-ptscotch --download-slepc --download-slepc-commit=origin/main 
--download-slepc-configure-arguments="--download-arpack=https://github.com/prj-/arpack-ng/archive/b64dccb.tar.gz;
 --download-tetgen --download-triangle --download-zlib 
--with-blaslapack-include=/opt/intel/oneapi/mkl/latest/include 
--with-blaslapack-lib="[/opt/intel/oneapi/compiler/latest/mac/compiler/lib/libiomp5.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_thread.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.dylib]"
 --with-fortran-bindings=1 
--with-mkl_cpardiso-include=/opt/intel/oneapi/mkl/latest/include 
--with-mkl_cpardiso=1 
--with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include 
--with-mkl_pardiso=1 --with-opengl=1 --with-petsc4py 
--with-scalapack-include=/opt/intel/oneapi/mkl/latest/include 
--with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_scalapack_lp64.dylib,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_mpich_lp64.dylib]"
 --with-scalar-type=real --with-shared-libraries=1 CFLAGS="-Wall -g" 
CXXFLAGS="-Wall -g" FFLAGS="-Wall -ffree-line-length-512 -g" 
PETSC_ARCH=arch-darwin-c-debug-real
#   [0]PETSC ERROR: #1 MatSetValues() at 
/Users/jolivet/Documents/repositories/petsc/src/mat/interface/matrix.c:1376
#   [0]PETSC ERROR: #2 MatShift_Basic() at 
/Users/jolivet/Documents/repositories/petsc/src/mat/utils/gcreate.c:26
#   [0]PETSC ERROR: #3 MatShift() at 
/Users/jolivet/Documents/repositories/petsc/src/mat/utils/axpy.c:309
#   [0]PETSC ERROR: #4 CRSetup_Private() at 
/Users/jolivet/Documents/repositories/petsc/src/ksp/pc/impls/mg/mg.c:300
#   [0]PETSC ERROR: #5 PCSetUp_Shell() at 
/Users/jolivet/Documents/repositories/petsc/src/ksp/pc/impls/shell/shellpc.c:101
#   [0]PETSC ERROR: #6 PCSetUp() at 
/Users/jolivet/Documents/repositories/petsc/src/ksp/pc/interface/precon.c:1016
#   [0]PETSC ERROR: #7 PCApply() at 
/Users/jolivet/Documents/repositories/petsc/src/ksp/pc/interface/precon.c:441
#   [0]PETSC ERROR: #8 PCApply_Composite_Multiplicative() at 
/Users/jolivet/Documents/repositories/petsc/src/ksp/pc/impls/composite/composite.c:50
#   [0]PETSC ERROR: #9 PCApply() at 
/Users/jolivet/Documents/repositories/petsc/src/ksp/pc/interface/precon.c:445
#   [0]PETSC ERROR: #10 KSP_PCApply() at 
/Users/jolivet/repo/petsc/include/petsc/private/kspimpl.h:365

[petsc-dev] Missing petsc4py bindings

[Pierre Jolivet]

Hello,
I’d like to repartition a MATMPIAIJ in petsc4py, is there an equivalent of:
MatPartitioning mpart;
ierr = MatPartitioningCreate(PETSC_COMM_WORLD,);CHKERRQ(ierr);
ierr = MatPartitioningSetAdjacency(mpart,B);CHKERRQ(ierr);
ierr = MatPartitioningSetFromOptions(mpart);CHKERRQ(ierr);
ierr = MatPartitioningApply(mpart,);CHKERRQ(ierr);
ierr = MatPartitioningDestroy();CHKERRQ(ierr);
ierr = ISBuildTwoSided(is,NULL,);CHKERRQ(ierr);
I can’t find any reference to MatPartitioning or ISBuildTwoSided in 
src/binding/petsc4py

Thanks,
Pierre

Re: [petsc-dev] -info from just one process?

[Pierre Jolivet]

> On 22 Apr 2021, at 7:45 PM, Barry Smith  wrote:
> 
> 
>   It would be straightforward to add an option to -info to output only from a 
> particular MPI_COMM_WORLD global rank.
> 
>   Maybe an extra :rank at the end of the info options?
> 
>   Open 128k files is nuts.

It is what it is, I’ve made peace with the fact that I can’t expect any 
meaningful timings for large runs when this option is on.
As a side note, one would expect that this kind of behavior is consistent 
across the library.
But in fact, it isn’t. If you do something like -sub_ksp_view_pmat file with, 
for example, DD preconditioners, you end up with a single file with all local 
matrices scrambled in.
It would be nice to have the same logic, i.e., file.%p, and be able to filter 
ranks that actually print to file like you suggest.
These kind of options are less critical than -info so I can just insert them in 
the option database after initialization, if rank == view_this_rank 
PetscOptionsInsertString(), but doing that using command line arguments would 
be easier and more elegant, IMO. 

Thanks,
Pierre

>   Barry
> 
>> On Apr 22, 2021, at 11:14 AM, Mark Adams > <mailto:mfad...@lbl.gov>> wrote:
>> 
>> OK, so -info out.%p would do what I asked. Alas, what I really need it to 
>> reduce the output with 128K processes.
>> Thanks,
>> 
>> On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> 
>> 
>>> On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch >> <mailto:jacob@gmail.com>> wrote:
>>> 
>>> Not out of the box no. There is -info ::self which makes info only print 
>>> from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. 
>>> Best advice would be to print to a file and grep the file for any lines 
>>> starting with the rank number you want.
>> 
>> There is no need for grep, IMHO, just do -info file and then open file.%p
>> 
>> Thanks,
>> Pierre
>> 
>>> Best regards,
>>> 
>>> Jacob Faibussowitsch
>>> (Jacob Fai - booss - oh - vitch)
>>> 
>>>> On Apr 22, 2021, at 10:16, Mark Adams >>> <mailto:mfad...@lbl.gov>> wrote:
>>>> 
>>>> Can I make -info only print from one process?
>>>> Mark
>>> 
>> 
> 

Re: [petsc-dev] -info from just one process?

[Pierre Jolivet]

> On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch  wrote:
> 
> Not out of the box no. There is -info ::self which makes info only print from 
> calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. Best 
> advice would be to print to a file and grep the file for any lines starting 
> with the rank number you want.

There is no need for grep, IMHO, just do -info file and then open file.%p

Thanks,
Pierre

> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> 
>> On Apr 22, 2021, at 10:16, Mark Adams > > wrote:
>> 
>> Can I make -info only print from one process?
>> Mark
> 

Re: [petsc-dev] Fortran macOS Linkage Warning

[Pierre Jolivet]

> On 16 Apr 2021, at 10:03 PM, Barry Smith  wrote:
> 
> 
>I am not sure what the point is here. Are you unable to configure? or make 
> all? or make check PETSc? 

I’m unable to configure.
$ git clone https://gitlab.com/petsc/petsc
Cloning into 'petsc'...
warning: redirecting to https://gitlab.com/petsc/petsc.git/
remote: Enumerating objects: 64141, done.
remote: Counting objects: 100% (64141/64141), done.
remote: Compressing objects: 100% (7098/7098), done.
remote: Total 939408 (delta 62202), reused 57532 (delta 57038), pack-reused 
875267
Receiving objects: 100% (939408/939408), 267.66 MiB | 1.07 MiB/s, done.
Resolving deltas: 100% (720467/720467), done.
Updating files: 100% (9849/9849), done.
$ cd petsc
$ export PETSC_DIR=`pwd`
$ ./configure
===
 Configuring PETSc to compile on your system
===
TESTING: checkFortranLibraries from 
config.compilers(config/BuildSystem/config/compilers.py:921)  
***
 UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
details):
---
Fortran libraries cannot be used with C as linker
***
configure.log is there: http://joliv.et/configure_broken_fortran_filter.log 
<http://joliv.et/configure_broken_fortran_filter.log>
Workaround is: MACOSX_DEPLOYMENT_TARGET=11.2 ./configure 
FFLAGS="-mmacosx-version-min=11.2"
I tried setting one and not the other additional flag but I respectively had 
failures re-compiling ParMETIS or Elemental.

Thanks,
Pierre

> 
>If so that should clearly be reported as a bug with all relevant 
> information supplied. 
> 
>No, we have never and do not ever intend for PETSc users even developers 
> to know or care about MACOSX_DEPLOYMENT_TARGET
> 
>Anyways if you are reporting a bug  and I am guess the bug correctly the 
> fix is to add "was built for newer macOS version" to the list of filtered 
> link warnings. There will be a bunch of such warnings that will need to be 
> added to the filtering in the next few months but once they are in place the 
> filtering should not be an issue.
> 
> 
>Barry
> 
> 
> 
>> On Apr 16, 2021, at 6:20 AM, Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> 
>> Barry,
>> That issue + this commit:
>> $ git bisect bad
>> 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8 is the first bad commit
>> commit 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8
>> Author: Barry Smith mailto:bsm...@mcs.anl.gov>>
>> Date:   Sat Apr 10 13:40:10 2021 -0500
>> makes it actually impossible to ./configure on macOS.
>> Do we really expect all macOS users to know about the variable 
>> MACOSX_DEPLOYMENT_TARGET?
>> 
>> Thanks,
>> Pierre
>> 
>> $ ./configure
>> ===
>>  Configuring PETSc to compile on your system
>> ===
>> TESTING: checkFortranLibraries from 
>> config.compilers(config/BuildSystem/config/compilers.py:921) 
>> 
>>  
>> ***
>>  UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
>> details):
>> ---
>> Fortran libraries cannot be used with C as linker
>> ***
>> 
>> Executing: mpicc  -o 
>> /var/folders/_9/0qn1m4z92wzbhz9434vx39fcgn/T/petsc-bhav6feq/config.setCompilers/conftest
>>   -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
>> -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first 
>> -Wl,-no_compact_unwind  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing 
>> -Wno-unknown-pragmas -fstack-protector -fno-stack-check -Qunused-arguments 
>> -fvisibility=hidden -g3 
>> /var/folders/_9/0qn1m4z92wzbhz9434vx39fcgn/T/petsc-bhav6feq/config.setCompilers/conftest.o
>>  -lc++ -ldl  -lc++ -ldl -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib 
>> -L/usr/local/Cellar/mpich/3.4.1_1/lib -lmpifort -lmpi -lpmpi -lgfortran 
>> -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0
>>  
>> -L/usr/local/Cellar/gcc/10.2.0_4/l

Re: [petsc-dev] Fortran macOS Linkage Warning

[Pierre Jolivet]

Barry,
That issue + this commit:
$ git bisect bad
36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8 is the first bad commit
commit 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8
Author: Barry Smith 
Date:   Sat Apr 10 13:40:10 2021 -0500
makes it actually impossible to ./configure on macOS.
Do we really expect all macOS users to know about the variable 
MACOSX_DEPLOYMENT_TARGET?

Thanks,
Pierre

$ ./configure
===
 Configuring PETSc to compile on your system
===
TESTING: checkFortranLibraries from 
config.compilers(config/BuildSystem/config/compilers.py:921)
 
 ***
 UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
details):
---
Fortran libraries cannot be used with C as linker
***

Executing: mpicc  -o 
/var/folders/_9/0qn1m4z92wzbhz9434vx39fcgn/T/petsc-bhav6feq/config.setCompilers/conftest
  -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
-Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first 
-Wl,-no_compact_unwind  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing 
-Wno-unknown-pragmas -fstack-protector -fno-stack-check -Qunused-arguments 
-fvisibility=hidden -g3 
/var/folders/_9/0qn1m4z92wzbhz9434vx39fcgn/T/petsc-bhav6feq/config.setCompilers/conftest.o
 -lc++ -ldl  -lc++ -ldl -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib 
-L/usr/local/Cellar/mpich/3.4.1_1/lib -lmpifort -lmpi -lpmpi -lgfortran 
-Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0
 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 
-Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 
-L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -lgfortran -lquadmath -lm
Possible ERROR while running linker:
stderr:
ld: warning: dylib (/usr/local/Cellar/mpich/3.4.1_1/lib/libmpifort.dylib) was 
built for newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib 
(/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgfortran.dylib) was built for 
newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib 
(/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.dylib) was built for 
newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib 
(/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.0.dylib) was built for 
newer macOS version (11.2) than being linked (11.0)
  compilers: Cannot compile/link C with mpicc.

> On 12 Mar 2021, at 6:16 PM, Barry Smith  wrote:
> 
> 
>   You should be able to edit lib/petsc/conf/rules and add it to the filter 
> for .cu.PETSc .kokkos.cxx.PETSc .c.PETSc .cxx.PETSc:  and .F.PETSc .F90.PETSc:
> 
>   I think it is fine to hide this warning; it is impossible for me to 
> understand anyway :-(.
> 
>Barry
> 
> 
> 
> 
>> On Mar 12, 2021, at 2:44 AM, Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> 
>> Sorry to dig up this old thread, but this is triggering false positives 
>> while make checking for people who don’t want to toy around with 
>> MACOSX_DEPLOYMENT_TARGET.
>> Could/should we filter this somehow, or have BuildSystem tinker with the 
>> environment variable itself?
>> 
>> Thanks,
>> Pierre
>> 
>> $ make check
>> Running check examples to verify correct installation
>> Using PETSC_DIR=/Users/jolivet/repo/petsc and 
>> PETSC_ARCH=arch-darwin-c-debug-real
>> ***Error detected during compile or link!***
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html 
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html>
>> /Users/jolivet/repo/petsc/src/snes/tutorials ex19
>> *
>> mpicc -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
>> -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first 
>> -Wl,-no_compact_unwind -Wall -g  -Wall -g
>> -I/Users/jolivet/repo/petsc/include 
>> -I/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/include 
>> -I/opt/intel/mkl/include ex19.c  
>> -Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
>> -L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
>> -Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
>> -L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
>> -Wl,-rpath,/opt/intel/mkl/lib -L/opt/intel/mkl/lib -Wl,-rpath,/opt/intel/lib 
>> -L/opt/intel/lib -Wl,-rpath,

Re: [petsc-dev] Regression in manualpages generation?

[Pierre Jolivet]

> On 6 Apr 2021, at 5:03 PM, Satish Balay  wrote:
> 
> 
> https://www.mcs.anl.gov/petsc/petsc-current/ is from petsc-3.15.0 -
> and is normally gets updated with the next patch release - i.e
> petsc-3.15.1. [ok - I'll update now with current release snapshot]

Ah, of course, I don’t know why I checked there instead of the dev version, my 
bad.
Well, thank you for updating the 3.15.0 pages as well I guess.

Thanks,
Pierre

> Also dev pages should get updated daily [there is a bug here
> that I should fix]
> 
> With the latest snapshots - both the pages appear to list MatGetRow_MPIDense
> 
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.html
> https://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetRow.html
> 
> Satish
> 
> On Tue, 6 Apr 2021, Pierre Jolivet wrote:
> 
>> Thanks for the recent MR.
>> This sadly only partially addresses the issue.
>> https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.html
>>  
>> <https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.html>
>>  correctly has MatGetRow_MPIDense while 
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.html
>>  
>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.html>
>>  hasn’t.
>> 
>> Thanks,
>> Pierre
>> 
>>> On 3 Apr 2021, at 1:24 PM, Patrick Sanan  wrote:
>>> 
>>> I think this was likely a non-portable regex that was being used to match 
>>> the function signatures,
>>> causing it to miss anything with an internal "s" in the function name. 
>>> Here's a quick fix for the release branch:
>>> 
>>> https://gitlab.com/petsc/petsc/-/merge_requests/3813 
>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3813>
>>> 
>>> (This sort of thing will become easier to work with, soon, as we'll have an 
>>> automated doc build which can be examined for each MR.)
>>> 
>>> P.S. nice email address! 
>>> 
>>>> Am 03.04.2021 um 11:19 schrieb Patrick Sanan >>> <mailto:patrick.sa...@gmail.com>>:
>>>> 
>>>> That isn't expected - thanks for pointing this out! I can reproduce 
>>>> locally so will take a look. 
>>>> 
>>>>> Am 03.04.2021 um 11:10 schrieb Pierre Jolivet >>>> <mailto:pie...@joliv.et>>:
>>>>> 
>>>>> Hello,
>>>>> https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.html
>>>>>  
>>>>> <https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.html>
>>>>>  lists available implementations while 
>>>>> https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.html
>>>>>  
>>>>> <https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.html>
>>>>>  doesn’t.
>>>>> Is this expected?
>>>>> 
>>>>> Thanks,
>>>>> Pierre
>>>> 
>>> 
>> 
>> 

Re: [petsc-dev] Regression in manualpages generation?

[Pierre Jolivet]

Thanks for the recent MR.
This sadly only partially addresses the issue.
https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.html 
<https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.html>
 correctly has MatGetRow_MPIDense while 
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.html 
<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.html>
 hasn’t.

Thanks,
Pierre

> On 3 Apr 2021, at 1:24 PM, Patrick Sanan  wrote:
> 
> I think this was likely a non-portable regex that was being used to match the 
> function signatures,
> causing it to miss anything with an internal "s" in the function name. Here's 
> a quick fix for the release branch:
> 
> https://gitlab.com/petsc/petsc/-/merge_requests/3813 
> <https://gitlab.com/petsc/petsc/-/merge_requests/3813>
> 
> (This sort of thing will become easier to work with, soon, as we'll have an 
> automated doc build which can be examined for each MR.)
> 
> P.S. nice email address! 
> 
>> Am 03.04.2021 um 11:19 schrieb Patrick Sanan > <mailto:patrick.sa...@gmail.com>>:
>> 
>> That isn't expected - thanks for pointing this out! I can reproduce locally 
>> so will take a look. 
>> 
>>> Am 03.04.2021 um 11:10 schrieb Pierre Jolivet >> <mailto:pie...@joliv.et>>:
>>> 
>>> Hello,
>>> https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.html
>>>  
>>> <https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.html>
>>>  lists available implementations while 
>>> https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.html
>>>  
>>> <https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.html>
>>>  doesn’t.
>>> Is this expected?
>>> 
>>> Thanks,
>>> Pierre
>> 
> 

[petsc-dev] Regression in manualpages generation?

[Pierre Jolivet]

Hello,
https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.html 

 lists available implementations while 
https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.html 

 doesn’t.
Is this expected?

Thanks,
Pierre

Re: [petsc-dev] reproducing crashes in the test harness

[Pierre Jolivet]

> On 30 Mar 2021, at 12:37 PM, Matthew Knepley  wrote:
> 
> On Tue, Mar 30, 2021 at 12:25 AM Barry Smith  > wrote:
> 
> # FAILED snes_tutorials-ex12_quad_hpddm_reuse_threshold 
> snes_tutorials-ex12_p4est_nc_singular_2d_hpddm snes_tutorials-ex56_hpddm 
> snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij sys_tests-ex53_2 
> snes_tutorials-ex12_quad_hpddm_reuse_baij 
> snes_tutorials-ex12_quad_hpddm_reuse 
> snes_tutorials-ex12_p4est_singular_2d_hpddm 
> snes_tutorials-ex12_tri_parmetis_hpddm 
> snes_tutorials-ex12_quad_singular_hpddm sys_tests-ex26_1 sys_tests-ex26_2 
> snes_tutorials-ex12_tri_parmetis_hpddm_baij 
> snes_tutorials-ex12_tri_hpddm_reuse_baij snes_tutorials-ex12_tri_hpddm_reus
> 
> Scott,
> 
>   Any thoughts on how the test harness could tell the developer exactly how 
> to reproduce a problematic cases in the debugger without them digging around 
> in the code to check arguments etc.
> 
>   So for example "Run: mpiexec -n N ./xxx args -start_in_debugger" to 
> reproduce this problem? Then one could just cut and paste and be debugging 
> away.
> 
> I always just EXTRA_ARGUMENTS="-start_in_debugger". However, if you really 
> want to run standalone, V=1 will display the args I think

I don’t think it does.
It displays the path of the .sh script being invoked, but then you have to 
decipher that as well.
Also, when there are failures on workers such as diff-test+param, you can’t 
copy/paste that directly in $PETSC_ARCH/tests/echofailures.sh, you have to 
filter out the diff- and +param part, which is not very practical I think.

Thanks,
Pierre

>   Thanks,
> 
> Matt
>  
>   Thanks
> 
>   Barry
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] mpi.mod --with-fortran-interfaces=0

[Pierre Jolivet]

For now, I’ve merely removed the runtime error from there 
https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31#379d0ed64a11c2a30793be16c20a756695f74407_515_513
 
<https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31#379d0ed64a11c2a30793be16c20a756695f74407_515_513>
And configure seems to be happy (for now… my Windows VM is way to slow…) 
http://joliv.et/configure_msmpi.log <http://joliv.et/configure_msmpi.log>
I will try your diff as well (in about one hour and a half when the first build 
is through).

Thanks,
Pierre

> On 24 Mar 2021, at 3:10 PM, Satish Balay  wrote:
> 
> On Wed, 24 Mar 2021, Pierre Jolivet wrote:
> 
>> 
>> 
>>> On 24 Mar 2021, at 2:55 PM, Matthew Knepley  wrote:
>>> 
>>> On Wed, Mar 24, 2021 at 9:37 AM Pierre Jolivet >> <mailto:pie...@joliv.et>> wrote:
>>> Hello,
>>> It seems mpi.mod is now mandatory 
>>> https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31
>>>  
>>> <https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31>.
>>> mpi.mod is not shipped by MS-MPI.
>>> I want to build MUMPS through PETSc and don’t really care about the Fortran 
>>> PETSc interface on our Windows boxes (--with-fortran-interfaces=0).
>>> 
>>> I think you want --with-fc=0, which should turn off this check.
>> 
>> No, I have gfortran, I want to compile MUMPS with PETSc, that won’t work 
>> --with-fc=0.
>> Also, the error message in the configure is wrong, it should read "PETSc 
>> Fortran bindings” (not “fortran interface”), and in my email, one should 
>> replace --with-fortran-interfaces by --with-fortran-bindings, sorry about 
>> that.
>> I guess there is a missing check before raise RuntimeError('mpi.mod not 
>> found! PETSc fortran interface requires a working mpi.mod'), like if 
>> self.argDB['with-fortran-interfaces']
>> But I don’t know if that makes sense.
> 
> I guess you mean:
> 
> diff --git a/config/BuildSystem/config/packages/MPI.py 
> b/config/BuildSystem/config/packages/MPI.py
> index 7ea2716cc5..5a1baff9f9 100644
> --- a/config/BuildSystem/config/packages/MPI.py
> +++ b/config/BuildSystem/config/packages/MPI.py
> @@ -497,7 +497,7 @@ Unable to run hostname to check the network')
> 
>   def FortranMPICheck(self):
> '''Make sure fortran include [mpif.h] and library symbols are found'''
> -if not hasattr(self.compilers, 'FC'):
> +if not hasattr(self.compilers, 'FC') or not 
> self.argDB['with-fortran-bindings']:
>   return 0
> # Fortran compiler is being used - so make sure mpif.h exists
> self.libraries.pushLanguage('FC')
> 
> If MS-MPI is missing mpi.mod - I guess we'll have to revert this requirement. 
> For some reason the tests in CI don't error out. I'll have to check.
> 
> I can add back mpif.h usage  [without the extra complex code we had for 
> handling some corner cases - in this mode]
> 
> Satish
> 
>> 
>> Thanks,
>> Pierre
>> 
>>>  Thanks,
>>> 
>>>Matt
>>> 
>>> Yet, I’m now hit by:
>>> ***
>>> UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
>>> details):
>>> ---
>>> mpi.mod not found! PETSc fortran interface requires a working mpi.mod
>>> ***
>>> Could this be fixed somehow, please?
>>> Thanks,
>>> Pierre
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their 
>>> experiments is infinitely more interesting than any results to which their 
>>> experiments lead.
>>> -- Norbert Wiener
>>> 
>>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-dev] mpi.mod --with-fortran-interfaces=0

[Pierre Jolivet]

> On 24 Mar 2021, at 2:55 PM, Matthew Knepley  wrote:
> 
> On Wed, Mar 24, 2021 at 9:37 AM Pierre Jolivet  <mailto:pie...@joliv.et>> wrote:
> Hello,
> It seems mpi.mod is now mandatory 
> https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31
>  
> <https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31>.
> mpi.mod is not shipped by MS-MPI.
> I want to build MUMPS through PETSc and don’t really care about the Fortran 
> PETSc interface on our Windows boxes (--with-fortran-interfaces=0).
> 
> I think you want --with-fc=0, which should turn off this check.

No, I have gfortran, I want to compile MUMPS with PETSc, that won’t work 
--with-fc=0.
Also, the error message in the configure is wrong, it should read "PETSc 
Fortran bindings” (not “fortran interface”), and in my email, one should 
replace --with-fortran-interfaces by --with-fortran-bindings, sorry about that.
I guess there is a missing check before raise RuntimeError('mpi.mod not found! 
PETSc fortran interface requires a working mpi.mod'), like if 
self.argDB['with-fortran-interfaces']
But I don’t know if that makes sense.

Thanks,
Pierre

>   Thanks,
> 
> Matt
>  
> Yet, I’m now hit by:
> ***
>  UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
> details):
> ---
> mpi.mod not found! PETSc fortran interface requires a working mpi.mod
> ***
> Could this be fixed somehow, please?
> Thanks,
> Pierre
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

[petsc-dev] mpi.mod --with-fortran-interfaces=0

[Pierre Jolivet]

Hello,
It seems mpi.mod is now mandatory 
https://gitlab.com/petsc/petsc/-/commit/68bd534a466d12bdec28b4c298b715271ec2dd31
 
.
mpi.mod is not shipped by MS-MPI.
I want to build MUMPS through PETSc and don’t really care about the Fortran 
PETSc interface on our Windows boxes (--with-fortran-interfaces=0).
Yet, I’m now hit by:
***
 UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
details):
---
mpi.mod not found! PETSc fortran interface requires a working mpi.mod
***
Could this be fixed somehow, please?
Thanks,
Pierre

Re: [petsc-dev] Test harness + PETSC_HAVE_DEFINED

[Pierre Jolivet]

> On 22 Mar 2021, at 10:15 PM, Scott Kruger  wrote:
> 
> 
> It keys off of $PETSC_ARCH/include/petscconf.h so is a configure/build
> time configuration not a runtime configuration.  

Right, and since 3.12, The ./configure option --with-avx512-kernels 
defaults to true now so the manually optimized AVX-512 kernels are used by 
default.
So instead of having a single worker with:
config/examples/arch-ci-linux-knl.py:'--with-avx512-kernels=1',
There should probably have been some workers with the flag 
--with-avx512-kernels=0, I guess.
ksp_ksp_tests-ex6_3_skip_pipegcr is in fact skipped by all workers now.

Thanks,
Pierre

> Scott
> 
> 
> 
> On 2021-03-22 21:40, Pierre Jolivet did write:
>> 
>> 
>>> On 22 Mar 2021, at 9:24 PM, Pierre Jolivet  wrote:
>>> 
>>> Hello,
>>> My make check is skipping tests which have a “requires: 
>>> defined(PETSC_USE_SHARED_LIBRARIES)” with the message "SKIP 
>>> PETSC_HAVE_DEFINED(PETSC_USE_SHARED_LIBRARIES) requirement not met" even 
>>> though in configure.log I have:
>>> 2021-03-22T16:17:43.1626452Z #define PETSC_USE_SHARED_LIBRARIES 1
>>> Is this expected?
>> 
>> Sorry for the double send, I’m now realizing it should read define, not 
>> defined.
>> 
>>> Here is an ever more puzzling behavior.
>>> https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/
>>>  
>>> <https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/><https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/
>>>  
>>> <https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/>>
>>> ok ksp_ksp_tests-ex6_3_skip_pipegcr # SKIP Null requirement not met: 
>>> define(PETSC_USE_AVX512_KERNELS)
>>>  #PIPEGCR generates nans on linux-knl
>>>  test:
>>>requires: !define(PETSC_USE_AVX512_KERNELS)
>>>suffix: 3_skip_pipegcr
>> 
>> I’ve also realized that PETSC_USE_AVX512_KERNELS is defined on that worker, 
>> which seems a little weird to me (is it defined for all workers, even those 
>> which are not AVX512-capable?).
>> 
>> Thanks,
>> Pierre
>> 
>>> Why is this test skipped (on a worker other than linux-knl)?
>>> Thanks,
>>> Pierre
>> 
> 
> -- 
> Scott Kruger
> Tech-X Corporation   kru...@txcorp.com <mailto:kru...@txcorp.com>
> 5621 Arapahoe Ave, Suite A   Phone: (720) 466-3196
> Boulder, CO 80303Fax:   (303) 448-7756

Re: [petsc-dev] Test harness + PETSC_HAVE_DEFINED

[Pierre Jolivet]

> On 22 Mar 2021, at 9:24 PM, Pierre Jolivet  wrote:
> 
> Hello,
> My make check is skipping tests which have a “requires: 
> defined(PETSC_USE_SHARED_LIBRARIES)” with the message "SKIP 
> PETSC_HAVE_DEFINED(PETSC_USE_SHARED_LIBRARIES) requirement not met" even 
> though in configure.log I have:
> 2021-03-22T16:17:43.1626452Z #define PETSC_USE_SHARED_LIBRARIES 1
> Is this expected?

Sorry for the double send, I’m now realizing it should read define, not defined.

> Here is an ever more puzzling behavior.
> https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/
>  
> <https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/>
>  ok ksp_ksp_tests-ex6_3_skip_pipegcr # SKIP Null requirement not met: 
> define(PETSC_USE_AVX512_KERNELS)
>   #PIPEGCR generates nans on linux-knl
>   test:
> requires: !define(PETSC_USE_AVX512_KERNELS)
> suffix: 3_skip_pipegcr

I’ve also realized that PETSC_USE_AVX512_KERNELS is defined on that worker, 
which seems a little weird to me (is it defined for all workers, even those 
which are not AVX512-capable?).

Thanks,
Pierre

> Why is this test skipped (on a worker other than linux-knl)?
> Thanks,
> Pierre

[petsc-dev] Test harness + PETSC_HAVE_DEFINED

[Pierre Jolivet]

Hello,
My make check is skipping tests which have a “requires: 
defined(PETSC_USE_SHARED_LIBRARIES)” with the message "SKIP 
PETSC_HAVE_DEFINED(PETSC_USE_SHARED_LIBRARIES) requirement not met" even though 
in configure.log I have:
2021-03-22T16:17:43.1626452Z #define PETSC_USE_SHARED_LIBRARIES 1
Is this expected?

Here is an ever more puzzling behavior.
https://gitlab.com/petsc/petsc/-/jobs/1118286502/artifacts/browse/arch-ci-freebsd-pkgs-opt/tests/
 

 ok ksp_ksp_tests-ex6_3_skip_pipegcr # SKIP Null requirement not met: 
define(PETSC_USE_AVX512_KERNELS)
  #PIPEGCR generates nans on linux-knl
  test:
requires: !define(PETSC_USE_AVX512_KERNELS)
suffix: 3_skip_pipegcr

Why is this test skipped (on a worker other than linux-knl)?
Thanks,
Pierre

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

There is a fix there: 
https://gitlab.com/petsc/petsc/-/commit/ea41da7ae9ca74379a24f049c09adf372dd0fe67
 
<https://gitlab.com/petsc/petsc/-/commit/ea41da7ae9ca74379a24f049c09adf372dd0fe67>
It’s a spin-off of 
https://gitlab.com/petsc/petsc/-/commit/83f9b43b15ff2b6a8a4ff6970cc325feb78dc91b?merge_request_iid=3608
 
<https://gitlab.com/petsc/petsc/-/commit/83f9b43b15ff2b6a8a4ff6970cc325feb78dc91b?merge_request_iid=3608>
I don’t think it’s OK to have different behaviors between 
PCASM/PCGASM/PCBJacobi_Multiblock on one side and PCBJacobi_Singleblock on the 
other.
I’m also not sure what was the rationale for setting the same prefix for the 
SubMats and the Mat, as it makes it very hard to scope options properly, I 
think.
I can make a MR if you are OK with this change.

Thanks,
Pierre

$ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi
[…]
Mat Object: (sub_) 1 MPI processes
  type: seqaij
  rows=64, cols=64
[…]
$ mpirun -n 4 src/snes/tutorials/ex56 -cells 2,2,1 -max_conv_its 2 
-petscspace_degree 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 
1.e-10 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg 
-pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 
-pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 
0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason -snes_monitor_short 
-ksp_monitor_short -snes_converged_reason -use_mat_nearnullspace true 
-mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev 
-mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.1 -mg_levels_pc_type jacobi 
-petscpartitioner_type simple -matptap_via scalable -ex56_dm_view -snes_view 
-mg_coarse_sub_pc_factor_mat_solver_type mumps 
-mg_coarse_sub_mat_mumps_icntl_11 1
[…]
  ICNTL(11) (error analysis):  1
[…]
  linear system matrix = precond matrix:
  Mat Object: (mg_coarse_sub_) 1 MPI processes
type: seqaij
rows=6, cols=6, bs=6
[…]

> On 19 Mar 2021, at 12:01 PM, Pierre Jolivet  wrote:
> 
> 
> 
>> On 19 Mar 2021, at 5:00 AM, Barry Smith  wrote:
>> 
>> 
>> Eric,
>> 
>>> -Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1
>> 
>> If an option is skipped it is often due to the exact string name used with 
>> the option. I see your KSP option is 
>> Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 but then below I see 
>> 
>>> Coarse grid solver -- level ---
>>>   KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes
>>> type: preonly
>> 
>> That is, it seems to be looking for an option without the sub. This is 
>> normal. For the coarsest level of multigrid it uses coarse otherwise it uses 
>> levels.  Sub is used for block methods in PETSc such as block Jacobi methods 
>> but that doesn't seem to apply in your run with one process. It is possible 
>> since the run is on one process without a method such as block Jacobi the 
>> sub is just not relevant.
> 
> There is something wrong with the option, IMHO, Eric is right in thinking 
> that he should prepend sub_.
> The prefix is not being propagated properly, I’ll investigate.
> Here is a simple reproducer:
> $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi // KO
> […]
>linear system matrix = precond matrix:
>Mat Object: 1 MPI processes
>  type: seqaij
> […]
> $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type asm // OK
> […]
>linear system matrix = precond matrix:
>Mat Object: (sub_) 1 MPI processes
>  type: seqaij
> […]
> $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type gasm // OK
> […]
>  linear system matrix = precond matrix:
>  Mat Object: (sub_) 1 MPI processes
>type: seqaij
> […]
> $ mpirun -n 4 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi 
> -pc_bjacobi_blocks 1 // OK
> […]
>linear system matrix = precond matrix:
>Mat Object: (sub_) 4 MPI processes
>  type: mpiaij
> […]
> 
> Eric, in the meantime, you can just put the MUMPS options in the global 
> scope, i.e., -mat_mumps_icntl_24 1, but this will apply to all unprefixed 
> MUMPS instances.
> 
> Thanks,
> Pierre
> 
>> You can run any code with your current options and with -help  and then grep 
>> for particular options that you may wish to add. Or you can run with 
>> -ts/snes/ksp_view to see the option prefixes needed for each inner solve. 
>> 
>> I am not sure how to make the code bullet proof in your situation. ideally 
>> it would explain why your options don't work but I am not sure if that is 
>> possible.
>> 
>> Barry
>> 
>> 
>> 
>> 
>> 
>>> On Mar 18, 2021, at 8:46 PM, 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

> On 19 Mar 2021, at 5:00 AM, Barry Smith  wrote:
> 
> 
>  Eric,
> 
>> -Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1
> 
>  If an option is skipped it is often due to the exact string name used with 
> the option. I see your KSP option is 
> Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 but then below I see 
> 
>>  Coarse grid solver -- level ---
>>KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes
>>  type: preonly
> 
> That is, it seems to be looking for an option without the sub. This is 
> normal. For the coarsest level of multigrid it uses coarse otherwise it uses 
> levels.  Sub is used for block methods in PETSc such as block Jacobi methods 
> but that doesn't seem to apply in your run with one process. It is possible 
> since the run is on one process without a method such as block Jacobi the sub 
> is just not relevant.

There is something wrong with the option, IMHO, Eric is right in thinking that 
he should prepend sub_.
The prefix is not being propagated properly, I’ll investigate.
Here is a simple reproducer:
$ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi // KO
[…]
linear system matrix = precond matrix:
Mat Object: 1 MPI processes
  type: seqaij
[…]
$ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type asm // OK
[…]
linear system matrix = precond matrix:
Mat Object: (sub_) 1 MPI processes
  type: seqaij
[…]
$ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type gasm // OK
[…]
  linear system matrix = precond matrix:
  Mat Object: (sub_) 1 MPI processes
type: seqaij
[…]
$ mpirun -n 4 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi 
-pc_bjacobi_blocks 1 // OK
[…]
linear system matrix = precond matrix:
Mat Object: (sub_) 4 MPI processes
  type: mpiaij
[…]

Eric, in the meantime, you can just put the MUMPS options in the global scope, 
i.e., -mat_mumps_icntl_24 1, but this will apply to all unprefixed MUMPS 
instances.

Thanks,
Pierre

> You can run any code with your current options and with -help  and then grep 
> for particular options that you may wish to add. Or you can run with 
> -ts/snes/ksp_view to see the option prefixes needed for each inner solve. 
> 
> I am not sure how to make the code bullet proof in your situation. ideally it 
> would explain why your options don't work but I am not sure if that is 
> possible.
> 
>  Barry
> 
> 
> 
> 
> 
>> On Mar 18, 2021, at 8:46 PM, Eric Chamberland 
>>  wrote:
>> 
>> Hi again,
>> 
>> ok, just saw that some matrices have lines of "0" in case of 3D hermite DOFs 
>> (ex: du/dz derivatives) when used into a 2D plane mesh...
>> 
>> So, my last problem about hypre smoother is "normal".
>> 
>> However, just to play with one of this matrix, I tried to do a "LU" with 
>> mumps icntl_24 option activated on the global system: fine it works.
>> 
>> Then I tried to switche to GAMG with mumps for the coarse_sub level, but it 
>> seems my icntl_24 option is then ignored and I don't know why...
>> 
>> See my KSP:
>> 
>> KSP Object: (Options_ProjectionL2_0) 1 MPI processes
>>  type: bcgs
>>  maximum iterations=1, initial guess is zero
>>  tolerances:  relative=1e-15, absolute=1e-15, divergence=1e+12
>>  left preconditioning
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: (Options_ProjectionL2_0) 1 MPI processes
>>  type: gamg
>>type is MULTIPLICATIVE, levels=2 cycles=v
>>  Cycles per PCApply=1
>>  Using externally compute Galerkin coarse grid matrices
>>  GAMG specific options
>>Threshold for dropping small values in graph on each level =
>>Threshold scaling factor for each level not specified = 1.
>>AGG specific options
>>  Symmetric graph false
>>  Number of levels to square graph 1
>>  Number smoothing steps 1
>>Complexity:grid = 1.09756
>>  Coarse grid solver -- level ---
>>KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes
>>  type: preonly
>>  maximum iterations=1, initial guess is zero
>>  tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>>  left preconditioning
>>  using NONE norm type for convergence test
>>PC Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes
>>  type: bjacobi
>>number of blocks = 1
>>Local solver is the same for all blocks, as in the following KSP and 
>> PC objects on rank 0:
>>  KSP Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes
>>type: preonly
>>maximum iterations=1, initial guess is zero
>>tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>>left preconditioning
>>using NONE norm type for convergence test
>>  PC Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes
>>type: lu
>>  out-of-place factorization
>>  tolerance for zero pivot 2.22045e-14
>>  

Re: [petsc-dev] MatTransposeMatMult() bug

[Pierre Jolivet]

Thanks for the suggestion Hong.
I’ve been somehow putting on hold edits in the documentation 
(src/docs/website/documentation/changes/dev.html notwithstanding) because I’m 
not sure yet of what will be migrated or not.
But I’ll edit the page according to Patrick’s comment.

Thanks,
Pierre

> On 18 Mar 2021, at 4:00 PM, Patrick Sanan  wrote:
> 
> Sorry about the current mess but that page is halfway migrated, so any 
> updates should go here:
> https://docs.petsc.org/en/main/install/externalsoftware_documentation/ 
> <https://docs.petsc.org/en/main/install/externalsoftware_documentation/>
> 
> 
> 
>> Am 18.03.2021 um 15:22 schrieb Zhang, Hong via petsc-dev 
>> mailto:petsc-dev@mcs.anl.gov>>:
>> 
>> Pierre,
>> This is an external package to petsc. Shall it be listed at
>> https://www.mcs.anl.gov/petsc/miscellaneous/external.html 
>> <https://www.mcs.anl.gov/petsc/miscellaneous/external.html>?
>> PETSc: External Software - anl.gov 
>> <https://www.mcs.anl.gov/petsc/miscellaneous/external.html>
>> PETSc interfaces to the following optional external software (installing 
>> packages) (manual pages):. AMD - Approximate minimum degree orderings.; BLAS 
>> and LAPACK; Chaco - a graph partitioning package.; ESSL - IBM's math library 
>> for fast sparse direct LU factorization. FFTW - Fastest Fourier Transform in 
>> the West, developed at MIT by Matteo Frigo and Steven G. Johnson.
>> www.mcs.anl.gov <http://www.mcs.anl.gov/>
>> Hong
>> From: Pierre Jolivet mailto:pie...@joliv.et>>
>> Sent: Thursday, March 18, 2021 1:16 AM
>> To: Zhang, Hong mailto:hzh...@mcs.anl.gov>>
>> Cc: For users of the development version of PETSc > <mailto:petsc-dev@mcs.anl.gov>>
>> Subject: Re: [petsc-dev] MatTransposeMatMult() bug
>>  
>> https://www.sciencedirect.com/science/article/abs/pii/S089812212155 
>> <https://www.sciencedirect.com/science/article/abs/pii/S089812212155>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html
>>  
>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html 
>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html>
>> I need to update the PETSc user manual though, specifically with respect to 
>> systems with multiple right-hand sides.
>> But don’t worry, Stefano has sorted the bug out, which was due to a faulty 
>> MatSetFromOptions() in MatMAIJ, used by MatTransposeMatMult().
>> 
>> Thanks,
>> Pierre
>> 
>>> On 17 Mar 2021, at 11:21 PM, Zhang, Hong >> <mailto:hzh...@mcs.anl.gov>> wrote:
>>> 
>>> What is hpddm? I do not see its document.
>>> Hong
>>> 
>>> From: Matthew Knepley mailto:knep...@gmail.com>>
>>> Sent: Wednesday, March 17, 2021 2:49 PM
>>> To: Zhang, Hong mailto:hzh...@mcs.anl.gov>>
>>> Cc: Pierre Jolivet mailto:pie...@joliv.et>>; For users of 
>>> the development version of PETSc >> <mailto:petsc-dev@mcs.anl.gov>>
>>> Subject: Re: [petsc-dev] MatTransposeMatMult() bug
>>>  
>>> On Wed, Mar 17, 2021 at 3:27 PM Zhang, Hong via petsc-dev 
>>> mailto:petsc-dev@mcs.anl.gov>> wrote:
>>> Pierre,
>>> Do you mean a possible bug in C=AtB MatTransposeMatMult()?
>>> Can you provide a stand-alone test without hpddm that reproduces this 
>>> error? 
>>> 
>>> Hong, you should be able to just configure with --download-hpddm and then 
>>> run that ex76 test.
>>> 
>>>   Thanks,
>>> 
>>>  Matt
>>>  
>>> Hong
>>> From: petsc-dev >> <mailto:petsc-dev-boun...@mcs.anl.gov>> on behalf of Pierre Jolivet 
>>> mailto:pie...@joliv.et>>
>>> Sent: Wednesday, March 17, 2021 4:31 AM
>>> To: For users of the development version of PETSc >> <mailto:petsc-dev@mcs.anl.gov>>
>>> Subject: [petsc-dev] MatTransposeMatMult() bug
>>>  
>>> Hello,
>>> While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird 
>>> numerical errors when reusing a Mat for both MatProduct_AB and 
>>> MatProduct_AtB.
>>> This reminded me that there has been a long-standing issue with 
>>> MatTransposeMatMult(), see 
>>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608
>>>  
>>> <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.c

Re: [petsc-dev] MatTransposeMatMult() bug

[Pierre Jolivet]

https://www.sciencedirect.com/science/article/abs/pii/S089812212155 
<https://www.sciencedirect.com/science/article/abs/pii/S089812212155>
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html 
<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPHPDDM.html>
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html 
<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHPDDM.html>
I need to update the PETSc user manual though, specifically with respect to 
systems with multiple right-hand sides.
But don’t worry, Stefano has sorted the bug out, which was due to a faulty 
MatSetFromOptions() in MatMAIJ, used by MatTransposeMatMult().

Thanks,
Pierre

> On 17 Mar 2021, at 11:21 PM, Zhang, Hong  wrote:
> 
> What is hpddm? I do not see its document.
> Hong
> 
> From: Matthew Knepley 
> Sent: Wednesday, March 17, 2021 2:49 PM
> To: Zhang, Hong 
> Cc: Pierre Jolivet ; For users of the development version of 
> PETSc 
> Subject: Re: [petsc-dev] MatTransposeMatMult() bug
>  
> On Wed, Mar 17, 2021 at 3:27 PM Zhang, Hong via petsc-dev 
> mailto:petsc-dev@mcs.anl.gov>> wrote:
> Pierre,
> Do you mean a possible bug in C=AtB MatTransposeMatMult()?
> Can you provide a stand-alone test without hpddm that reproduces this error? 
> 
> Hong, you should be able to just configure with --download-hpddm and then run 
> that ex76 test.
> 
>   Thanks,
> 
>  Matt
>  
> Hong
> From: petsc-dev  <mailto:petsc-dev-boun...@mcs.anl.gov>> on behalf of Pierre Jolivet 
> mailto:pie...@joliv.et>>
> Sent: Wednesday, March 17, 2021 4:31 AM
> To: For users of the development version of PETSc  <mailto:petsc-dev@mcs.anl.gov>>
> Subject: [petsc-dev] MatTransposeMatMult() bug
>  
> Hello,
> While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird 
> numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB.
> This reminded me that there has been a long-standing issue with 
> MatTransposeMatMult(), see 
> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608
>  
> <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608>,
>  that I never looked into.
> I’ve now been trying to figure this out, because this has side effects in 
> multiple places (PCMG and PCHPDDM at least), and thus could impact user-code 
> as well?
> With this commit: 
> https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0
>  
> <https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0>
> +
> $ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm 
> -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir 
> ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction 
> balanced -C_input_mattransposematmult -D_output_mattransposematmult
> I’m seeing that C is nonzero, but D is full of zeros.
> Mat Object: 4 MPI processes
>   type: mpidense
> 5.7098316584361917e-08 1.0159399260517841e-07
> 1.5812349976211856e-07 2.0688121715350138e-07
> 2.4887556933361981e-08 4.8111092300772958e-08
> 1.4606298643602107e-07 1.7213611729839211e-07
> […]
> Mat Object: 4 MPI processes
>   type: mpidense
> 0.e+00 0.e+00
> 0.e+00 0.e+00
> 0.e+00 0.e+00
> 0.e+00 0.e+00
> […]
> 
> If one switches to a MatType which has no MatProduct_AtB implementation with 
> B of type MPIDense (reminder: in that case, the product is computed 
> column-by-column), e.g., -mat_type sbaij, one gets the expected result.
> Mat Object: 4 MPI processes
>   type: mpidense
> 7.2003197398135299e-01 9.5191869895699011e-01
> 6.1793966541680234e-02 9.3884397585488877e-01
> 1.0022337823233585e-02 2.4653068080134588e-01
> 1.4463931936094099e-01 8.6111517670701687e-01
> 
> Is there a bug somewhere with the MatAIJ implementation, or am I doing 
> something which is not allowed by the MatProduct() machinery?
> 
> Thanks,
> Pierre
> 
> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3717 
> <https://gitlab.com/petsc/petsc/-/merge_requests/3717> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-dev] MatTransposeMatMult() bug

[Pierre Jolivet]

Thanks Hong, this has been fixed by Stefano there: 
https://gitlab.com/petsc/petsc/-/commit/38faa66a3f98cdf432182095b5ca9cf1fc38724a?merge_request_iid=3717
 
<https://gitlab.com/petsc/petsc/-/commit/38faa66a3f98cdf432182095b5ca9cf1fc38724a?merge_request_iid=3717>
Pierre

> On 17 Mar 2021, at 8:27 PM, Zhang, Hong  wrote:
> 
> Pierre,
> Do you mean a possible bug in C=AtB MatTransposeMatMult()?
> Can you provide a stand-alone test without hpddm that reproduces this error? 
> Hong
> From: petsc-dev  on behalf of Pierre Jolivet 
> 
> Sent: Wednesday, March 17, 2021 4:31 AM
> To: For users of the development version of PETSc 
> Subject: [petsc-dev] MatTransposeMatMult() bug
>  
> Hello,
> While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird 
> numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB.
> This reminded me that there has been a long-standing issue with 
> MatTransposeMatMult(), see 
> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608
>  
> <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608>,
>  that I never looked into.
> I’ve now been trying to figure this out, because this has side effects in 
> multiple places (PCMG and PCHPDDM at least), and thus could impact user-code 
> as well?
> With this commit: 
> https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0
>  
> <https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0>
> +
> $ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm 
> -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir 
> ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction 
> balanced -C_input_mattransposematmult -D_output_mattransposematmult
> I’m seeing that C is nonzero, but D is full of zeros.
> Mat Object: 4 MPI processes
>   type: mpidense
> 5.7098316584361917e-08 1.0159399260517841e-07
> 1.5812349976211856e-07 2.0688121715350138e-07
> 2.4887556933361981e-08 4.8111092300772958e-08
> 1.4606298643602107e-07 1.7213611729839211e-07
> […]
> Mat Object: 4 MPI processes
>   type: mpidense
> 0.e+00 0.e+00
> 0.e+00 0.e+00
> 0.e+00 0.e+00
> 0.e+00 0.e+00
> […]
> 
> If one switches to a MatType which has no MatProduct_AtB implementation with 
> B of type MPIDense (reminder: in that case, the product is computed 
> column-by-column), e.g., -mat_type sbaij, one gets the expected result.
> Mat Object: 4 MPI processes
>   type: mpidense
> 7.2003197398135299e-01 9.5191869895699011e-01
> 6.1793966541680234e-02 9.3884397585488877e-01
> 1.0022337823233585e-02 2.4653068080134588e-01
> 1.4463931936094099e-01 8.6111517670701687e-01
> 
> Is there a bug somewhere with the MatAIJ implementation, or am I doing 
> something which is not allowed by the MatProduct() machinery?
> 
> Thanks,
> Pierre
> 
> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3717 
> <https://gitlab.com/petsc/petsc/-/merge_requests/3717>

[petsc-dev] MatTransposeMatMult() bug

[Pierre Jolivet]

Hello,
While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird 
numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB.
This reminded me that there has been a long-standing issue with 
MatTransposeMatMult(), see 
https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cxx.html#line608
 
,
 that I never looked into.
I’ve now been trying to figure this out, because this has side effects in 
multiple places (PCMG and PCHPDDM at least), and thus could impact user-code as 
well?
With this commit: 
https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aaceba0
 

+
$ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm 
-pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir 
${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction 
balanced -C_input_mattransposematmult -D_output_mattransposematmult
I’m seeing that C is nonzero, but D is full of zeros.
Mat Object: 4 MPI processes
  type: mpidense
5.7098316584361917e-08 1.0159399260517841e-07
1.5812349976211856e-07 2.0688121715350138e-07
2.4887556933361981e-08 4.8111092300772958e-08
1.4606298643602107e-07 1.7213611729839211e-07
[…]
Mat Object: 4 MPI processes
  type: mpidense
0.e+00 0.e+00
0.e+00 0.e+00
0.e+00 0.e+00
0.e+00 0.e+00
[…]

If one switches to a MatType which has no MatProduct_AtB implementation with B 
of type MPIDense (reminder: in that case, the product is computed 
column-by-column), e.g., -mat_type sbaij, one gets the expected result.
Mat Object: 4 MPI processes
  type: mpidense
7.2003197398135299e-01 9.5191869895699011e-01
6.1793966541680234e-02 9.3884397585488877e-01
1.0022337823233585e-02 2.4653068080134588e-01
1.4463931936094099e-01 8.6111517670701687e-01

Is there a bug somewhere with the MatAIJ implementation, or am I doing 
something which is not allowed by the MatProduct() machinery?

Thanks,
Pierre

(*) https://gitlab.com/petsc/petsc/-/merge_requests/3717 
 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

-pc_hypre_boomeramg_relax_type_all Jacobi => 
  Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 3
-pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => 
OK, independently of the architecture it seems (Eric Docker image with 1 or 2 
threads or my macOS), but contraction factor is higher
  Linear solve converged due to CONVERGED_RTOL iterations 8
  Linear solve converged due to CONVERGED_RTOL iterations 24
  Linear solve converged due to CONVERGED_RTOL iterations 26
v. currently
  Linear solve converged due to CONVERGED_RTOL iterations 7
  Linear solve converged due to CONVERGED_RTOL iterations 9
  Linear solve converged due to CONVERGED_RTOL iterations 10

Do we change this? Or should we force OMP_NUM_THREADS=1 for make test?

Thanks,
Pierre

> On 13 Mar 2021, at 2:26 PM, Mark Adams  wrote:
> 
> Hypre uses a multiplicative smoother by default. It has a chebyshev smoother. 
> That with a Jacobi PC should be thread invariant.
> Mark
> 
> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet  <mailto:pie...@joliv.et>> wrote:
> 
>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet > <mailto:pie...@joliv.et>> wrote:
>> 
>> Hello Eric,
>> I’ve made an “interesting” discovery, so I’ll put back the list in c/c.
>> It appears the following snippet of code which uses Allreduce() + lambda 
>> function + MPI_IN_PLACE is:
>> - Valgrind-clean with MPICH;
>> - Valgrind-clean with OpenMPI 4.0.5;
>> - not Valgrind-clean with OpenMPI 4.1.0.
>> I’m not sure who is to blame here, I’ll need to look at the MPI 
>> specification for what is required by the implementors and users in that 
>> case.
>> 
>> In the meantime, I’ll do the following:
>> - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI 4.1.0, 
>> see if any other error appears;
>> - provide a hotfix to bypass the segfaults;
> 
> I can confirm that splitting the single Allreduce with my own MPI_Op into two 
> Allreduce with MAX and BAND fixes the segfaults with OpenMPI (*).
> 
>> - look at the hypre issue and whether they should be deferred to the hypre 
>> team.
> 
> I don’t know if there is something wrong in hypre threading or if it’s just a 
> side effect of threading, but it seems that the number of threads has a 
> drastic effect on the quality of the PC.
> By default, it looks that there are two threads per process with your Docker 
> image.
> If I force OMP_NUM_THREADS=1, then I get the same convergence as in the 
> output file.
> 
> Thanks,
> Pierre
> 
> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 
> <https://gitlab.com/petsc/petsc/-/merge_requests/3712>
>> Thank you for the Docker files, they were really useful.
>> If you want to avoid oversubscription failures, you can edit the file 
>> /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line:
>> localhost slots=12
>> If you want to increase the timeout limit of PETSc test suite for each test, 
>> you can add the extra flag in your command line TIMEOUT=180 (default is 60, 
>> units are seconds).
>> 
>> Thanks, I’ll ping you on GitLab when I’ve got something ready for you to try,
>> Pierre
>> 
>> 
>> 
>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland 
>>> >> <mailto:eric.chamberl...@giref.ulaval.ca>> wrote:
>>> 
>>> Hi Pierre,
>>> 
>>> I now have a docker container reproducing the problems here.
>>> 
>>> Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm  it fails 
>>> like this:
>>> 
>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59
>>> #   Initial guess
>>> #   L_2 Error: 0.00803099
>>> #   Initial Residual
>>> #   L_2 Residual: 1.09057
>>> #   Au - b = Au + F(0)
>>> #   Linear L_2 Residual: 1.09057
>>> #   [d470c54ce086:14127] Read -1, expected 4096, errno = 1
>>> #   [d470c54ce086:14128] Read -1, expected 4096, errno = 1
>>> #   [d470c54ce086:14129] Read -1, expected 4096, errno = 1
>>> #   [3]PETSC ERROR: 
>>> 
>>> #   [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation 
>>> Violation, probably memory access out of range
>>> #   [3]PETSC ERROR: Try option -start_in_debugger or 
>>> -on_error_attach_debugger
>>> #   [3]PETSC ERROR: or see 
>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind 
>>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>>> #   [3]PETSC ERROR: or try http://va

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet  wrote:
> 
> Hello Eric,
> I’ve made an “interesting” discovery, so I’ll put back the list in c/c.
> It appears the following snippet of code which uses Allreduce() + lambda 
> function + MPI_IN_PLACE is:
> - Valgrind-clean with MPICH;
> - Valgrind-clean with OpenMPI 4.0.5;
> - not Valgrind-clean with OpenMPI 4.1.0.
> I’m not sure who is to blame here, I’ll need to look at the MPI specification 
> for what is required by the implementors and users in that case.
> 
> In the meantime, I’ll do the following:
> - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI 4.1.0, 
> see if any other error appears;
> - provide a hotfix to bypass the segfaults;

I can confirm that splitting the single Allreduce with my own MPI_Op into two 
Allreduce with MAX and BAND fixes the segfaults with OpenMPI (*).

> - look at the hypre issue and whether they should be deferred to the hypre 
> team.

I don’t know if there is something wrong in hypre threading or if it’s just a 
side effect of threading, but it seems that the number of threads has a drastic 
effect on the quality of the PC.
By default, it looks that there are two threads per process with your Docker 
image.
If I force OMP_NUM_THREADS=1, then I get the same convergence as in the output 
file.

Thanks,
Pierre

(*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 
<https://gitlab.com/petsc/petsc/-/merge_requests/3712>
> Thank you for the Docker files, they were really useful.
> If you want to avoid oversubscription failures, you can edit the file 
> /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line:
> localhost slots=12
> If you want to increase the timeout limit of PETSc test suite for each test, 
> you can add the extra flag in your command line TIMEOUT=180 (default is 60, 
> units are seconds).
> 
> Thanks, I’ll ping you on GitLab when I’ve got something ready for you to try,
> Pierre
> 
> 
> 
>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland 
>> mailto:eric.chamberl...@giref.ulaval.ca>> 
>> wrote:
>> 
>> Hi Pierre,
>> 
>> I now have a docker container reproducing the problems here.
>> 
>> Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm  it fails 
>> like this:
>> 
>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59
>> #   Initial guess
>> #   L_2 Error: 0.00803099
>> #   Initial Residual
>> #   L_2 Residual: 1.09057
>> #   Au - b = Au + F(0)
>> #   Linear L_2 Residual: 1.09057
>> #   [d470c54ce086:14127] Read -1, expected 4096, errno = 1
>> #   [d470c54ce086:14128] Read -1, expected 4096, errno = 1
>> #   [d470c54ce086:14129] Read -1, expected 4096, errno = 1
>> #   [3]PETSC ERROR: 
>> 
>> #   [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation 
>> Violation, probably memory access out of range
>> #   [3]PETSC ERROR: Try option -start_in_debugger or 
>> -on_error_attach_debugger
>> #   [3]PETSC ERROR: or see 
>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind 
>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>> #   [3]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on 
>> GNU/linux and Apple Mac OS X to find memory corruption errors
>> #   [3]PETSC ERROR: likely location of problem given in stack below
>> #   [3]PETSC ERROR: -  Stack Frames 
>> 
>> #   [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not 
>> available,
>> #   [3]PETSC ERROR:   INSTEAD the line number of the start of the 
>> function
>> #   [3]PETSC ERROR:   is given.
>> #   [3]PETSC ERROR: [3] buildTwo line 987 
>> /opt/petsc-main/include/HPDDM_schwarz.hpp
>> #   [3]PETSC ERROR: [3] next line 1130 
>> /opt/petsc-main/include/HPDDM_schwarz.hpp
>> #   [3]PETSC ERROR: - Error Message 
>> --
>> #   [3]PETSC ERROR: Signal received
>> #   [3]PETSC ERROR: [0]PETSC ERROR: 
>> 
>> 
>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read -1, expected 
>> ..." which I don't know where they come from...?
>> 
>> Hypre (like in diff-snes_tutorials-ex56_hypre)  is also having 
>> DIVERGED_INDEFINITE_PC failures...
>> 
>> Please see the 3 attached docker files:
>> 
>> 1) fedora_m

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

Hello Eric,I’ve made an “interesting” discovery, so I’ll put back the list in c/c.It appears the following snippet of code which uses Allreduce() + lambda function + MPI_IN_PLACE is:- Valgrind-clean with MPICH;- Valgrind-clean with OpenMPI 4.0.5;- not Valgrind-clean with OpenMPI 4.1.0.I’m not sure who is to blame here, I’ll need to look at the MPI specification for what is required by the implementors and users in that case.In the meantime, I’ll do the following:- update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI 4.1.0, see if any other error appears;- provide a hotfix to bypass the segfaults;- look at the hypre issue and whether they should be deferred to the hypre team.Thank you for the Docker files, they were really useful.If you want to avoid oversubscription failures, you can edit the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line:localhost slots=12If you want to increase the timeout limit of PETSc test suite for each test, you can add the extra flag in your command line TIMEOUT=180 (default is 60, units are seconds).Thanks, I’ll ping you on GitLab when I’ve got something ready for you to try,Pierre

ompi.cxx
Description: Binary data
On 12 Mar 2021, at 8:54 PM, Eric Chamberland <eric.chamberl...@giref.ulaval.ca> wrote:
  

  
  Hi Pierre,I now have a docker container reproducing the problems here.Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm 
  it fails like this:not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59
  #   Initial guess
  #   L_2 Error: 0.00803099
  #   Initial Residual
  #   L_2 Residual: 1.09057
  #   Au - b = Au + F(0)
  #   Linear L_2 Residual: 1.09057
  #   [d470c54ce086:14127] Read -1, expected 4096, errno = 1
  #   [d470c54ce086:14128] Read -1, expected 4096, errno = 1
  #   [d470c54ce086:14129] Read -1, expected 4096, errno = 1
  #   [3]PETSC ERROR:
  
  #   [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
  Violation, probably memory access out of range
  #   [3]PETSC ERROR: Try option -start_in_debugger or
  -on_error_attach_debugger
  #   [3]PETSC ERROR: or see
  https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
  #   [3]PETSC ERROR: or try http://valgrind.org on GNU/linux
  and Apple Mac OS X to find memory corruption errors
  #   [3]PETSC ERROR: likely location of problem given in stack
  below
  #   [3]PETSC ERROR: -  Stack Frames
  
  #   [3]PETSC ERROR: Note: The EXACT line numbers in the stack
  are not available,
  #   [3]PETSC ERROR:   INSTEAD the line number of the start
  of the function
  #   [3]PETSC ERROR:   is given.
  #   [3]PETSC ERROR: [3] buildTwo line 987
  /opt/petsc-main/include/HPDDM_schwarz.hpp
  #   [3]PETSC ERROR: [3] next line 1130
  /opt/petsc-main/include/HPDDM_schwarz.hpp
  #   [3]PETSC ERROR: - Error Message
  --
  #   [3]PETSC ERROR: Signal received
  #   [3]PETSC ERROR: [0]PETSC ERROR:
  also ex12_quad_hpddm_reuse_baij fails with a lot more "Read -1,
  expected ..." which I don't know where they come from...?Hypre (like in diff-snes_tutorials-ex56_hypre)  is also having
  DIVERGED_INDEFINITE_PC failures...Please see the 3 attached docker files:1) fedora_mkl_and_devtools : the DockerFile which install fedore
  33 with gnu compilers and MKL and everything to develop.2) openmpi: the DockerFile to bluid OpenMPI3) petsc: The las DockerFile that build/install and test PETScI build the 3 like this:docker build -t fedora_mkl_and_devtools -f
  fedora_mkl_and_devtools .docker build -t openmpi -f openmpi .docker build -t petsc -f petsc .Disclaimer: I am not a docker expert, so I may do things that are
  not docker-stat-of-the-art but I am opened to suggestions... ;)
I have just ran it on my portable (long) which have not enough
  cores, so many more tests failed (should force --oversubscribe but
  don't know how to).  I will relaunch on my workstation in a few
  minutes.I will now test your branch! (sorry for the delay).Thanks,Eric

On 2021-03-11 9:03 a.m., Eric
  Chamberland wrote:


  Hi Pierre,ok, that's interesting!I will try to build a docker image until tomorrow and give you
the exact recipe to reproduce the bugs.Eric
  
  On 2021-03-11 2:46 a.m., Pierre
Jolivet wrote:
  
  



  
On 11 Mar 2021, at 6:16 AM, Barry Smith <bsm.

Re: [petsc-dev] Are pipelines down?

[Pierre Jolivet]

You are linking to the forks, i.e., not …/petsc/petsc/…, you’ll never see 
anything going on there, I think.
I’ve started the pipelines by clicking “Run Pipeline” here 
https://gitlab.com/petsc/petsc/-/merge_requests/3616/pipelines 
 and there 
https://gitlab.com/petsc/petsc/-/merge_requests/3707/pipelines 


Thanks,
Pierre

> On 12 Mar 2021, at 3:02 PM, Matthew Knepley  wrote:
> 
> On Fri, Mar 12, 2021 at 7:29 AM Matthew Knepley  > wrote:
> On Thu, Mar 11, 2021 at 11:24 PM Satish Balay  > wrote:
> For forks - a member of the developer group needs to start a new pipeline. 
> (after basic checks on the changes)
> 
> The original pipeline just sat pending in the source check, so I cancelled it.
> 
> I created a new pipeline, but it is just sitting pending in the source check
> 
>   https://gitlab.com/danfinn/petsc/-/pipelines/269297070 
> 
> 
> Why will this not run?
> 
> Same problem for Connor's branch: 
> https://gitlab.com/connorjward/petsc/-/pipelines/269584870 
> 
> 
>Matt
>  
>   Thanks,
> 
>  Matt
>  
> Satish
> 
> On Thu, 11 Mar 2021, Jed Brown wrote:
> 
> > Says the pipeline was canceled. The branch comes from a fork so it should 
> > need approval to run.
> > 
> > https://gitlab.com/petsc/petsc/-/merge_requests/3707 
> > 
> > 
> > Matthew Knepley mailto:knep...@gmail.com>> writes:
> > 
> > > https://gitlab.com/danfinn/petsc/-/pipelines/269297070 
> > > 
> > >
> > >   Thanks,
> > >
> > >  Matt
> > >
> > > -- 
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which their
> > > experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/ 
> > >  
> > >  > > >
> > 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] Fortran macOS Linkage Warning

[Pierre Jolivet]

Sorry to dig up this old thread, but this is triggering false positives while 
make checking for people who don’t want to toy around with 
MACOSX_DEPLOYMENT_TARGET.
Could/should we filter this somehow, or have BuildSystem tinker with the 
environment variable itself?

Thanks,
Pierre

$ make check
Running check examples to verify correct installation
Using PETSC_DIR=/Users/jolivet/repo/petsc and 
PETSC_ARCH=arch-darwin-c-debug-real
***Error detected during compile or link!***
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/Users/jolivet/repo/petsc/src/snes/tutorials ex19
*
mpicc -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
-Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first 
-Wl,-no_compact_unwind -Wall -g  -Wall -g
-I/Users/jolivet/repo/petsc/include 
-I/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/include 
-I/opt/intel/mkl/include ex19.c  
-Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
-L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
-Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
-L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib 
-Wl,-rpath,/opt/intel/mkl/lib -L/opt/intel/mkl/lib -Wl,-rpath,/opt/intel/lib 
-L/opt/intel/lib -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib 
-L/usr/local/Cellar/mpich/3.4.1_1/lib 
-Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0
 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 
-Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 
-L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -lpetsc -lHYPRE -lcmumps -ldmumps 
-lsmumps -lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64 
-lmkl_blacs_mpich_lp64 -lEl -lElSuiteSparse -lpmrrr -lp4est -lsc -liomp5 
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lparmmg -lmmg -lmmg3d 
-lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch 
-lscotcherr -lparmetis -lmetis -ltriangle -ltet -lz -lX11 -lc++ -ldl -lmpifort 
-lmpi -lpmpi -lgfortran -lquadmath -lm -lc++ -ldl -o ex19
ld: warning: dylib (/usr/local/Cellar/mpich/3.4.1_1/lib/libmpifort.dylib) was 
built for newer macOS version (11.1) than being linked (11.0)
ld: warning: dylib 
(/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgfortran.dylib) was built for 
newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib 
(/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.dylib) was built for 
newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib 
(/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.0.dylib) was built for 
newer macOS version (11.2) than being linked (11.0)
ld: warning: object file 
(/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libcmumps.a(cana_aux.o))
 was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: object file 
(/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libdmumps.a(dana_aux.o))
 was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: object file 
(/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libsmumps.a(sana_aux.o))
 was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: object file 
(/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libzmumps.a(zana_aux.o))
 was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: object file 
(/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libmumps_common.a(front_data_mgt_m.o))
 was built for newer macOS version (11.2) than being linked (11.0)
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process


> On 15 Feb 2021, at 6:08 PM, Jacob Faibussowitsch  wrote:
> 
> Yes, this also seems to work on my end.
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
>> On Feb 15, 2021, at 12:05, Satish Balay > > wrote:
>> 
>> Does setting MACOSX_DEPLOYMENT_TARGET also work?
>> 
>> Satish
>> 
>> On Mon, 15 Feb 2021, Jacob Faibussowitsch wrote:
>> 
>>> I think this problem ultimately stems from the fact that gcc/g++ and 
>>> gfortran are not from the same people on macOS. It seems like clang is 
>>> configured to build for 11.0 target, whilst gcc defaults to newest 
>>> available target. And indeed if I change my compile command there is no 
>>> warning:
>>> 
>>> mkdir -p build && gfortran -c -o ./build/conftest.o conftest.F90 && gcc-10 
>>> -o ./build/libconftest.dylib ./build/conftest.o -dynamiclib -undefined 
>>> dynamic_lookup && rm -rf ./build
>>> 
>>> Note I now call gcc-10 as the linker which is the __actual__ gcc (not clang 
>>> in a trench coat).
>>> 
>>> Best regards,
>>> 
>>> Jacob Faibussowitsch
>>> (Jacob Fai - booss - oh - vitch)
>>> Cell: (312) 694-3391
>>> 
 On Feb 15, 2021, at 11:46, Jacob Faibussowitsch >>> > wrote:
 
 Ok so if I 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

> On 11 Mar 2021, at 8:46 AM, Pierre Jolivet  wrote:
> 
>> On 11 Mar 2021, at 6:16 AM, Barry Smith > <mailto:bsm...@petsc.dev>> wrote:
>> 
>>   Eric,
>> 
>>Sorry about not being more immediate. We still have this in our active 
>> email so you don't need to submit individual issues. We'll try to get to 
>> them as soon as we can.
> 
> Indeed, I’m still trying to figure this out.
> I realized that some of my configure flags were different than yours, e.g., 
> no --with-memalign.
> I’ve also added SuperLU_DIST to my installation.
> Still, I can’t reproduce any issue.
> I will continue looking into this, it appears I’m seeing some valgrind 
> errors, but I don’t know if this is some side effect of OpenMPI not being 
> valgrind-clean (last time I checked, there was no error with MPICH).

It looks like Valgrind + OpenMPI (+ OpenMP?) is complaining about uninitialized 
memory in PetscSFGetMultiSF().
Could you please try out the following branch 
https://gitlab.com/petsc/petsc/-/commits/jolivet/fix-valgrind-openmpi 
<https://gitlab.com/petsc/petsc/-/commits/jolivet/fix-valgrind-openmpi> ?
I’m not sure why there would be such a warning with OpenMPI and not with MPICH, 
and it is unlikely to fix anything, but for good measure, after compilation, 
could you please try:
$ make -f gmakefile test search='snes_tutorials-ex12_quad_hpddm_reuse_baij’

Thanks,
Pierre

> Thank you for your patience,
> Pierre
> 
> /usr/bin/gmake -f gmakefile test test-fail=1
> Using MAKEFLAGS: test-fail=1
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts
>  ok snes_tutorials-ex12_quad_hpddm_reuse_baij
>  ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts
>  ok ksp_ksp_tests-ex33_superlu_dist_2
>  ok diff-ksp_ksp_tests-ex33_superlu_dist_2
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0
>  ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1
>  ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts
>  ok ksp_ksp_tutorials-ex50_tut_2
>  ok diff-ksp_ksp_tutorials-ex50_tut_2
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts
>  ok ksp_ksp_tests-ex33_superlu_dist
>  ok diff-ksp_ksp_tests-ex33_superlu_dist
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts
>  ok snes_tutorials-ex56_hypre
>  ok diff-snes_tutorials-ex56_hypre
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts
>  ok ksp_ksp_tutorials-ex56_2
>  ok diff-ksp_ksp_tutorials-ex56_2
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts
>  ok snes_tutorials-ex17_3d_q3_trig_elas
>  ok diff-snes_tutorials-ex17_3d_q3_trig_elas
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts
>  ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij
>  ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts
> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1
> # srun: error: Unable to create step for job 1426755: More processors 
> requested than permitted
>  ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command failed so no diff
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts
>  ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran required for this test
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts
>  ok snes_tutorials-ex12_tri_parme

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

> On 11 Mar 2021, at 6:16 AM, Barry Smith  wrote:
> 
> 
>   Eric,
> 
>Sorry about not being more immediate. We still have this in our active 
> email so you don't need to submit individual issues. We'll try to get to them 
> as soon as we can.

Indeed, I’m still trying to figure this out.
I realized that some of my configure flags were different than yours, e.g., no 
--with-memalign.
I’ve also added SuperLU_DIST to my installation.
Still, I can’t reproduce any issue.
I will continue looking into this, it appears I’m seeing some valgrind errors, 
but I don’t know if this is some side effect of OpenMPI not being 
valgrind-clean (last time I checked, there was no error with MPICH).

Thank you for your patience,
Pierre

/usr/bin/gmake -f gmakefile test test-fail=1
Using MAKEFLAGS: test-fail=1
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts
 ok snes_tutorials-ex12_quad_hpddm_reuse_baij
 ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts
 ok ksp_ksp_tests-ex33_superlu_dist_2
 ok diff-ksp_ksp_tests-ex33_superlu_dist_2
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0
 ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1
 ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts
 ok ksp_ksp_tutorials-ex50_tut_2
 ok diff-ksp_ksp_tutorials-ex50_tut_2
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts
 ok ksp_ksp_tests-ex33_superlu_dist
 ok diff-ksp_ksp_tests-ex33_superlu_dist
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts
 ok snes_tutorials-ex56_hypre
 ok diff-snes_tutorials-ex56_hypre
TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts
 ok ksp_ksp_tutorials-ex56_2
 ok diff-ksp_ksp_tutorials-ex56_2
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts
 ok snes_tutorials-ex17_3d_q3_trig_elas
 ok diff-snes_tutorials-ex17_3d_q3_trig_elas
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts
 ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij
 ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts
not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1
#   srun: error: Unable to create step for job 1426755: More processors 
requested than permitted
 ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command failed so no diff
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts
 ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran required for this test
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts
 ok snes_tutorials-ex12_tri_parmetis_hpddm_baij
 ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts
 ok snes_tutorials-ex19_tut_3
 ok diff-snes_tutorials-ex19_tut_3
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts
 ok snes_tutorials-ex17_3d_q3_trig_vlap
 ok diff-snes_tutorials-ex17_3d_q3_trig_vlap
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts
 ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran required for this test
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts
 ok snes_tutorials-ex19_superlu_dist
 ok diff-snes_tutorials-ex19_superlu_dist
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts
 ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre
 ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts
 ok 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

Here is a fix for PETSc disregarding what kind of MPI implementation is being 
used with cluster PARDISO: https://gitlab.com/petsc/petsc/-/merge_requests/3678 
<https://gitlab.com/petsc/petsc/-/merge_requests/3678>
This doesn’t fix src/mat/tests/ex242, whose backtrace looks suspicious to me…
(gdb) #0  0x2ac152c9c6ed in mkl_trans_avx2_mkl_domatcopy2_t ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_avx2.so
#1  0x2ba345400dc7 in mkl_trans_avx2_mkl_domatcopy ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_avx2.so
#2  0x2ba31564fac2 in dmmdatdl_ ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so
#3  0x2ba3156d70de in PB_Cptran_DC ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so
#4  0x2ba315740062 in pdtran_ ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so
#5  0x2ba31370211b in MatTranspose_ScaLAPACK ()
   from 
/ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/lib/libpetsc.so.3.014

I’m now realizing that this is also segfaulting with BLACS IntelMPI, so it’s 
probably unrelated, and simply some Intel mess.
(gdb) #0  0x2ab818c4f3de in mkl_trans_def_mkl_domatcopy2_t ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_def.so
#1  0x2ab818c51247 in mkl_trans_def_mkl_domatcopy ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_def.so
#2  0x2ab76e0ba062 in dmmdatdl_ ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so
#3  0x2ab76e14157e in PB_Cptran_DC ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so
#4  0x2ab76e1aa502 in pdtran_ ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so
#5  0x2b7f87c833a1 in MatTranspose_ScaLAPACK ()
   from 
/ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-impi/lib/libpetsc.so.3.014

(Notice the switch from arch-linux2-c-opt-ompi to arch-linux2-c-opt-impi)

Thanks,
Pierre

> On 3 Mar 2021, at 8:42 AM, Pierre Jolivet  wrote:
> 
> 
>> If it ends that there is a problem combining MKL + openMP that relies on 
>> linking configuration for example, should it be a good thing to have this 
>> (--with-openmp=1) tested into the pipelines (with external packages of 
>> course)?
>> 
> As Barry said, there is not much (if any) OpenMP in PETSc.
> There is however some workers with the MKL (+ Intel compilers) turned on, but 
> I don’t think we test MKL + GNU compilers (which I feel like is a very niche 
> combination, hence not really worth testing, IMHO).
> 
>> Does the guys who maintain all these libs are reading petsc-dev? ;)
>> 
> I don’t think they are, but don’t worry, we do forward the appropriate 
> messages to them :)
> 
> About yesterday’s failures…
> 1) I cannot reproduce any of the PCHYPRE/PCBDDC/PCHPDDM errors (sorry I 
> didn’t bother putting the SuperLU_DIST tarball on my cluster)
> 2) I can reproduce the src/mat/tests/ex242.c error (which explicitly uses 
> ScaLAPACK, none of the above PC uses it explicitly, except PCBDDC/PCHPDDM 
> when using MUMPS on “big” problems where root nodes are factorized using 
> ScaLAPACK, see -mat_mumps_icntl_13)
> 3) I’m seeing that both on your machine and mine, PETSc BuildSystem insist on 
> linking libmkl_blacs_intelmpi_lp64.so even though we supply explicitly 
> libmkl_blacs_openmpi_lp64.so
> This for example yields a wrong Makefile.inc for MUMPS:
> $ cat arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep 
> blacs
> SCALAP  = […] -lmkl_blacs_openmpi_lp64
> LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […]
> 
> Despite what Barry says, I think PETSc is partially to blame as well (why use 
> libmkl_blacs_intelmpi_lp64.so even though BuildSystem is capable of detecting 
> we are using OpenMPI).
> I’ll try to fix this to see if it solves 2).
> 
> Thanks,
> Pierre
> 
> http://joliv.et/irene-rome-configure.log 
> <http://joliv.et/irene-rome-configure.log>
> $ /usr/bin/gmake -f gmakefile test test-fail=1
> Using MAKEFLAGS: test-fail=1
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts
>  ok snes_tutorials-ex12_quad_hpddm_reuse_baij
>  ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts
>  ok ksp_ksp_tutorials-ex50_tut_2 # SKIP PETSC_HAVE

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

n rank 1 in communicator MPI_COMM_WORLD
#   with errorcode 50176059.
#
#   NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
#   You may or may not see output from other processes, depending on
#   exactly when Open MPI kills them.
#   
--
#   srun: Job step aborted: Waiting up to 302 seconds for job step to 
finish.
#   slurmstepd-irene4047: error: *** STEP 1374176.36 ON irene4047 CANCELLED 
AT 2021-03-03T08:21:20 ***
#   srun: error: irene4047: task 0: Killed
#   srun: error: irene4047: tasks 1-2: Exited with exit code 16
 ok mat_tests-ex242_3 # SKIP Command failed so no diff
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts
 ok snes_tutorials-ex17_3d_q3_trig_vlap
 ok diff-snes_tutorials-ex17_3d_q3_trig_vlap
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts
 ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre
 ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre
TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts
 ok ksp_ksp_tutorials-ex49_hypre_nullspace
 ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace
TEST 
arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xper_ref.counts
 ok ts_tutorials-ex18_p1p1_xper_ref
 ok diff-ts_tutorials-ex18_p1p1_xper_ref
TEST 
arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xyper_ref.counts
 ok ts_tutorials-ex18_p1p1_xyper_ref
 ok diff-ts_tutorials-ex18_p1p1_xyper_ref
TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts
 ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre
 ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre
TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts
 ok ksp_ksp_tutorials-ex64_1 # SKIP PETSC_HAVE_SUPERLU_DIST requirement not met

> On 3 Mar 2021, at 6:21 AM, Eric Chamberland 
>  wrote:
> 
> Just started a discussion on the side:
> 
> https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-Link-Line-Advisor-as-external-tool/m-p/1260895#M30974
>  
> <https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-Link-Line-Advisor-as-external-tool/m-p/1260895#M30974>
> Eric
> 
> On 2021-03-02 3:50 p.m., Pierre Jolivet wrote:
>> Hello Eric,
>> src/mat/tests/ex237.c is a recent test with some code paths that should be 
>> disabled for “old” MKL versions. It’s tricky to check directly in the source 
>> (we do check in BuildSystem) because there is no such thing as 
>> PETSC_PKG_MKL_VERSION_LT, but I guess we can change if 
>> defined(PETSC_HAVE_MKL) to if defined(PETSC_HAVE_MKL) && 
>> defined(PETSC_HAVE_MKL_SPARSE_OPTIMIZE), I’ll make a MR, thanks for 
>> reporting this.
>> 
>> For the other issues, I’m sensing this is a problem with gomp + 
>> intel_gnu_thread, but this is pure speculation… sorry.
>> I’ll try to reproduce some of these problems if you are not given a more 
>> meaningful answer.
>> 
>> Thanks,
>> Pierre
>> 
>>> On 2 Mar 2021, at 9:14 PM, Eric Chamberland 
>>> >> <mailto:eric.chamberl...@giref.ulaval.ca>> wrote:
>>> 
>>> Hi,
>>> 
>>> It all started when I wanted to test PETSC/CUDA compatibility for our code.
>>> 
>>> I had to activate --with-openmp to configure with --with-cuda=1 
>>> successfully.
>>> 
>>> I then saw that PETSC_HAVE_OPENMP  is used at least in MUMPS (and some 
>>> other places).
>>> 
>>> So, I configured and tested petsc with openmp activated, without CUDA.
>>> 
>>> The first thing I see is that our code CI pipelines now fails for many 
>>> tests.
>>> 
>>> After looking deeper, it seems that PETSc itself fails many tests when I 
>>> activate openmp!
>>> 
>>> Here are all the configurations I have results for, after/before activating 
>>> OpenMP for PETSc:
>>> ==
>>> 
>>> ==
>>> 
>>> For petsc/master + OpenMPI 4.0.4 + MKL 2019.4.243:
>>> 
>>> With OpenMP=1
>>> 
>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_make_test.log
>>>  
>>> <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_make_test.log>
>>> https://giref.ulaval.ca/~cmpgi

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

Hello Eric,
src/mat/tests/ex237.c is a recent test with some code paths that should be 
disabled for “old” MKL versions. It’s tricky to check directly in the source 
(we do check in BuildSystem) because there is no such thing as 
PETSC_PKG_MKL_VERSION_LT, but I guess we can change if defined(PETSC_HAVE_MKL) 
to if defined(PETSC_HAVE_MKL) && defined(PETSC_HAVE_MKL_SPARSE_OPTIMIZE), I’ll 
make a MR, thanks for reporting this.

For the other issues, I’m sensing this is a problem with gomp + 
intel_gnu_thread, but this is pure speculation… sorry.
I’ll try to reproduce some of these problems if you are not given a more 
meaningful answer.

Thanks,
Pierre

> On 2 Mar 2021, at 9:14 PM, Eric Chamberland 
>  wrote:
> 
> Hi,
> 
> It all started when I wanted to test PETSC/CUDA compatibility for our code.
> 
> I had to activate --with-openmp to configure with --with-cuda=1 successfully.
> 
> I then saw that PETSC_HAVE_OPENMP  is used at least in MUMPS (and some other 
> places).
> 
> So, I configured and tested petsc with openmp activated, without CUDA.
> 
> The first thing I see is that our code CI pipelines now fails for many tests.
> 
> After looking deeper, it seems that PETSc itself fails many tests when I 
> activate openmp!
> 
> Here are all the configurations I have results for, after/before activating 
> OpenMP for PETSc:
> ==
> 
> ==
> 
> For petsc/master + OpenMPI 4.0.4 + MKL 2019.4.243:
> 
> With OpenMP=1
> 
> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_make_test.log
>  
> 
> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_configure.log
>  
> 
> # -
> #   Summary
> # -
> # FAILED snes_tutorials-ex12_quad_hpddm_reuse_baij 
> diff-ksp_ksp_tests-ex33_superlu_dist_2 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 
> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 
> ksp_ksp_tutorials-ex50_tut_2 diff-ksp_ksp_tests-ex33_superlu_dist 
> diff-snes_tutorials-ex56_hypre snes_tutorials-ex17_3d_q3_trig_elas 
> snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij 
> ksp_ksp_tutorials-ex5_superlu_dist_3 ksp_ksp_tutorials-ex5f_superlu_dist 
> snes_tutorials-ex12_tri_parmetis_hpddm_baij diff-snes_tutorials-ex19_tut_3 
> mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap 
> ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex19_superlu_dist 
> diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre 
> diff-ksp_ksp_tutorials-ex49_hypre_nullspace ts_tutorials-ex18_p1p1_xper_ref 
> ts_tutorials-ex18_p1p1_xyper_ref snes_tutorials-ex19_superlu_dist_2 
> ksp_ksp_tutorials-ex5_superlu_dist_2 
> diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre 
> ksp_ksp_tutorials-ex64_1 ksp_ksp_tutorials-ex5_superlu_dist 
> ksp_ksp_tutorials-ex5f_superlu_dist_2
> # success 8275/10003 tests (82.7%)
> # failed 33/10003 tests (0.3%)
> With OpenMP=0
> 
> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_make_test.log
>  
> 
> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_configure.log
>  
> 
> # -
> #   Summary
> # -
> # FAILED tao_constrained_tutorials-tomographyADMM_6 
> snes_tutorials-ex17_3d_q3_trig_elas mat_tests-ex242_3 
> snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 
> tao_constrained_tutorials-tomographyADMM_5
> # success 8262/9983 tests (82.8%)
> # failed 6/9983 tests (0.1%)
> ==
> 
> ==
> 
> For OpenMPI 3.1.x/master:
> 
> With OpenMP=1:
> 
> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.log 
> 
> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.log 
> 
> # -
> #   Summary
> # -
> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 

Re: [petsc-dev] Infinite loop in A*B

[Pierre Jolivet]

> On 1 Mar 2021, at 6:29 AM, Zhang, Hong  wrote:
> 
> Pierre,
> This is a bug in MatMatMultSymbolic_MPIAIJ_MPIDense() during optimization of 
> block column size of B. Run your code with
> '-matmatmult_Bbn 1', the infinite loop should not occur.

Thanks Hong, I can confirm this option makes the more complex use case run 
smoothly as well. 

> I'll try to figure out a fix tomorrow.

Great.

Thanks,
Pierre

> Hong
> From: Zhang, Hong 
> Sent: Sunday, February 28, 2021 11:05 PM
> To: Pierre Jolivet ; For users of the development version of 
> PETSc ; Zhang, Hong 
> Subject: Re: [petsc-dev] Infinite loop in A*B
>  
> The infinite loop in MatMatMultNumeric_MPIAIJ_MPIDense()
> for (i=0; i }
> is cause by n=contents->workB->cmap->n=0 (line 590 in mpimatmatmult.c)
> Hong
> From: petsc-dev  on behalf of Zhang, Hong via 
> petsc-dev 
> Sent: Sunday, February 28, 2021 10:33 PM
> To: Pierre Jolivet ; For users of the development version of 
> PETSc 
> Subject: Re: [petsc-dev] Infinite loop in A*B
>  
> I can reproduce the hang with 
> mpiexec -n 2 ./matmatmult
> 
> It seems in an infinite loop of calling MatDensePlaceArray() from 
> 
> #0  MatDensePlaceArray (mat=0xda5c50, array=0xd15e60)
> at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:2047
> #1  0x7fa0d13bf4f7 in MatDenseGetSubMatrix_SeqDense (A=0xcfb2b0, cbegin=0,
> cend=0, v=0xd90370)
> at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:2997
> #2  0x7fa0d13c574e in MatDenseGetSubMatrix (A=0xcfb2b0, cbegin=0, cend=0,
> v=0xd90370) at 
> /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371
> #3  0x7fa0d13db5ce in MatDenseGetSubMatrix_MPIDense (A=0xca5250, cbegin=0,
> cend=0, v=0x7ffe87d41de0)
> at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:1835
> #4  0x7fa0d13c574e in MatDenseGetSubMatrix (A=0xca5250, cbegin=0, cend=0,
> v=0x7ffe87d41de0)
> at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371
> #5  0x7fa0d179c2fa in MatMatMultNumeric_MPIAIJ_MPIDense (A=0xc55490,
> B=0xca5250, C=0xd282b0)
> at /home/hongsu/soft/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:593
> #6  0x7fa0d1181331 in MatProductNumeric_AB (mat=0xd282b0)
> at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:567
> #7  0x7fa0d1182c14 in MatProductNumeric (mat=0xd282b0)
> at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:679
> #8  0x7fa0d115ef69 in MatProduct_Private (A=0xc55490, B=0xca5250,
> scall=MAT_INITIAL_MATRIX, fill=-2, ptype=MATPRODUCT_AB, C=0x7ffe87d42018)
> at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9405
> ---Type  to continue, or q  to quit---
> #9  0x7fa0d115f274 in MatMatMult (A=0xc55490, B=0xca5250, 
> scall=MAT_INITIAL_MATRIX, fill=-2,
> C=0x7ffe87d42018) at 
> /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9445
> #10 0x0040130a in main (argc=2, argv=0x7ffe87d42108) at ex1.c:20
> 
> I'll try to figure out what is going on. If anyone has a clue, please help. 
> The above stack comes from 'release' branch.
> Hong
> From: petsc-dev  on behalf of Pierre Jolivet 
> 
> Sent: Sunday, February 28, 2021 4:17 PM
> To: For users of the development version of PETSc 
> Subject: [petsc-dev] Infinite loop in A*B
>  
> Hello,
> The following MWE loops indefinitely for MPI_Comm_size in {2; 3}.
> Nothing fancy, just MatAIJ and MatDense.
> The problem is either in MatMPIDenseScatter() or 
> MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with 
> those routines can figure out a hot fix, I’m all ears.
> I could of course switch to a MatMult(), but the same infinite loop happens 
> in another more complex code with
> A = rows=8, cols=35212
> B = rows=35212, cols=9
> So I’ll need a fix eventually.
> 
> Thanks,
> Pierre

[petsc-dev] Infinite loop in A*B

[Pierre Jolivet]

Hello,
The following MWE loops indefinitely for MPI_Comm_size in {2; 3}.
Nothing fancy, just MatAIJ and MatDense.
The problem is either in MatMPIDenseScatter() or 
MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with 
those routines can figure out a hot fix, I’m all ears.
I could of course switch to a MatMult(), but the same infinite loop happens in 
another more complex code with
A = rows=8, cols=35212
B = rows=35212, cols=9
So I’ll need a fix eventually.

Thanks,
Pierre



matmatmult.c
Description: Binary data

Re: [petsc-dev] headsup: switch git default branch from 'master' to 'main'

[Pierre Jolivet]

What about https://www.mcs.anl.gov/petsc/petsc-master 
 ? Will it be moved to 
https://www.mcs.anl.gov/petsc/petsc-main 
 or will only 
https://www.mcs.anl.gov/petsc/petsc-dev 
 remain online?

Thanks,
Pierre

> On 26 Feb 2021, at 6:06 PM, Satish Balay via petsc-dev 
>  wrote:
> 
> I plan to delete 'master' immediately - so that folk don't assume it still 
> exits and work with it [assuming its the latest, creating MRs against it 
> etc..]
> 
> Satish
> 
> On Fri, 26 Feb 2021, Patrick Sanan wrote:
> 
>> The answers to these were probably stated already, but the reminder might be 
>> useful to others, as well.
>> 
>> What will happen to "master" after today? Will it be deleted immediately or 
>> at some planned time? If not immediately deleted, will it be updated to 
>> match main?
>> 
>>> Am 23.02.2021 um 18:19 schrieb Satish Balay via petsc-dev 
>>> :
>>> 
>>> All,
>>> 
>>> This is a heads-up, we are to switch the default branch in petsc git
>>> repo from 'master' to 'main'
>>> 
>>> [Will plan to do the switch on friday the 26th]
>>> 
>>> We've previously switched 'maint' branch to 'release' before 3.14
>>> release - and this change (to 'main') is the next step in this direction.
>>> 
>>> Satish
>>> 
>> 
> 

Re: [petsc-dev] error with flags PETSc uses for determining AVX

[Pierre Jolivet]

> On 15 Feb 2021, at 1:41 AM, Barry Smith  wrote:
> 
> 
> 
>> On Feb 14, 2021, at 4:04 PM, Jed Brown  wrote:
>> 
>> Barry Smith  writes:
>> 
>>>  My feeling is 90+% of users don't care about portability, they want to get 
>>> fast performance on the machine they are compiling with (or a collection of 
>>> machines they have around).  
>> 
>> This is an exclusionary view of users. PETSc could and would be used in more 
>> places if it was easy to install, e.g., if "pip install petsc" was a 
>> 3-second binary install instead of several minutes that requires compilers.
> 
>  You misunderstood my statement, or I phrased it poorly. For those building 
> PETSc themselves on their system using configure I don't think they care 
> about building a library that is portable, that is that they can copy to any 
> machine in the world and use. They want it to run fast on the machine(s) they 
> are building on. Hence I think for users who build it themselves configure 
> should lean for performance not portability. 
> 
>  Yes, having nice prebuilts (with as much performance as realistically 
> possible in a portable library) is desirable. 
> 
>  Barry
> 
>  This is why I think configure should use native and those (making portable 
> packages) should have to indicate they want portability. I hate to see 
> everyone who builds PETSc themselves (which is many) get lousy performance 
> just so package developers don't need to indicate to PETSc's configure they 
> want portability. Thus a flag like --with-portable or something is desirable, 
> where it is off by default. If we get to the point where we have both the 
> performance and the portability with any build then, of course, we will not 
> need such a flag anymore. But one step at a time.
> 

I’m playing the Devil’s advocate since the beginning, we’ve had a 
--enable-generic (off by default) which turns -march=native into -march=generic 
for 20+ years (I’m seeing references to PPC7450 and Intel Coppermine in our 
configure…).
We turn this flag on when building our .exe/.pkg/.deb.
I just think you’ll have to find a very very efficient way to advertise this 
change, because let’s face it, very few people outside of upstream developers 
read the changelog, and so people may start bundling up PETSc without this 
additional flag, and could potentially get spammed by end-users having a hard 
time figuring out why they get illegal instruction errors.
On some occasions we have such a message, but PETSc stands much lower in the 
software stack than our library so it’s likely it's bundled (instead of used on 
the same architecture as the compile node) way more often, IMHO (I guess you 
know the figure better than me…).

Thanks,
Pierre

Re: [petsc-dev] error with flags PETSc uses for determining AVX

[Pierre Jolivet]

> On 14 Feb 2021, at 4:52 PM, Zhang, Hong via petsc-dev  
> wrote:
> 
> 
> 
>> On Feb 14, 2021, at 5:05 AM, Patrick Sanan > > wrote:
>> 
>> 
>> 
>>> Am 14.02.2021 um 07:22 schrieb Barry Smith >> >:
>>> 
>>> 
>>> 
 On Feb 13, 2021, at 11:58 PM, Jed Brown >>> > wrote:
 
 I usually configure --with-debugging=0 COPTFLAGS='-O2 -march=native' or 
 similar. There's a tension here between optimizing aggressively for the 
 current machine and making binaries that work on other machines. Most 
 configure systems default to making somewhat portable binaries, so that's 
 a principal of least surprise. (Though you're no novice and seem to have 
 been surprised anyway.)
 
 I'd kinda prefer if we recommended making portable binaries that run-time 
 detected when to use newer instructions where it matters.
>>> 
>>>   How do we do this? What can we put in configure to do this.
>>> 
>>>   Yes, I never paid attention to the AVX nonsense over the years and never 
>>> realized that Intel and Gnu (and hence PETSc)  both compile by default for 
>>> machines I used in my twenties.
>>> 
>>>   Expecting PETSc users to automatically add -march= is not realistic.  I 
>>> will try to rig something up in configure where if the user does not 
>>> provide march something reasonable is selected. 
>> A softer (yet trivial to implement) option might also be to just alert the 
>> user that these flags exist in the usual message about using default 
>> optimization flags. Something like this would encourage users to do what Jed 
>> is doing:
>> 
>>   * WARNING: Using default optimization C flags -g -O3
>> You might consider manually setting optimal optimization flags for your 
>> system with
>> COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for 
>> examples. 
>> In particular, you may want to supply specific flags (e.g. -march=native) 
>> to take advantage of higher-performance instructions.
> 
> I think this is a reasonable thing to do.

This is a reasonable message to print on the screen, but I don’t think this is 
a reasonable flag to impose by default.
You are basically asking all package managers to add a new flag 
(-march=generic) which was previously not needed.

I’m crossing my fingers Jed has a clever way of "making portable binaries that 
run-time detected when to use newer instructions where it matters”, because 
-march=native by default is just not practical when deploying software.

Thanks,
Pierre

> We should also inform users that tuning -march options may enable 
> vectorization instructions such as SSE(3 and above) and AVX but generate 
> nonportable binaries.
> 
> If we add -march=native to the configure test, we will need to run 
> executables to make sure the specified instruction sets are supported by the 
> CPU where the code is running. For PETSc, the executables should cover all 
> the intrinsics we use in the code ideally; otherwise, users will get run-time 
> errors when there is a mismatch in vectorization instructions between 
> compiler support and CPU support.
> 
> Hong
> 
>> 
>> None of the examples in config/examples actually use -march=native, and this 
>> is a very common thing to do that, as you point out, isn't obvious until you 
>> know you have to do it, so it seems to be worth the screen space.
>> 
>> 
>> 
>> 
>> 
>>> 
>>>  Barry
>>> 
>>> 
 
 Barry Smith mailto:bsm...@petsc.dev>> writes:
 
> Shouldn't configure be setting something appropriate for this 
> automatically? This is nuts, it means when users do a ./configure make 
> unless they pass weird arguments they sure as heck don't know about to 
> the compiler they won't get any of the glory that they expect and that 
> has been in almost all Intel systems forever.
> 
> Barry
> 
> I run ./configure --with-debugging=0 and I get none of the stuff added by 
> Intel for 15+ years?
> 
> 
>> On Feb 13, 2021, at 11:26 PM, Jed Brown > > wrote:
>> 
>> Use -march=native or similar. The default target is basic x86_64, which 
>> has only SSE2.
>> 
>> Barry Smith mailto:bsm...@petsc.dev>> writes:
>> 
>>> PETSc source has code like defined(__AVX2__) in the source but it does 
>>> not seem to be able to find any of these macros (icc or gcc) on the 
>>> petsc-02 system
>>> 
>>> Are these macros supposed to be defined? How does on get them to be 
>>> defined? Why are they not define? What am I doing wrong?
>>> 
>>> Keep reading
>>> 
>>> $ lscpu 
>>> Architecture:x86_64
>>> CPU op-mode(s):  32-bit, 64-bit
>>> Byte Order:  Little Endian
>>> CPU(s):  64
>>> On-line CPU(s) list: 0-63
>>> Thread(s) per core:  2
>>> Core(s) per socket:  16
>>> Socket(s):   2
>>> NUMA node(s):2

Re: [petsc-dev] Memory problem with OpenMP and Fieldsplit sub solvers

[Pierre Jolivet]

> On 24 Jan 2021, at 4:54 PM, Mark Adams  wrote:
> 
> Hi Sherry, I have this running with OMP, with cuSparse solves (PETSc CPU 
> factorizations)
> 
> Building SuperLU_dist w/o _OPENMP was not easy for me.

Expanding on Sherry’s answer on Thu Jan 21, it should be as easy as adding to 
your configure script 
'--download-superlu-cmake-arguments=-Denable_openmp=FALSE' ' 
--download-superlu_dist-cmake-arguments=-Denable_openmp=FALSE'.
Is that not easy enough, or is it not working?

Thanks,
Pierre

> We need to get a better way to do this. (Satish or Barry?)
> 
> SuperLU works with one thread and two subdomains. With two threads I see this 
> (appended). So this seems to be working in that before it was hanging.
> 
> I set the solver up so that it does not use threads the first time it is 
> called so that solvers can get any lazy allocations done in serial. This is 
> just to be safe in that we do not use a Krylov method here and I don't 
> believe "preonly" allocates any work vectors, and SuperLU does the symbolic 
> factorizations without threads.
> 
> Let me know how you want to proceed.
> 
> Thanks,
> Mark
> 
> ijcusparse -dm_vec_type cuda' NC=2 |g energy
>   0) species-0: charge density= -1.6022862392985e+01 z-momentum= 
> -3.4369550192576e-19 energy=  9.6063873494138e+04
>   0) species-1: charge density=  1.6029950760009e+01 z-momentum= 
> -2.7844197929124e-18 energy=  9.6333444502318e+04
>  0) Total: charge density=  7.0883670236874e-03, momentum= 
> -3.1281152948382e-18, energy=  1.9239731799646e+05 (m_i[0]/m_e = 1835.47, 92 
> cells)
> ex2: 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/externalpackages/git.superlu_dist/SRC/dSchCompUdt-cuda.c:157:
>  pdgstrf: Assertion `jjj-1 [h16n13:21073] *** Process received signal ***
> [h16n13:21073] Signal: Aborted (6)
> [h16n13:21073] Signal code: User function (kill, sigsend, abort, etc.) (0)
> [h16n13:21073] [ 0] [0x200504d8]
> [h16n13:21073] [ 1] /lib64/libc.so.6(abort+0x2b4)[0x200020ef2094]
> [h16n13:21073] [ 2] /lib64/libc.so.6(+0x356d4)[0x200020ee56d4]
> [h16n13:21073] [ 3] /lib64/libc.so.6(__assert_fail+0x64)[0x200020ee57c4]
> [h16n13:21073] [ 4] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libsuperlu_dist.so.6(pdgstrf+0x3848)[0x222fe5d8]
> [h16n13:21073] [ 5] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libsuperlu_dist.so.6(pdgssvx+0x1220)[0x222dc4a8]
> [h16n13:21073] [ 6] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0x9aff28)[0x20a9ff28]
> [h16n13:21073] [ 7] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(MatLUFactorNumeric+0x144)[0x207d273c]
> [h16n13:21073] [ 8] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0xecffc4)[0x20fbffc4]
> [h16n13:21073] [ 9] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(PCSetUp+0x134)[0x2107dd38]
> [h16n13:21073] [10] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPSetUp+0x9f8)[0x210b272c]
> [h16n13:21073] [11] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0xfc46f0)[0x210b46f0]
> [h16n13:21073] [12] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(KSPSolve+0x20)[0x210b6fb8]
> [h16n13:21073] [13] 
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-cuda-omp/lib/libpetsc.so.3.014(+0xf1e4f8)[0x2100e4f8]
> [h16n13:21073] [14] 
> /sw/summit/gcc/6.4.0/lib64/libgomp.so.1(+0x1a51c)[0x200020e2a51c]
> [h16n13:21073] [15] /lib64/libpthread.so.0(+0x8b94)[0x200020e78b94]
> [h16n13:21073] [16] /lib64/libc.so.6(clone+0xe4)[0x200020fd85f4]
> [h16n13:21073] *** End of error message ***
> ERROR:  One or more process (first noticed rank 0) terminated with signal 6 
> (core dumped)
> make: [runasm] Error 134 (ignored)
> 
> On Thu, Jan 21, 2021 at 11:57 PM Xiaoye S. Li  > wrote:
> All the OpenMP calls are surrounded by
> 
> #ifdef _OPENMP
> ...
> #endif
> 
> You can disable openmp during Cmake installation, with the following:
> -Denable_openmp=FALSE
> (the default is true)
> 
> (I think Satish knows how to do this with PETSc installation)
> 
> ---
> The reason to use mixed MPI & OpenMP is mainly less memory consumption, 
> compared to pure MPI.  Timewise probably it is just slightly faster. (I think 
> that's the case with many codes.)
> 
> 
> Sherry
> 
> On Thu, Jan 21, 2021 at 7:20 PM Mark Adams  > wrote:
> 
> 
> On Thu, Jan 21, 2021 at 10:16 PM Barry Smith  > wrote:
> 
> 
>> On Jan 21, 2021, at 9:11 PM, Mark Adams > > wrote:
>> 
>> I have tried it and it hangs, but that is expected. This is not something 
>> she has prepared for.
>> 
>> I am working with Sherry on it.

Re: [petsc-dev] bin/sh error while running tests

[Pierre Jolivet]

Sorry for the noise, I'm now just realizing that it is in fact exactly the same 
bash-ism… (I prefer my “fix” though, but I guess the rest of Stefano’s comment 
still holds true).

Thanks,
Pierre

> On 20 Jan 2021, at 1:59 PM, Pierre Jolivet  wrote:
> 
> 
> 
>> On 20 Jan 2021, at 12:11 PM, Stefano Zampini > <mailto:stefano.zamp...@gmail.com>> wrote:
>> 
>> This is an issue with the default shell used by the makefile. Below is my 
>> fix. We should probably have a CI machine that checks for these 
>> shell-related errors.
> 
> I second this. Just spent too much time finding this other bash-ism in 
> gmakefile.test… 
> https://gitlab.com/petsc/petsc/-/commit/e4b11943e93779206a0e5f2091646de2e86b10e3#551c4017403b9179c385d5600f43348b6288a751
>  
> <https://gitlab.com/petsc/petsc/-/commit/e4b11943e93779206a0e5f2091646de2e86b10e3#551c4017403b9179c385d5600f43348b6288a751>
> 2021-01-20T11:21:22.5942304Z /usr/bin/sh: 1: test: false: unexpected operator
> 2021-01-20T11:21:22.5981176Z make: *** [gmakefile.test:270: 
> check-test-errors] Error 1
> Thanks,
> Pierre
> 
>> diff --git a/gmakefile.test b/gmakefile.test
>> index c38e37f..ffd7bdb 100644
>> --- a/gmakefile.test
>> +++ b/gmakefile.test
>> @@ -379,10 +379,11 @@ starttime: pre-clean $(libpetscall)
>> @$(eval STARTTIME := $(shell date +%s))
>>  
>>  report_tests: starttime $(TESTTARGETS)
>> +ifeq ($(showreport),true)
>> @$(eval ENDTIME := $(shell date +%s))
>> -   -@if test ${showreport} == "true"; then  
>> elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \
>> -   $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d 
>> $(TESTDIR)/counts -t 5 -e $${elapsed_time};\
>> -fi
>> +   elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \
>> +   $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d 
>> $(TESTDIR)/counts -t 5 -e $${elapsed_time};
>> +endif
>> 
>> Il giorno mar 19 gen 2021 alle ore 20:41 Scott Kruger > <mailto:kru...@txcorp.com>> ha scritto:
>> 
>> 
>> I can't reproduce this with the latest master:
>> 
>> hip 1261: git pull
>> Already up to date.
>> hip 1262: make -f gmakefile.test test search='notatest'
>> Using MAKEFLAGS: -- search=notatest
>> hip 1263:
>> 
>> 
>> 
>> On 1/19/21 8:19 AM, Stefano Zampini wrote:
>>> Just rebased over latest master and got this
>>> 
>>> zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test search='notatest'
>>> Using MAKEFLAGS: -- search=notatest
>>> /bin/sh: 1: test: false: unexpected operator
>>> 
>>> -- 
>>> Stefano
>> 
>> -- 
>> Tech-X Corporation   kru...@txcorp.com <mailto:kru...@txcorp.com>
>> 5621 Arapahoe Ave, Suite A   Phone: (720) 466-3196
>> Boulder, CO 80303Fax:   (303) 448-7756
>> 
>> 
>> -- 
>> Stefano
> 

Re: [petsc-dev] bin/sh error while running tests

[Pierre Jolivet]

> On 20 Jan 2021, at 12:11 PM, Stefano Zampini  
> wrote:
> 
> This is an issue with the default shell used by the makefile. Below is my 
> fix. We should probably have a CI machine that checks for these shell-related 
> errors.

I second this. Just spent too much time finding this other bash-ism in 
gmakefile.test… 
https://gitlab.com/petsc/petsc/-/commit/e4b11943e93779206a0e5f2091646de2e86b10e3#551c4017403b9179c385d5600f43348b6288a751
 

2021-01-20T11:21:22.5942304Z /usr/bin/sh: 1: test: false: unexpected operator
2021-01-20T11:21:22.5981176Z make: *** [gmakefile.test:270: check-test-errors] 
Error 1
Thanks,
Pierre

> diff --git a/gmakefile.test b/gmakefile.test
> index c38e37f..ffd7bdb 100644
> --- a/gmakefile.test
> +++ b/gmakefile.test
> @@ -379,10 +379,11 @@ starttime: pre-clean $(libpetscall)
> @$(eval STARTTIME := $(shell date +%s))
>  
>  report_tests: starttime $(TESTTARGETS)
> +ifeq ($(showreport),true)
> @$(eval ENDTIME := $(shell date +%s))
> -   -@if test ${showreport} == "true"; then  elapsed_time=$$(($(ENDTIME)- 
> $(STARTTIME))) && \
> -   $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d 
> $(TESTDIR)/counts -t 5 -e $${elapsed_time};\
> -fi
> +   elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \
> +   $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d 
> $(TESTDIR)/counts -t 5 -e $${elapsed_time};
> +endif
> 
> Il giorno mar 19 gen 2021 alle ore 20:41 Scott Kruger  > ha scritto:
> 
> 
> I can't reproduce this with the latest master:
> 
> hip 1261: git pull
> Already up to date.
> hip 1262: make -f gmakefile.test test search='notatest'
> Using MAKEFLAGS: -- search=notatest
> hip 1263:
> 
> 
> 
> On 1/19/21 8:19 AM, Stefano Zampini wrote:
>> Just rebased over latest master and got this
>> 
>> zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test search='notatest'
>> Using MAKEFLAGS: -- search=notatest
>> /bin/sh: 1: test: false: unexpected operator
>> 
>> -- 
>> Stefano
> 
> -- 
> Tech-X Corporation   kru...@txcorp.com 
> 5621 Arapahoe Ave, Suite A   Phone: (720) 466-3196
> Boulder, CO 80303Fax:   (303) 448-7756
> 
> 
> -- 
> Stefano

Re: [petsc-dev] builds of PETSc based packages should not get their own secret files

[Pierre Jolivet]

> On 5 Jan 2021, at 1:33 AM, Barry Smith  wrote:
> 
> 
>  For packages that are built after PETSc configure (and or install is done) 
> slepc, hpddm etc we've traditional saved the output  in its own file stashed 
> away somewhere. 
> 
>  For the CI this is driving me nuts because when they fail the output is 
> essentially "lost" and thus it is impossible to determine what has gone 
> wrong. 
> 
>  I have started to directly output in the same stream as the PETSc compiles 
> to make debugging much easier. Generally the packages are relatively small 
> and don't have a huge amount of output when compiling correctly.  I did it 
> for PETSc4py and SLEPc (slepc4py is a mystery yet how it get's hidden in 
> slepc). 

I guess we could change the redirect rule here 
https://gitlab.com/slepc/slepc/-/blob/master/config/packages/slepc4py.py#L53 
?
But we’d need to check whether slepc4py is built with --download-slepc 
--download-slepc-configure-arguments="--download-slepc4py” (inside PETSc) or 
simply --download-slepc4py (inside SLEPc).

I’m in favour of having a single file because it can be quite nightmarish to 
ask users for multiple .log files hidden in different folders, but I can 
understand if we stick with the current approach as well.

Thanks,
Pierre

>  Are there any large downsides to this plan?
> 
>  Barry
> 
> 

[petsc-dev] Documentation version mismatch?

[Pierre Jolivet]

Hello,
https://docs.petsc.org/en/latest/  shows 
"PETSc 3.14.0 documentation” in the header and footer.
Shouldn’t it either display 3.14 or 3.14.2?

Thanks,
Pierre

Re: [petsc-dev] Can't find test matrix

[Pierre Jolivet]

Here you go: 
https://gitlab.com/petsc/datafiles/-/tree/master/matrices/hpddm/GCRODR 


Thanks,
Pierre

> On 18 Dec 2020, at 6:02 PM, Matthew Knepley  wrote:
> 
> I need this one
> 
> /PETSc3/petsc/petsc-dev/../datafiles/matrices/hpddm/GCRODR/A_400.dat
> 
> but I cannot find it on the FTP site.
> 
>   Thanks,
> 
>  Matt
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] Building PETSc on LLNL Lassen

[Pierre Jolivet]

Don’t know if this still applies, but here is what Jed said about this 10 
months ago (https://gitlab.com/petsc/petsc/-/merge_requests/2466#note_275686270 
):
“A minor issue is that -ffp-contract=off is implied by -std=c*, versus 
-ffp-contract=fast being default otherwise (with GCC).”

Thanks,
Pierre

> On 14 Dec 2020, at 9:35 AM, Stefano Zampini  wrote:
> 
> While we are discussing this dialect stuff, do we still want to test for 
> gnu++{11|14} extensions before testing for c++{11|14} ? We get warnings when 
> using KOKKOS since it replaces gnu++14 with c++14. What is the added value of 
> using gnu++{11|14}?
> 
> Il giorno lun 14 dic 2020 alle ore 08:21 Barry Smith  > ha scritto:
> 
>   So two configure sins. Setting values running configure without logging 
> them into configure.log and setting values without testing them with 
> configure tests :-)
> 
>   We need to finally deal properly with setting -cbin in configure; just 
> defaulting causes all kinds of undebuggable problems. Why can't we also just 
> set it to the C++ compiler? Is it because it cannot handle the MPI c++ 
> compiler so it needs to be set to the underlying C++ compiler?
> 
>   Barry
> 
>   It is also strange that Jacob's configure.log seems to believe the IBM 
> compiler is a GNU compiler (but because of poor logging configure does not 
> log this information; fix coming in a MR soon).
> 
>> On Dec 13, 2020, at 6:32 PM, Junchao Zhang > > wrote:
>> 
>> Jacob,
>>   Do you need to add  'CUDAFLAGS=-ccbin xlc++' to specify the host compiler 
>> for CUDA? Note in cuda.py I added
>> 
>> if self.compilers.cxxdialect in ['C++11','C++14']: #nvcc is a C++ 
>> compiler so it is always good to add -std=xxx. It is even crucial when using 
>> thrust complex (see MR 2822)
>>   self.setCompilers.CUDAFLAGS += ' -std=' + 
>> self.compilers.cxxdialect.lower()
>> 
>>  In your configure.log, there are 
>> #define PETSC_HAVE_CXX_DIALECT_CXX11 1
>> #define PETSC_HAVE_CXX_DIALECT_CXX14 1
>> 
>> I guess without -ccbin, nvcc uses gcc by default and your gcc does not 
>> support C++14.
>> 
>> --Junchao Zhang
>> 
>> 
>> On Sun, Dec 13, 2020 at 1:25 PM Jacob Faibussowitsch > > wrote:
>> Hello All,
>> 
>> Does anyone have any experience building petsc with cuda support on Lassen? 
>> I’ve been having trouble building with ibm xl compilers + spectrum-mpi + 
>> nvcc. NVCC seems to not like -std=c++14 argument, complaining that its 
>> configured host compiler doesn’t support it, but compiling the following 
>> “test.cc ":
>> 
>> #include 
>> 
>> int main(int argc, char **argv)
>> {
>>
>>   int i = 1;
>>   i += argc;
>>   return(i);
>> }
>> 
>> With mpicc -std=c++14 test.cc  produces zero errors. 
>> 
>> 
>> Modules loaded:
>> 
>> module load xl/2020.11.12-cuda-11.1.1
>>
>> module load spectrum-mpi
>> module load cuda/11.1.1
>> module load python/3.8.2
>> module load cmake
>> module load valgrind
>> module load lapack
>> 
>> My configure commands:
>> 
>> ./configure  --with-cc=mpicc --with-cxx=mpiCC --with-fc=mpifort --with-cuda 
>> --with-debugging=1 PETSC_ARCH=arch-linux-c-debug
>> 
>> The error:
>> 
>> TESTING: findMPIInc from 
>> config.packages.MPI(config/BuildSystem/config/packages/MPI.py:636) 
>> ***
>>  UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
>> details):
>> ---
>> Bad compiler flag: 
>> -I/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/include
>> ***
>> 
>> The actual configure.log error:
>> 
>> Executing: nvcc -c -o /var/tmp/petsc-2v0k4k61/config.setCompilers/conftest.o 
>> -I/var/tmp/petsc-2v0k4\
>> k61/config.setCompilers -I/var/tmp/petsc-2v0k4k61/config.types  -g 
>> -std=c++14 -I/usr/tce/packages/s\
>> pectrum-mpi/ibm/spectrum-mpi-rolling-release/include  
>> -Wno-deprecated-gpu-targets /var/tmp/petsc-2v\
>> 0k4k61/config.setCompilers/conftest.cu  
>> Possible ERROR while running compiler:
>> stderr:
>> nvcc warning : The -std=c++14 flag is not supported with the configured host 
>> compiler. Flag will be\
>>  ignored.
>> Source:
>> #include "confdefs.h"
>> #include "conffix.h"
>> 
>> int main() {
>> ;
>>   return 0;
>> }
>>   Rejecting compiler flag 
>> -I/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/include  
>> due to 
>> nvcc warning : The -std=c++14 flag is not supported with the 

Re: [petsc-dev] PETSc always recompiles mpi4py petsc4py even when it does not have too

[Pierre Jolivet]

This actually applies to all packages with builtafterpetsc = 1.
I think you want to do make libs instead of make all?

Thanks,
Pierre

> On 11 Oct 2020, at 9:36 AM, Barry Smith  wrote:
> 
> 
>   I change a single PETSc file and do make all that single file gets compiled 
> but all of mpi4py and petsc4py get recompiled even though I have not changed 
> any thing to affect them. No include files for example
> 
> 
> *** Building mpi4py ***
> *** Installing mpi4py ***
> =
> To use mpi4py, add 
> /Users/barrysmith/Src/petsc/arch-cleanup-dmd-domaindecomposition/lib to 
> PYTHONPATH
> =
> *** Building petsc4py ***
> 

Re: [petsc-dev] PDIPDM questions

[Pierre Jolivet]

art,,,NULL);CHKERRQ(ierr);
>   }
> 
>   /* 2nd Row block of KKT matrix: [grad Ce, 0, 0, 0] */
>   if (pdipm->Ng) {
> ierr = 
> MatGetOwnershipRange(tao->jacobian_equality,,NULL);CHKERRQ(ierr);
> for (i=0; i < pdipm->ng; i++){
>   row = rstart + pdipm->off_lambdae + i;
> 
>   ierr = 
> MatGetRow(tao->jacobian_equality,i+rjstart,,,NULL);CHKERRQ(ierr);
>   proc = 0;
>   for (j=0; j < nc; j++) {
> while (aj[j] >= cranges[proc+1]) proc++;
> col = aj[j] - cranges[proc] + Jranges[proc];
> ierr = MatPreallocateSet(row,1,,dnz,onz);CHKERRQ(ierr); /* grad g 
> */
>   }
>   ierr = 
> MatRestoreRow(tao->jacobian_equality,i+rjstart,,,NULL);CHKERRQ(ierr);
> }
>   }
>   /* Jce_xfixed */
>   if (pdipm->Nxfixed) {
> ierr = 
> MatGetOwnershipRange(pdipm->Jce_xfixed,,NULL);CHKERRQ(ierr);
> for (i=0; i < (pdipm->nce - pdipm->ng); i++){
>   row = rstart + pdipm->off_lambdae + pdipm->ng + i;
> 
>   ierr = 
> MatGetRow(pdipm->Jce_xfixed,i+Jcrstart,,,NULL);CHKERRQ(ierr);
>   if (nc != 1) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"nc != 1");
> 
>   proc = 0;
>   j= 0;
>   while (cols[j] >= cranges[proc+1]) proc++;
>   col = cols[j] - cranges[proc] + Jranges[proc];
>   ierr = MatPreallocateSet(row,1,,dnz,onz);CHKERRQ(ierr);
>   ierr = 
> MatRestoreRow(pdipm->Jce_xfixed,i+Jcrstart,,,NULL);CHKERRQ(ierr);
> }
>   }
> 
>   /* 3rd Row block of KKT matrix: [ gradCi, 0, 0, -I] */
>   if (pdipm->Nh) {
> ierr = 
> MatGetOwnershipRange(tao->jacobian_inequality,,NULL);CHKERRQ(ierr);
> for (i=0; i < pdipm->nh; i++){
>   row = rstart + pdipm->off_lambdai + i;
> 
>   ierr = 
> MatGetRow(tao->jacobian_inequality,i+rjstart,,,NULL);CHKERRQ(ierr);
>   proc = 0;
>   for (j=0; j < nc; j++) {
> while (aj[j] >= cranges[proc+1]) proc++;
> col = aj[j] - cranges[proc] + Jranges[proc];
> ierr = MatPreallocateSet(row,1,,dnz,onz);CHKERRQ(ierr); /* grad h 
> */
>   }
>   ierr = 
> MatRestoreRow(tao->jacobian_inequality,i+rjstart,,,NULL);CHKERRQ(ierr);
> }
> /* -I */
> for (i=0; i < pdipm->nh; i++){
>   row = rstart + pdipm->off_lambdai + i;
>   col = rstart + pdipm->off_z + i;
>   ierr = MatPreallocateSet(row,1,,dnz,onz);CHKERRQ(ierr);
> }
>   }
> 
>   /* Jci_xb */
>   ierr = MatGetOwnershipRange(pdipm->Jci_xb,,NULL);CHKERRQ(ierr);
>   for (i=0; i < (pdipm->nci - pdipm->nh); i++){
> row = rstart + pdipm->off_lambdai + pdipm->nh + i;
> 
> ierr = MatGetRow(pdipm->Jci_xb,i+Jcrstart,,,NULL);CHKERRQ(ierr);
> if (nc != 1) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"nc != 1");
> proc = 0;
> for (j=0; j < nc; j++) {
>   while (cols[j] >= cranges[proc+1]) proc++;
>   col = cols[j] - cranges[proc] + Jranges[proc];
>   ierr = MatPreallocateSet(row,1,,dnz,onz);CHKERRQ(ierr);
> }
> ierr = 
> MatRestoreRow(pdipm->Jci_xb,i+Jcrstart,,,NULL);CHKERRQ(ierr);
> /* -I */
> col = rstart + pdipm->off_z + pdipm->nh + i;
> ierr = MatPreallocateSet(row,1,,dnz,onz);CHKERRQ(ierr);
>   }
> 
>   /* 4-th Row block of KKT matrix: Z and Ci */
>   for (i=0; i < pdipm->nci; i++) {
> row = rstart + pdipm->off_z + i;
> cols1[0] = rstart + pdipm->off_lambdai + i;
> cols1[1] = row;
> ierr = MatPreallocateSet(row,2,cols1,dnz,onz);CHKERRQ(ierr);
>   }
> 
> 
>> On Sep 15, 2020, at 4:03 PM, Abhyankar, Shrirang G 
>> mailto:shrirang.abhyan...@pnnl.gov>> wrote:
>> 
>> There is some code in PDIPM that copies over matrix elements from user 
>> jacobians/hessians to the big KKT matrix. I am not sure how/if that will 
>> work with MatMPI_Column. We’ll have to try out and we need an example for 
>> that 
>>  
>> Thanks,
>> Shri 
>>  
>>  
>> From: Barry Smith mailto:bsm...@petsc.dev>>
>> Date: Tuesday, September 15, 2020 at 1:21 PM
>> To: Pierre Jolivet > <mailto:pierre.joli...@enseeiht.fr>>
>> Cc: "Abhyankar, Shrirang G" > <mailto:shrirang.abhyan...@pnnl.gov>>, PETSc Development 
>> mailto:petsc-dev@mcs.anl.gov>>
>> Subject: Re: [petsc-dev] PDIPDM questions
>>  
>>  
>>   Pierre,
>>  
>> Based on my previous mail I am hoping that the PDIPM algorithm itself 
>> won't need a major refactorization to be scalable, only a custom matrix type 
>> is needed to store and compute with the  Hessian in a scalable 

Re: [petsc-dev] PDIPDM questions

[Pierre Jolivet]

> On 15 Sep 2020, at 9:25 PM, Matthew Knepley  wrote:
> 
> On Tue, Sep 15, 2020 at 3:05 PM Pierre Jolivet  <mailto:pierre.joli...@enseeiht.fr>> wrote:
> Thank you Barry for the very comprehensive answer, this gives me hope that 
> I’ll indeed be able to help our Ipopt users transition to Tao when this is in 
> place/if this gets done!
> Shri, my Hessian and my x are currently distributed (really standard 
> distribution, nothing fancy) using the same layout.
> I _need_ to centralize the Jacobian on rank 0, because there is currently no 
> way to distribute a matrix with a single row on more than one rank.
> So in this scenario, PDIPM is indeed not usable, cf. the previous trace.
> When you say: "The column layout of the equality/inequality Jacobian is same 
> as that for x”, this is currently not achievable with a distributed x and a 
> single equality/inequality Jacobian.
> The two fixes I see:
> 1) provide the transpose of the Jacobian (of dimension N x 1, so can be split 
> row-wise no problem, but this may have side effect I’m not anticipating)
> 
> This sounds like the easiest and most natural thing.

I think I’m opening a can of worms here…
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Mat type transpose
[0]PETSC ERROR: #1 MatTranspose() line 4946 in 
/Users/jolivet/Documents/repositories/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 TaoSetup_PDIPM() line 972 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c
[0]PETSC ERROR: #3 TaoSetUp() line 269 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/interface/taosolver.c
[0]PETSC ERROR: #4 TaoSolve() line 215 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/interface/taosolver.c

Trivial to fix by hardwiring a MatDuplicate() in 
src/mat/impls/transpose/transm.c, but this may not be so easy after all.

Thanks,
Pierre

>   Thanks,
> 
>   Matt
>  
> 2) provide the Jacobian as a Mat_MPIColumn (for lack of a better name, to 
> quote Barry) 
> 
> Thanks,
> Pierre
> 
>> On 15 Sep 2020, at 8:18 PM, Barry Smith > <mailto:bsm...@petsc.dev>> wrote:
>> 
>> 
>>   Pierre,
>> 
>> Based on my previous mail I am hoping that the PDIPM algorithm itself 
>> won't need a major refactorization to be scalable, only a custom matrix type 
>> is needed to store and compute with the  Hessian in a scalable way.
>> 
>>Barry
>> 
>> 
>>> On Sep 15, 2020, at 12:50 PM, Pierre Jolivet >> <mailto:pierre.joli...@enseeiht.fr>> wrote:
>>> 
>>> 
>>> 
>>>> On 15 Sep 2020, at 5:40 PM, Abhyankar, Shrirang G 
>>>> mailto:shrirang.abhyan...@pnnl.gov>> wrote:
>>>> 
>>>> Pierre,
>>>>You are right. There are a few MatMultTransposeAdd that may need 
>>>> conforming layouts for the equality/inequality constraint vectors and 
>>>> equality/inequality constraint Jacobian matrices. I need to check if 
>>>> that’s the case. We only have ex1 example currently, we need to add more 
>>>> examples. We are currently working on making PDIPM robust and while doing 
>>>> it will work on adding another example.
>>>>  
>>>> Very naive question, but given that I have a single constraint, how do I 
>>>> split a 1 x N matrix column-wise? I thought it was not possible.
>>>>  
>>>> When setting the size of the constraint vector, you need to set the local 
>>>> size on one rank to 1 and all others to zero. For the Jacobian, the local 
>>>> row size on that rank will be 1 and all others to zero. The column layout 
>>>> for the Jacobian should follow the layout for vector x. So each rank will 
>>>> set the local column size of the Jacobian to local size of x.
>>> 
>>> That is assuming I don’t want x to follow the distribution of the Hessian, 
>>> which is not my case.
>>> Is there some plan to make PDIPM handle different layouts?
>>> I hope I’m not the only one thinking that having a centralized Hessian when 
>>> there is a single constraint is not scalable?
>>> 
>>> Thanks,
>>> Pierre
>>> 
>>>> Shri
>>>>  
>>>>> On 15 Sep 2020, at 2:21 AM, Abhyankar, Shrirang G 
>>>>> mailto:shrirang.abhyan...@pnnl.gov>> wrote:
>>>>>  
>>>>> Hello Pierre,
>>>>>PDIPM works in parallel so you can have distributed Hessian, 
>>>>> Jacobians, cons

Re: [petsc-dev] PDIPDM questions

[Pierre Jolivet]

Thank you Barry for the very comprehensive answer, this gives me hope that I’ll 
indeed be able to help our Ipopt users transition to Tao when this is in 
place/if this gets done!
Shri, my Hessian and my x are currently distributed (really standard 
distribution, nothing fancy) using the same layout.
I _need_ to centralize the Jacobian on rank 0, because there is currently no 
way to distribute a matrix with a single row on more than one rank.
So in this scenario, PDIPM is indeed not usable, cf. the previous trace.
When you say: "The column layout of the equality/inequality Jacobian is same as 
that for x”, this is currently not achievable with a distributed x and a single 
equality/inequality Jacobian.
The two fixes I see:
1) provide the transpose of the Jacobian (of dimension N x 1, so can be split 
row-wise no problem, but this may have side effect I’m not anticipating)
2) provide the Jacobian as a Mat_MPIColumn (for lack of a better name, to quote 
Barry) 

Thanks,
Pierre

> On 15 Sep 2020, at 8:18 PM, Barry Smith  wrote:
> 
> 
>   Pierre,
> 
> Based on my previous mail I am hoping that the PDIPM algorithm itself 
> won't need a major refactorization to be scalable, only a custom matrix type 
> is needed to store and compute with the  Hessian in a scalable way.
> 
>Barry
> 
> 
>> On Sep 15, 2020, at 12:50 PM, Pierre Jolivet > <mailto:pierre.joli...@enseeiht.fr>> wrote:
>> 
>> 
>> 
>>> On 15 Sep 2020, at 5:40 PM, Abhyankar, Shrirang G 
>>> mailto:shrirang.abhyan...@pnnl.gov>> wrote:
>>> 
>>> Pierre,
>>>You are right. There are a few MatMultTransposeAdd that may need 
>>> conforming layouts for the equality/inequality constraint vectors and 
>>> equality/inequality constraint Jacobian matrices. I need to check if that’s 
>>> the case. We only have ex1 example currently, we need to add more examples. 
>>> We are currently working on making PDIPM robust and while doing it will 
>>> work on adding another example.
>>>  
>>> Very naive question, but given that I have a single constraint, how do I 
>>> split a 1 x N matrix column-wise? I thought it was not possible.
>>>  
>>> When setting the size of the constraint vector, you need to set the local 
>>> size on one rank to 1 and all others to zero. For the Jacobian, the local 
>>> row size on that rank will be 1 and all others to zero. The column layout 
>>> for the Jacobian should follow the layout for vector x. So each rank will 
>>> set the local column size of the Jacobian to local size of x.
>> 
>> That is assuming I don’t want x to follow the distribution of the Hessian, 
>> which is not my case.
>> Is there some plan to make PDIPM handle different layouts?
>> I hope I’m not the only one thinking that having a centralized Hessian when 
>> there is a single constraint is not scalable?
>> 
>> Thanks,
>> Pierre
>> 
>>> Shri
>>>  
>>>> On 15 Sep 2020, at 2:21 AM, Abhyankar, Shrirang G 
>>>> mailto:shrirang.abhyan...@pnnl.gov>> wrote:
>>>>  
>>>> Hello Pierre,
>>>>PDIPM works in parallel so you can have distributed Hessian, Jacobians, 
>>>> constraints, variables, gradients in any layout you want.  If you are 
>>>> using a DM then you can have it generate the Hessian. 
>>>  
>>> Could you please show an example where this is the case?
>>> pdipm->x, which I’m assuming is a working vector, is both used as input for 
>>> Hessian and Jacobian functions, e.g., 
>>> https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L369
>>>  
>>> <https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L369>
>>>  (Hessian) + 
>>> https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L473
>>>  
>>> <https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L473>
>>>  (Jacobian)
>>> I thus doubt that it is possible to have different layouts?
>>> In practice, I end up with the following error when I try this (2 
>>> processes, distributed Hessian with centralized Jacobian):
>>> [1]PETSC ERROR: - Error Message 
>>> --
>>> [1]PETSC ERROR: Nonconforming object sizes
>>> [1]PETSC ERROR: Vector wrong size 14172 for scatter 0 (scatter reverse and 
>>> vector to != ctx from size)
>>> [1]PETSC ERROR: #1 VecScatterBegin() line 96 in 
>>> /Users/jolivet/Document

Re: [petsc-dev] PDIPDM questions

[Pierre Jolivet]

nts across ranks.
>> · The layout of vectors x, bounds on x, and gradient is same.
>> · The row layout of the equality/inequality Jacobian is same as the 
>> equality/inequality constraint vector layout.
>> · The column layout of the equality/inequality Jacobian is same as 
>> that for x.
>  
> Very naive question, but given that I have a single constraint, how do I 
> split a 1 x N matrix column-wise? I thought it was not possible.
>  
> Thanks,
> Pierre
> 
> 
>> · The row and column layout for the Hessian is same as x.
>>  
>> The tutorial example ex1 is extremely small (only 2 variables) so its 
>> implementation is very simplistic. I think, in parallel, it ships off 
>> constraints etc. to rank 0. It’s not an ideal example w.r.t demonstrating a 
>> parallel implementation. We aim to add more examples as we develop PDIPM. If 
>> you have an example to contribute then we would most welcome it and provide 
>> help on adding it.
>>  
>> Thanks,
>> Shri
>> From: petsc-dev > <mailto:petsc-dev-boun...@mcs.anl.gov>> on behalf of Pierre Jolivet 
>> mailto:pierre.joli...@enseeiht.fr>>
>> Date: Monday, September 14, 2020 at 1:52 PM
>> To: PETSc Development mailto:petsc-dev@mcs.anl.gov>>
>> Subject: [petsc-dev] PDIPDM questions
>>  
>> Hello,
>> In my quest to help users migrate from Ipopt to Tao, I’ve a new question.
>> When looking at src/tao/constrained/tutorials/ex1.c, it seems that almost 
>> everything is centralized on rank 0 (local sizes are 0 but on rank 0).
>> I’d like to have my Hessian distributed more naturally, as in (almost?) all 
>> other SNES/TS examples, but still keep the Jacobian of my equality 
>> constraint, which is of dimension 1 x N (N >> 1), centralized on rank 0.
>> Is this possible?
>> If not, is it possible to supply the transpose of the Jacobian, of dimension 
>> N x 1, which could then be distributed row-wise like the Hessian?
>> Or maybe use some trick to distribute a MatAIJ/MatDense of dimension 1 x N 
>> column-wise? Use a MatNest with as many blocks as processes?
>>  
>> So, just to sum up, how can I have a distributed Hessian with a Jacobian 
>> with a single row?
>>  
>> Thanks in advance for your help,
>> Pierre

Re: [petsc-dev] PDIPDM questions

[Pierre Jolivet]

> On 15 Sep 2020, at 2:21 AM, Abhyankar, Shrirang G 
>  wrote:
> 
> Hello Pierre,
>PDIPM works in parallel so you can have distributed Hessian, Jacobians, 
> constraints, variables, gradients in any layout you want.  If you are using a 
> DM then you can have it generate the Hessian. 

Could you please show an example where this is the case?
pdipm->x, which I’m assuming is a working vector, is both used as input for 
Hessian and Jacobian functions, e.g., 
https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L369
 
<https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L369>
 (Hessian) + 
https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L473
 
<https://gitlab.com/petsc/petsc/-/blob/master/src/tao/constrained/impls/ipm/pdipm.c#L473>
 (Jacobian)
I thus doubt that it is possible to have different layouts?
In practice, I end up with the following error when I try this (2 processes, 
distributed Hessian with centralized Jacobian):
[1]PETSC ERROR: - Error Message 
--
[1]PETSC ERROR: Nonconforming object sizes
[1]PETSC ERROR: Vector wrong size 14172 for scatter 0 (scatter reverse and 
vector to != ctx from size)
[1]PETSC ERROR: #1 VecScatterBegin() line 96 in 
/Users/jolivet/Documents/repositories/petsc/src/vec/vscat/interface/vscatfce.c
[1]PETSC ERROR: #2 MatMultTransposeAdd_MPIAIJ() line 1223 in 
/Users/jolivet/Documents/repositories/petsc/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: #3 MatMultTransposeAdd() line 2648 in 
/Users/jolivet/Documents/repositories/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Vector wrong size 13790 for scatter 27962 (scatter reverse and 
vector to != ctx from size)
[1]PETSC ERROR: #4 TaoSNESFunction_PDIPM() line 510 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c
[0]PETSC ERROR: #5 TaoSolve_PDIPM() line 712 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c
[1]PETSC ERROR: #6 TaoSolve() line 222 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/interface/taosolver.c
[0]PETSC ERROR: #1 VecScatterBegin() line 96 in 
/Users/jolivet/Documents/repositories/petsc/src/vec/vscat/interface/vscatfce.c
[0]PETSC ERROR: #2 MatMultTransposeAdd_MPIAIJ() line 1223 in 
/Users/jolivet/Documents/repositories/petsc/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #3 MatMultTransposeAdd() line 2648 in 
/Users/jolivet/Documents/repositories/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 TaoSNESFunction_PDIPM() line 510 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c
[0]PETSC ERROR: #5 TaoSolve_PDIPM() line 712 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/constrained/impls/ipm/pdipm.c
[0]PETSC ERROR: #6 TaoSolve() line 222 in 
/Users/jolivet/Documents/repositories/petsc/src/tao/interface/taosolver.c

I think this can be reproduced by ex1.c by just distributing the Hessian 
instead of having it centralized on rank 0.

> Ideally, you want to have the layout below to minimize movement of 
> matrix/vector elements across ranks.
> The layout of vectors x, bounds on x, and gradient is same.
> The row layout of the equality/inequality Jacobian is same as the 
> equality/inequality constraint vector layout.
> The column layout of the equality/inequality Jacobian is same as that for x.

Very naive question, but given that I have a single constraint, how do I split 
a 1 x N matrix column-wise? I thought it was not possible.

Thanks,
Pierre

> The row and column layout for the Hessian is same as x.
>  
> The tutorial example ex1 is extremely small (only 2 variables) so its 
> implementation is very simplistic. I think, in parallel, it ships off 
> constraints etc. to rank 0. It’s not an ideal example w.r.t demonstrating a 
> parallel implementation. We aim to add more examples as we develop PDIPM. If 
> you have an example to contribute then we would most welcome it and provide 
> help on adding it.
>  
> Thanks,
> Shri
> From: petsc-dev  on behalf of Pierre Jolivet 
> 
> Date: Monday, September 14, 2020 at 1:52 PM
> To: PETSc Development 
> Subject: [petsc-dev] PDIPDM questions
>  
> Hello,
> In my quest to help users migrate from Ipopt to Tao, I’ve a new question.
> When looking at src/tao/constrained/tutorials/ex1.c, it seems that almost 
> everything is centralized on rank 0 (local sizes are 0 but on rank 0).
> I’d like to have my Hessian distributed more naturally, as in (almost?) all 
> other SNES/TS examples, but still keep the Jacobian of my equality 
> constraint, which is of dimension 1 x N (N >> 1), centralized on rank 0.
> Is this possible?
> If not, is it possible to supply the transpose of the Jacobian, o

[petsc-dev] PDIPDM questions

[Pierre Jolivet]

Hello,
In my quest to help users migrate from Ipopt to Tao, I’ve a new question.
When looking at src/tao/constrained/tutorials/ex1.c, it seems that almost 
everything is centralized on rank 0 (local sizes are 0 but on rank 0).
I’d like to have my Hessian distributed more naturally, as in (almost?) all 
other SNES/TS examples, but still keep the Jacobian of my equality constraint, 
which is of dimension 1 x N (N >> 1), centralized on rank 0.
Is this possible?
If not, is it possible to supply the transpose of the Jacobian, of dimension N 
x 1, which could then be distributed row-wise like the Hessian?
Or maybe use some trick to distribute a MatAIJ/MatDense of dimension 1 x N 
column-wise? Use a MatNest with as many blocks as processes?

So, just to sum up, how can I have a distributed Hessian with a Jacobian with a 
single row?

Thanks in advance for your help,
Pierre

Re: [petsc-dev] TAOPDIPM

[Pierre Jolivet]

Sorry for these questions, I’m trying to help the said user transitioning from 
IPOPT over to Tao, and since I’m no expert I run into a bunch of errors myself.
For example, if I run PDIPM without equality constraint, I now get:
[0]PETSC ERROR: VecSet() line 522 in src/vec/vec/interface/rvector.c Null 
Object: Parameter # 1
Indeed, I can see that lambdae is conditionally created line 815, but always 
used in VecSet() line 522.
I think the method is general enough to handle cases without such constraints, 
and should not thus fail here, right?
You should be able to reproduce this by commenting all 
DE/De/EqualityConstraints/JacobianEquality code from ex1.c.

Thanks,
Pierre

> On 5 Aug 2020, at 5:27 AM, Abhyankar, Shrirang G 
>  wrote:
> 
> Pierre,
>   Thanks for catching this issue. As Alp pointed out, the issue was because 
> the tolerance was not set correctly.
>  
> @Alp: Let me know if I can help you with the patch.
>  
> Thanks,
> Shri 
>  
>  
> From: petsc-dev  on behalf of Pierre Jolivet 
> 
> Date: Tuesday, August 4, 2020 at 1:46 PM
> To: "Dener, Alp" 
> Cc: PETSc Development 
> Subject: Re: [petsc-dev] TAOPDIPM
>  
> Thanks to you both, I’ll forward this to my user.
> Pierre
>  
>> On 4 Aug 2020, at 7:30 PM, Dener, Alp > <mailto:ade...@anl.gov>> wrote:
>>  
>> Hi Pierre,
>>  
>> This is indeed an issue with TAO default tolerances.
>>  
>> Specifically it has to do with constraint tolerances. The default value is 
>> an exact zero.
>>  
>> The problem should still work with line 89 commented out. Our default 
>> gradient tolerance is 1e-8. In your case, commenting out line 90 is causing 
>> the solver to try to converge constraints to exact zero and it cannot.
>>  
>> I will get a patch out for this within the week but in the meantime please 
>> ensure that the constraint tolerances are set for any constrained problem.
>>  
>> Thank you!
>> —
>> Alp
>>  
>>> On Aug 4, 2020, at 12:24 PM, Munson, Todd via petsc-dev 
>>> mailto:petsc-dev@mcs.anl.gov>> wrote:
>>>  
>>>  
>>> Hi Pierre,
>>>  
>>> I would say the answer to that question is "no, its not expected".  
>>>  
>>> We will look into fixing it.  It seems like the default tolerances are 
>>> being set to zero and the result is an inability to satisfy the constraints 
>>> or gradient of the Lagrangian to that small of a tolerance.
>>>  
>>> Thanks for point this out to us!  We will get it resolved.
>>>  
>>> Thanks, Todd.
>>>  
>>> On 8/4/20, 12:12 PM, "petsc-dev on behalf of Pierre Jolivet" 
>>> mailto:petsc-dev-boun...@mcs.anl.gov> on 
>>> behalf of pierre.joli...@enseeiht.fr <mailto:pierre.joli...@enseeiht.fr>> 
>>> wrote:
>>>  
>>>Hello,
>>>If I comment line 89 and 90 of src/tao/constrained/tutorials/ex1.c from 
>>> master, the example deadlocks with a single process.
>>>Is this expected?
>>>  
>>>Thanks,
>>>Pierre
>>>  
>>>$  ./ex1 -tao_monitor
>>> Constrained Problem -
>>>Solution should be f(1,1)=-2
>>>  0 TAO,  Function value: 8.,  Residual: 8.48528
>>>  1 TAO,  Function value: 1.28532,  Residual: 10.7411
>>>  2 TAO,  Function value: -2.56703,  Residual: 1.87847
>>>  3 TAO,  Function value: -2.03161,  Residual: 0.12881
>>>  4 TAO,  Function value: -1.99961,  Residual: 0.0450855
>>>  5 TAO,  Function value: -1.3,  Residual: 0.000916939
>>>  6 TAO,  Function value: -1.9,  Residual: 6.69184e-05
>>>  7 TAO,  Function value: -2.,  Residual: 7.15427e-06
>>>  8 TAO,  Function value: -2.,  Residual: 7.15779e-07
>>>  9 TAO,  Function value: -2.,  Residual: 7.15777e-08
>>> 10 TAO,  Function value: -2.,  Residual: 7.15777e-09
>>> 11 TAO,  Function value: -2.,  Residual: 7.15779e-10
>>> 12 TAO,  Function value: -2.,  Residual: 7.15775e-11
>>> 13 TAO,  Function value: -2.,  Residual: 7.1599e-12
>>> 14 TAO,  Function value: -2.,  Residual: 7.1599e-13
>>> 15 TAO,  Function value: -2.,  Residual: 7.22085e-14
>>> 16 TAO,  Function value: -2.,  Residual: 6.44626e-15
>>> 17 TAO,  Function value: -2.,  Residual: 1.00751e-15
>>> 18 TAO,  Function value: -2.,  Residual: 1.70295e-17
>>> 19 TAO,  Function value: -2.,  Residual: 1.70295e-18
>>> 20 TAO,  Function value: -2.,  Residual: 1.70295e-19
>>> 21 

Re: [petsc-dev] TAOPDIPM

[Pierre Jolivet]

Thanks to you both, I’ll forward this to my user.
Pierre

> On 4 Aug 2020, at 7:30 PM, Dener, Alp  wrote:
> 
> Hi Pierre,
> 
> This is indeed an issue with TAO default tolerances.
> 
> Specifically it has to do with constraint tolerances. The default value is an 
> exact zero.
> 
> The problem should still work with line 89 commented out. Our default 
> gradient tolerance is 1e-8. In your case, commenting out line 90 is causing 
> the solver to try to converge constraints to exact zero and it cannot.
> 
> I will get a patch out for this within the week but in the meantime please 
> ensure that the constraint tolerances are set for any constrained problem.
> 
> Thank you!
> —
> Alp
> 
>> On Aug 4, 2020, at 12:24 PM, Munson, Todd via petsc-dev 
>>  wrote:
>> 
>> 
>> Hi Pierre,
>> 
>> I would say the answer to that question is "no, its not expected".  
>> 
>> We will look into fixing it.  It seems like the default tolerances are being 
>> set to zero and the result is an inability to satisfy the constraints or 
>> gradient of the Lagrangian to that small of a tolerance.
>> 
>> Thanks for point this out to us!  We will get it resolved.
>> 
>> Thanks, Todd.
>> 
>> On 8/4/20, 12:12 PM, "petsc-dev on behalf of Pierre Jolivet" 
>>  
>> wrote:
>> 
>>   Hello,
>>   If I comment line 89 and 90 of src/tao/constrained/tutorials/ex1.c from 
>> master, the example deadlocks with a single process.
>>   Is this expected?
>> 
>>   Thanks,
>>   Pierre
>> 
>>   $  ./ex1 -tao_monitor
>>    Constrained Problem -
>>   Solution should be f(1,1)=-2
>> 0 TAO,  Function value: 8.,  Residual: 8.48528
>> 1 TAO,  Function value: 1.28532,  Residual: 10.7411
>> 2 TAO,  Function value: -2.56703,  Residual: 1.87847
>> 3 TAO,  Function value: -2.03161,  Residual: 0.12881
>> 4 TAO,  Function value: -1.99961,  Residual: 0.0450855
>> 5 TAO,  Function value: -1.3,  Residual: 0.000916939
>> 6 TAO,  Function value: -1.9,  Residual: 6.69184e-05
>> 7 TAO,  Function value: -2.,  Residual: 7.15427e-06
>> 8 TAO,  Function value: -2.,  Residual: 7.15779e-07
>> 9 TAO,  Function value: -2.,  Residual: 7.15777e-08
>>10 TAO,  Function value: -2.,  Residual: 7.15777e-09
>>11 TAO,  Function value: -2.,  Residual: 7.15779e-10
>>12 TAO,  Function value: -2.,  Residual: 7.15775e-11
>>13 TAO,  Function value: -2.,  Residual: 7.1599e-12
>>14 TAO,  Function value: -2.,  Residual: 7.1599e-13
>>15 TAO,  Function value: -2.,  Residual: 7.22085e-14
>>16 TAO,  Function value: -2.,  Residual: 6.44626e-15
>>17 TAO,  Function value: -2.,  Residual: 1.00751e-15
>>18 TAO,  Function value: -2.,  Residual: 1.70295e-17
>>19 TAO,  Function value: -2.,  Residual: 1.70295e-18
>>20 TAO,  Function value: -2.,  Residual: 1.70295e-19
>>21 TAO,  Function value: -2.,  Residual: 1.70295e-20
>>22 TAO,  Function value: -2.,  Residual: 1.70295e-21
>>23 TAO,  Function value: -2.,  Residual: 1.70295e-22
>>24 TAO,  Function value: -2.,  Residual: 1.70295e-23
>>25 TAO,  Function value: -2.,  Residual: 1.70295e-24
>>26 TAO,  Function value: -2.,  Residual: 1.70295e-25
>>27 TAO,  Function value: -2.,  Residual: 1.70295e-26
>>28 TAO,  Function value: -2.,  Residual: 1.70295e-27
>>29 TAO,  Function value: -2.,  Residual: 1.70295e-28
>>30 TAO,  Function value: -2.,  Residual: 1.70295e-29
>>31 TAO,  Function value: -2.,  Residual: 1.70295e-30
>>32 TAO,  Function value: -2.,  Residual: 1.70295e-31
>>33 TAO,  Function value: -2.,  Residual: 1.70295e-32
>>34 TAO,  Function value: -2.,  Residual: 1.70295e-33
>>35 TAO,  Function value: -2.,  Residual: 1.70295e-34
>>36 TAO,  Function value: -2.,  Residual: 1.70295e-35
>>37 TAO,  Function value: -2.,  Residual: 1.70295e-36
>>38 TAO,  Function value: -2.,  Residual: 1.70295e-37
>>39 TAO,  Function value: -2.,  Residual: 1.70295e-38
>>40 TAO,  Function value: -2.,  Residual: 1.70295e-39
>>41 TAO,  Function value: -2.,  Residual: 1.70295e-40
>>42 TAO,  Function value: -2.,  Residual: 1.70295e-41
>>43 TAO,  Function value: -2.,  Residual: 1.70295e-42
>>44 TAO,  Function value: -2.,  Residual: 1.70295e-43
>>45 TAO,  Function value: -2.,  Residual: 1.70295e-44
>>46 TAO,  Function value: -2.,  Residual: 1.70295e-45
>>47 TAO,  Function value: -2.,  Residual: 1.70295e-46
>>48 TAO,  Function value: -2.,  Residual: 1.70295e-47
>>49 TAO,  Function value: -2.,  Residual: 1.70295e-48
>>50 TAO,  Function value: -2.,  Residual: 1.70295e-49
>>51 TAO,  Function value: -2.,  Residual: 1.70295e-50
>>52 TAO,  Function value: -2.,  Residual: 1.70295e-51
>>   ^C
>> 
> 

[petsc-dev] TAOPDIPM

[Pierre Jolivet]

Hello,
If I comment line 89 and 90 of src/tao/constrained/tutorials/ex1.c from master, 
the example deadlocks with a single process.
Is this expected?

Thanks,
Pierre

$  ./ex1 -tao_monitor
 Constrained Problem -
Solution should be f(1,1)=-2
  0 TAO,  Function value: 8.,  Residual: 8.48528
  1 TAO,  Function value: 1.28532,  Residual: 10.7411
  2 TAO,  Function value: -2.56703,  Residual: 1.87847
  3 TAO,  Function value: -2.03161,  Residual: 0.12881
  4 TAO,  Function value: -1.99961,  Residual: 0.0450855
  5 TAO,  Function value: -1.3,  Residual: 0.000916939
  6 TAO,  Function value: -1.9,  Residual: 6.69184e-05
  7 TAO,  Function value: -2.,  Residual: 7.15427e-06
  8 TAO,  Function value: -2.,  Residual: 7.15779e-07
  9 TAO,  Function value: -2.,  Residual: 7.15777e-08
 10 TAO,  Function value: -2.,  Residual: 7.15777e-09
 11 TAO,  Function value: -2.,  Residual: 7.15779e-10
 12 TAO,  Function value: -2.,  Residual: 7.15775e-11
 13 TAO,  Function value: -2.,  Residual: 7.1599e-12
 14 TAO,  Function value: -2.,  Residual: 7.1599e-13
 15 TAO,  Function value: -2.,  Residual: 7.22085e-14
 16 TAO,  Function value: -2.,  Residual: 6.44626e-15
 17 TAO,  Function value: -2.,  Residual: 1.00751e-15
 18 TAO,  Function value: -2.,  Residual: 1.70295e-17
 19 TAO,  Function value: -2.,  Residual: 1.70295e-18
 20 TAO,  Function value: -2.,  Residual: 1.70295e-19
 21 TAO,  Function value: -2.,  Residual: 1.70295e-20
 22 TAO,  Function value: -2.,  Residual: 1.70295e-21
 23 TAO,  Function value: -2.,  Residual: 1.70295e-22
 24 TAO,  Function value: -2.,  Residual: 1.70295e-23
 25 TAO,  Function value: -2.,  Residual: 1.70295e-24
 26 TAO,  Function value: -2.,  Residual: 1.70295e-25
 27 TAO,  Function value: -2.,  Residual: 1.70295e-26
 28 TAO,  Function value: -2.,  Residual: 1.70295e-27
 29 TAO,  Function value: -2.,  Residual: 1.70295e-28
 30 TAO,  Function value: -2.,  Residual: 1.70295e-29
 31 TAO,  Function value: -2.,  Residual: 1.70295e-30
 32 TAO,  Function value: -2.,  Residual: 1.70295e-31
 33 TAO,  Function value: -2.,  Residual: 1.70295e-32
 34 TAO,  Function value: -2.,  Residual: 1.70295e-33
 35 TAO,  Function value: -2.,  Residual: 1.70295e-34
 36 TAO,  Function value: -2.,  Residual: 1.70295e-35
 37 TAO,  Function value: -2.,  Residual: 1.70295e-36
 38 TAO,  Function value: -2.,  Residual: 1.70295e-37
 39 TAO,  Function value: -2.,  Residual: 1.70295e-38
 40 TAO,  Function value: -2.,  Residual: 1.70295e-39
 41 TAO,  Function value: -2.,  Residual: 1.70295e-40
 42 TAO,  Function value: -2.,  Residual: 1.70295e-41
 43 TAO,  Function value: -2.,  Residual: 1.70295e-42
 44 TAO,  Function value: -2.,  Residual: 1.70295e-43
 45 TAO,  Function value: -2.,  Residual: 1.70295e-44
 46 TAO,  Function value: -2.,  Residual: 1.70295e-45
 47 TAO,  Function value: -2.,  Residual: 1.70295e-46
 48 TAO,  Function value: -2.,  Residual: 1.70295e-47
 49 TAO,  Function value: -2.,  Residual: 1.70295e-48
 50 TAO,  Function value: -2.,  Residual: 1.70295e-49
 51 TAO,  Function value: -2.,  Residual: 1.70295e-50
 52 TAO,  Function value: -2.,  Residual: 1.70295e-51
^C

[petsc-dev] git status to test list

[Pierre Jolivet]

Hello,
To git/regexp enthusiasts and/or test harness people: is there some 
script/command lying around to convert a list of modified examples (according 
to git) into a list understandable by the test system?
Thanks for your help,
Pierre

PS: here is such a list
$ git status -uno
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)

modified:   config/BuildSystem/config/types.py
modified:   include/petscsys.h
modified:   src/dm/label/tutorials/ex1f90.F90
modified:   src/ksp/ksp/tests/ex16f.F90
modified:   src/ksp/ksp/tutorials/ex5f.F90
modified:   src/ksp/ksp/tutorials/ex75f.F90
modified:   src/ksp/ksp/tutorials/ex76f.F90
modified:   src/ksp/ksp/tutorials/ex77f.F90
modified:   src/ksp/ksp/tutorials/ex7f.F90
modified:   src/mat/tests/ex196f90.F90
modified:   src/sys/tests/ex13f.F90
modified:   src/sys/tests/ex47f.F90
modified:   src/sys/tutorials/ex17f.F90
modified:   src/sys/tutorials/ex2f.F90
modified:   src/vec/vec/tutorials/ex11f90.F90
modified:   src/vec/vec/tutorials/ex18f.F90
modified:   src/vec/vec/tutorials/ex1f.F90

no changes added to commit (use "git add" and/or "git commit -a")

[petsc-dev] -on_error_attach_debugger

[Pierre Jolivet]

Hello,
(Probably nobody cares, but) it looks like -on_error_attach_debugger is neither 
“-malloc_dump"-clean nor valgrind-clean.

$ cd src/ksp/ksp/tutorials
$ ~/petsc/arch-darwin-c-debug-real/bin/mpiexec -n 2 valgrind --leak-check=full 
--show-leak-kinds=all ./ex2 -on_error_attach_debugger
Norm of error 0.000411674 iterations 7
==2074== 24 bytes in 1 blocks are still reachable in loss record 1 of 1
==2074==at 0x483E340: memalign (in 
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2074==by 0x4A80902: PetscMallocAlign (mal.c:49)
==2074==by 0x4A822E8: PetscMallocA (mal.c:422)
==2074==by 0x4AD2B45: PetscPushErrorHandler (err.c:161)
==2074==by 0x4B1D47B: PetscOptionsCheckInitial_Private (init.c:545)
==2074==by 0x4B25AE7: PetscInitialize (pinit.c:1019)
==2074==by 0x10B4FA: main (ex2.c:31)
$ ~/petsc/arch-darwin-c-debug-real/bin/mpiexec -n 2 ./ex2 
-on_error_attach_debugger -malloc_dump
Norm of error 0.000411674 iterations 7
[ 0]32 bytes PetscPushErrorHandler() line 161 in 
/home/john/petsc/src/sys/error/err.c
[ 1]32 bytes PetscPushErrorHandler() line 161 in 
/home/john/petsc/src/sys/error/err.c

Thanks,
Pierre

Re: [petsc-dev] [petsc-users] PETSc and Windows 10

[Pierre Jolivet]

(moving to petsc-dev, feel free to continue the “general” discussion on 
petsc-users)

> On 29 Jun 2020, at 10:03 PM, Pierre Jolivet  
> wrote:
> 
> Sorry for the noise, but maybe it’s better to put this in PETSc ./configure, 
> like you did here Satish 
> https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523
>  
> <https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523>
>  ?
> If Gfortran100 && MS-MPI, then FFLAGS += "-fallow-invalid-boz"
> WDY(PETSc-)GT?

So, it looks like -fallow-invalid-boz is not as old as I thought…
g++.exe (Rev1, Built by MSYS2 project) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Running Executable WITHOUT threads to time it out
Executing: gfortran -c -o /tmp/petsc-kmp34jh9/config.setCompilers/conftest.o 
-I/tmp/petsc-kmp34jh9/config.setCompilers -fallow-invalid-boz  
/tmp/petsc-kmp34jh9/config.setCompilers/conftest.F90 
Possible ERROR while running compiler: exit code 1
stderr:
gfortran.exe: error: unrecognized command line option '-fallow-invalid-boz'
Source:
  program main

  end
  Error testing Fortran compiler: Cannot compile FC with gfortran.
Deleting "FC"
Again, would it be OK if I add a check in MPI.py to force this option when 
isGfortran100plus and defined(HAVE_MSMPI)? Or do we expect Windows users to 
know about this ninja gfortran flag?
Is this tested on some Cygwin box (MS-MPI 10.1.2 + gfortran 10)?
I see arch-ci-mswin-intel.py is using MS-MPI (which version?) but with ifort.

Thanks,
Pierre

[petsc-dev] Solving systems with (K,M)AIJ

[Pierre Jolivet]

Hello,
When solving systems with a KSP that has a (K,M)AIJ Pmat attached to it, it may 
be nice to define a preconditioner on the inner AIJ Pmat, so that we are not 
stuck with PCPBJACOBI.
For example, with FS, there is PCFieldSplitGetSubKSP and then you do what you 
want with the inner blocks.
Is there something similar for (K,M)AIJ, maybe with a fancy PCCOMPOSITE?
Of course, I could simply write a shell, but a new PC/option could be added to 
deal with such Pmat types?

Thanks,
Pierre

Re: [petsc-dev] Googling PetscMatlabEngine leads to 3.7 version

[Pierre Jolivet]

Speaking of the website, wrong links are being generated in the documentation.
https://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MATSEQDENSECUDA.html
 
<https://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MATSEQDENSECUDA.html>
 has a link to 
https://www.mcs.anl.gov/petsc/petsc-dev/src/mat/impls/dense/seq/cudadensecuda.cu#MATSEQDENSECUDA
 
<https://www.mcs.anl.gov/petsc/petsc-dev/src/mat/impls/dense/seq/cudadensecuda.cu#MATSEQDENSECUDA>
 which should read 
https://www.mcs.anl.gov/petsc/petsc-dev/src/mat/impls/dense/seq/cuda/densecuda.cu.html#MATSEQDENSECUDA
 
<https://www.mcs.anl.gov/petsc/petsc-dev/src/mat/impls/dense/seq/cuda/densecuda.cu.html#MATSEQDENSECUDA>
 (notice the added / and .html)

Thanks,
Pierre

> On 9 Jun 2020, at 7:52 PM, Satish Balay  wrote:
> 
> Wrt different petsc versions - Its best to disable google from searching old 
> versions.
> 
> I've updated robots.txt [it previously had some old values]
> 
>>>> 
> petsc@es:/mcs/web/research/projects/petsc$ cat robots.txt
> User-agent: *
> Disallow: bin
> Disallow: old
> Disallow: mirror
> Disallow: NCSA-UIUC
> Disallow: petsc-3.*
> Disallow: petsc4py-3.*
> Disallow: petsc-dev
> Disallow: petsc-master
> Disallow: style.css
>>>>> 
> 
> [Assuming the above is not buggy]  it should primarily index petsc-current 
> and petsc4py-current
> 
> Wrt PetscMatlabEngine - duckduckgo finds
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_MATLAB_ENGINE_.html
> but not:
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscMatlabEngine.html
> 
> So something is weird wrt search [within the same version]
> 
> Satish
> 
> 
> On Tue, 9 Jun 2020, Barry Smith wrote:
> 
>> 
>> 
>>   I think I know the problem; there are no links to PCMG from other PCMG*  
>> manual pages hence google does not find it or thinks it is unpopular so an 
>> older page with a small number of links has higher priority? 
>> PetscMatlabEngine has links so I am not sure if it is the same problem. 
>> 
>>   Barry
>> 
>> 
>> 
>> 
>>> On Jun 8, 2020, at 11:41 PM, Pierre Jolivet  
>>> wrote:
>>> 
>>> Googling PCMG is no fun either (and there is no actual reference to the 
>>> current man page).
>>> 
>>> Thanks,
>>> Pierre
>>> 
>>> 
>>> 
>>> 
>>>> On 9 Jun 2020, at 2:27 AM, Barry Smith >>> <mailto:bsm...@petsc.dev>> wrote:
>>>> 
>>>> 
>>>>  Googling PetscMatlabEngine leads to 3.7 version but 
>>>> PetscMatlabEngineDestroy leads to 3.13. 
>>>> 
>>>>  Any idea why some of the manual pages don't point to the latest version?
>>>> 
>>>>  Barry
>>>> 
>>> 
>> 
>> 
> 

Re: [petsc-dev] Googling PetscMatlabEngine leads to 3.7 version

[Pierre Jolivet]

Googling PCMG is no fun either (and there is no actual reference to the current man page).Thanks,Pierre

PCMG site_https___www.mcs.anl.gov_ - Google Search.pdf
Description: Adobe PDF document
On 9 Jun 2020, at 2:27 AM, Barry Smith  wrote:   Googling PetscMatlabEngine leads to 3.7 version but PetscMatlabEngineDestroy leads to 3.13.    Any idea why some of the manual pages don't point to the latest version?   Barry

Re: [petsc-dev] Are we suppressing stderr when comparing outputs of tests?

[Pierre Jolivet]

Stefano,
If we go back to your original question, then you are correct under the 
additional assumption that main returns 0.
Here is a MWE.

$ git diff --unified=1 src/ksp/ksp/tutorials/ex1.c
diff --git a/src/ksp/ksp/tutorials/ex1.c b/src/ksp/ksp/tutorials/ex1.c
index 3b3e776a6d..a5ed63ca18 100644
--- a/src/ksp/ksp/tutorials/ex1.c
+++ b/src/ksp/ksp/tutorials/ex1.c
@@ -33,2 +33,3 @@ int main(int argc,char **args)
   ierr = PetscInitialize(,,(char*)0,help);if (ierr) return ierr;
+  ierr = MatView(A,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(0); // not ierr, 0
   ierr = MPI_Comm_size(PETSC_COMM_WORLD,);CHKERRQ(ierr);
$ make -f gmakefile.test test search='ksp_ksp_tutorials-ex1_1'
Using MAKEFLAGS: search=ksp_ksp_tutorials-ex1_1
TEST 
arch-darwin-c-debug-real/tests/counts/ksp_ksp_tutorials-ex1_1.counts
 ok ksp_ksp_tutorials-ex1_1
 ok diff-ksp_ksp_tutorials-ex1_1
$ echo $?
0
$ mpirun -n 1 arch-darwin-c-debug-real/tests/ksp/ksp/tutorials/ex1 
-ksp_monitor_short -ksp_gmres_cgs_refinement_type refine_always
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Wrong type of object: Parameter # 1
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for 
trouble shooting.
[…]

I’m not sure I’m OK with this (shouldn’t both stderr and stdout be compared, 
and if there is no stderr to compare against, check that it’s null?), but I 
guess this is how testing works.
About KSPHPDDM, I’m currently implementing the necessary steps as Matt 
suggested to go from PetscInt (MatMatMult) -> int (my stuff) -> PetscInt 
(KSPSolve_HPDDM) to have proper error checking so that in fact main does not 
return 0. Anyway, as you know since our last iteration the error w.r.t. 
MatDensePlaceArray is fixed now in your branch.

Thanks,
Pierre

> On 13 May 2020, at 6:15 PM, Matthew Knepley  wrote:
> 
> On Wed, May 13, 2020 at 11:33 AM Pierre Jolivet  <mailto:pierre.joli...@enseeiht.fr>> wrote:
> I’m only CHKERRQ’ing inside the routine but I’m not propagating this down to 
> the return code of KSPSolve.
> I will try to fix this, or maybe switch to CHKERRABORT instead, because it 
> currently goes like KSPSolve (PETSc) -> my stuff (non PETSc) -> MatMatMult or 
> PCApply (PETSc), and I’m not sure I want to put some more PETSc code inside 
> “my stuff” as it can be a pain to maintain for people that uses “my stuff” 
> without PETSc. Sadly, they do exist.
> Also, I’m using PetscFunction[Begin|End]User, don’t know if switching to 
> PetscFunction[Begin|End] would change something here, though I’m guessing it 
> wouldn't.
> 
> No, those work the same.
> 
> I think you can propagate just with an integer return code, nothing PETSc 
> dependent.
> 
>   Thanks,
> 
>  Matt
>  
> I’ll keep you posted, thanks for reporting this,
> Pierre
> 
>> On 13 May 2020, at 3:03 PM, Stefano Zampini > <mailto:stefano.zamp...@gmail.com>> wrote:
>> 
>> The error is raised from HPDDM , that does not stop and I think does not 
>> return error codes. Pierre?
>> 
>> Il giorno mer 13 mag 2020 alle ore 16:01 Matthew Knepley > <mailto:knep...@gmail.com>> ha scritto:
>> I see my stderr with tests. I am running it right now.
>> 
>>Matt
>> 
>> On Wed, May 13, 2020 at 8:55 AM Stefano Zampini > <mailto:stefano.zamp...@gmail.com>> wrote:
>> I have a test that fails, but it only prints a small amount of information
>> 
>> [szampini@localhost petsc]$ make -f gmakefile.test test search='ksp%' 
>> searchin='ex75_2_icc' PETSC_ARCH=arch-opt
>> Using MAKEFLAGS: -- PETSC_ARCH=arch-opt searchin=ex75_2_icc search=ksp%
>> TEST arch-opt/tests/counts/ksp_ksp_tutorials-ex75_2_icc.counts
>>  ok ksp_ksp_tutorials-ex75_2_icc+reset-false
>> not ok diff-ksp_ksp_tutorials-ex75_2_icc+reset-false # Error code: 1
>> # 3c3
>> # < Linear solve converged due to CONVERGED_RTOL iterations 34
>> # ---
>> # > Linear solve did not converge due to DIVERGED_ITS iterations 1000
>>  ok ksp_ksp_tutorials-ex75_2_icc+reset-true
>>  ok diff-ksp_ksp_tutorials-ex75_2_icc+reset-true
>> 
>> If I run the test that fails myself, I get PETSC_ERROR messages
>> 
>> [szampini@localhost tutorials]$ ./ex75 -nmat 3 -pc_type icc 
>> -ksp_converged_reason -ksp_type hpddm -ksp_max_it 1000 -ksp_gmres_restart 40 
>> -ksp_rtol 1e-10 -ksp_hpddm_type gcrodr -ksp_hpddm_recycle 20 -reset false 
>> -load_dir ${DATAFILESPATH}/matrices/hpddm/GCRODR
>> Linear solve converged due to CONVERGED_RTOL iterations 94
>> Linear solve converged due to CONVERGED_RTOL iterations 37
>> [0]PETSC ERROR: - Error Message 
>> 

Re: [petsc-dev] Are we suppressing stderr when comparing outputs of tests?

[Pierre Jolivet]

I’m only CHKERRQ’ing inside the routine but I’m not propagating this down to 
the return code of KSPSolve.
I will try to fix this, or maybe switch to CHKERRABORT instead, because it 
currently goes like KSPSolve (PETSc) -> my stuff (non PETSc) -> MatMatMult or 
PCApply (PETSc), and I’m not sure I want to put some more PETSc code inside “my 
stuff” as it can be a pain to maintain for people that uses “my stuff” without 
PETSc. Sadly, they do exist.
Also, I’m using PetscFunction[Begin|End]User, don’t know if switching to 
PetscFunction[Begin|End] would change something here, though I’m guessing it 
wouldn't.

I’ll keep you posted, thanks for reporting this,
Pierre

> On 13 May 2020, at 3:03 PM, Stefano Zampini  wrote:
> 
> The error is raised from HPDDM , that does not stop and I think does not 
> return error codes. Pierre?
> 
> Il giorno mer 13 mag 2020 alle ore 16:01 Matthew Knepley  > ha scritto:
> I see my stderr with tests. I am running it right now.
> 
>Matt
> 
> On Wed, May 13, 2020 at 8:55 AM Stefano Zampini  > wrote:
> I have a test that fails, but it only prints a small amount of information
> 
> [szampini@localhost petsc]$ make -f gmakefile.test test search='ksp%' 
> searchin='ex75_2_icc' PETSC_ARCH=arch-opt
> Using MAKEFLAGS: -- PETSC_ARCH=arch-opt searchin=ex75_2_icc search=ksp%
> TEST arch-opt/tests/counts/ksp_ksp_tutorials-ex75_2_icc.counts
>  ok ksp_ksp_tutorials-ex75_2_icc+reset-false
> not ok diff-ksp_ksp_tutorials-ex75_2_icc+reset-false # Error code: 1
> # 3c3
> # < Linear solve converged due to CONVERGED_RTOL iterations 34
> # ---
> # > Linear solve did not converge due to DIVERGED_ITS iterations 1000
>  ok ksp_ksp_tutorials-ex75_2_icc+reset-true
>  ok diff-ksp_ksp_tutorials-ex75_2_icc+reset-true
> 
> If I run the test that fails myself, I get PETSC_ERROR messages
> 
> [szampini@localhost tutorials]$ ./ex75 -nmat 3 -pc_type icc 
> -ksp_converged_reason -ksp_type hpddm -ksp_max_it 1000 -ksp_gmres_restart 40 
> -ksp_rtol 1e-10 -ksp_hpddm_type gcrodr -ksp_hpddm_recycle 20 -reset false 
> -load_dir ${DATAFILESPATH}/matrices/hpddm/GCRODR
> Linear solve converged due to CONVERGED_RTOL iterations 94
> Linear solve converged due to CONVERGED_RTOL iterations 37
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Operation done in wrong order
> [0]PETSC ERROR: Need to call MatDenseRestoreArray first
> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html 
>  for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.13.1-192-g23916988c6  GIT 
> Date: 2020-05-09 02:16:44 +0300
> [0]PETSC ERROR: ./ex75 on a arch-debug named localhost.localdomain by 
> szampini Wed May 13 15:51:48 2020
> [0]PETSC ERROR: Configure options 
> --DATAFILESPATH=/home/szampini/Devel/petscdatafiles --download-elemental 
> --download-chaco --download-ctetgen --download-exodusii --download-glvis 
> --download-hara --download-hara-commit=HEAD --download-hdf5 --download-hpddm 
> --download-hpddm-commit=HEAD --download-hypre --download-metis 
> --download-mfem-commit=origin/master --download-mfem=1 --download-ml 
> --download-mumps --download-netcdf --download-p4est --download-parmetis 
> --download-pnetcdf --download-ptscotch --download-revolve 
> --download-scalapack --download-slepc --download-suitesparse 
> --download-superlu --download-superlu_dist --download-triangle 
> --with-blaslapack-include=/usr/include/openblas --with-cxx-dialect=C++11 
> --with-fortran-bindings=0 --with-opengl --with-openmp 
> --with-thrust-dir=/home/szampini/local --with-valgrind --with-zlib 
> CFLAGS="-Wall -g -O0" CXXFLAGS="-Wall -g -O0" FFLAGS="-g -O0" 
> PETSC_ARCH=arch-debug
> [0]PETSC ERROR: #1 MatDensePlaceArray_SeqDense() line 1364 in 
> /home/szampini/Devel/petsc/src/mat/impls/dense/seq/dense.c
> [0]PETSC ERROR: #2 MatDensePlaceArray() line 1900 in 
> /home/szampini/Devel/petsc/src/mat/impls/dense/mpi/mpidense.c
> [0]PETSC ERROR: #3 GMV() line 128 in 
> /home/szampini/Devel/petsc/arch-debug/include/HPDDM_PETSc.hpp
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Operation done in wrong order
> [0]PETSC ERROR: Need to call MatDenseRestoreArray first
> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html 
>  for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.13.1-192-g23916988c6  GIT 
> Date: 2020-05-09 02:16:44 +0300
> [0]PETSC ERROR: ./ex75 on a arch-debug named localhost.localdomain by 
> szampini Wed May 13 15:51:48 2020
> [0]PETSC ERROR: Configure options 
> --DATAFILESPATH=/home/szampini/Devel/petscdatafiles --download-elemental 
> --download-chaco --download-ctetgen --download-exodusii --download-glvis 
> 

Re: [petsc-dev] MATOP_MAT_MULT

[Pierre Jolivet]

MatShellSetMatProductOperation looks really nice to me, thanks!
Pierre

> On 12 May 2020, at 12:13 PM, Stefano Zampini  
> wrote:
> 
> Pierre and Jose
> 
> I have added support for MatMat callbacks operations for MATSHELL, you may 
> want to take a look here for how to use it 
> https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=7f809910e2bafe055242a87d70afd114664ffaf8
>  
> 
> This is the relevant commit 
> https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=e01f573d2ca0ec07c54db508a94a042fba4038de
>  
> 
> 
> Let me know if you need more customization (e.g. attach data to the product 
> in a more systematic way) or if it can already fit your frameworks.
> 
> Best
> Stefano
> 
> Il giorno dom 10 mag 2020 alle ore 21:04 Stefano Zampini 
> mailto:stefano.zamp...@gmail.com>> ha scritto:
> 
> 
>> On May 10, 2020, at 8:56 PM, Jose E. Roman > > wrote:
>> 
>> 
>> 
>>> El 10 may 2020, a las 19:42, Stefano Zampini >> > escribió:
>>> 
>>> Glad to hear it works. Anyway, without the MatShellVecSetType call the code 
>>> was erroring for me, not leaking memory.
>>> Where you also providing -vec_type cuda at command line or what? 
>>> Mark recently noted a similar leak, and I was wondering what was the cause 
>>> for yours. A MWE would be great.
>> 
>> I never use -vec_type cuda because all SLEPc tests that use CUDA create 
>> vectors with MatCreateVecs(). Until now I had never tested examples with 
>> shell matrices.
>> 
>> This kind of issues would be more easy to detect if macros such as 
>> PetscCheckSameTypeAndComm() would actually compare type_name.
> 
> VECCUDA specific code checks for type names (either VECSEQCUDA or VECMPICUDA) 
> and these errored in my case. (Try by simply removing MatShellSetVecType)
> 
> zampins@jasmine:~/petsc/src/mat/tests$ git diff
> diff --git a/src/mat/tests/ex69.c b/src/mat/tests/ex69.c
> index b04652d..2a9374d 100644
> --- a/src/mat/tests/ex69.c
> +++ b/src/mat/tests/ex69.c
> @@ -89,7 +89,7 @@ int main(int argc,char **argv)
>if (use_shell) {
>  ierr = 
> MatCreateShell(PetscObjectComm((PetscObject)v),nloc,nloc,n,n,A,);CHKERRQ(ierr);
>  ierr = 
> MatShellSetOperation(S,MATOP_MULT,(void(*)(void))MatMult_S);CHKERRQ(ierr);
> -ierr = MatShellSetVecType(S,vtype);CHKERRQ(ierr);
> +//ierr = MatShellSetVecType(S,vtype);CHKERRQ(ierr);
>  /* we could have called the general convertor also */
>  /* ierr = MatConvert(A,MATSHELL,MAT_INITIAL_MATRIX,);CHKERRQ(ierr); */
>} else {
> zampins@jasmine:~/petsc/src/mat/tests$ ./ex69 -use_shell
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda
> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html 
>  for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.13-213-g441560f  GIT Date: 
> 2020-04-25 17:01:13 +0300
> [0]PETSC ERROR: ./ex69 on a arch-gpu-double-openmp-openblas named jasmine by 
> zampins Sun May 10 21:03:09 2020
> [0]PETSC ERROR: Configure options --download-cub=1 
> --download-hara-commit=HEAD --download-hara=1 --download-kblas=1 
> --download-magma=1 --download-openblas-use-pthreads=1 --download-openblas=1 
> --download-superlu_dist-commit=HEAD --download-superlu_dist=1 
> --download-mumps=1 --download-scalapack=1 
> --with-cc=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpicc
>  --with-cuda-gencodearch=37 --with-cuda=1 
> --with-cudac=/opt/ecrc/cuda/10.1/bin/nvcc --with-cxx-dialect=C++11 
> --with-cxx=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpicxx
>  
> --with-fc=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpif90
>  --with-fortran-bindings=0 --with-magma-fortran-bindings=0 
> --with-opencl-include=/opt/ecrc/cuda/10.1/include 
> --with-opencl-lib="-L/opt/ecrc/cuda/10.1/lib64 -lOpenCL" --with-openmp=1 
> --with-precision=double PETSC_ARCH=arch-gpu-double-openmp-openblas
> [0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 2206 in 
> /home/zampins/petsc/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1448 in 
> /home/zampins/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu 
> 
> [0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1375 in 
> /home/zampins/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu 
> 
> [0]PETSC ERROR: #4 MatMult() line 2392 in 
> /home/zampins/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #5 MatMult_S() line 12 in ex69.c
> [0]PETSC ERROR: #6 MatMult_Shell() line 

Re: [petsc-dev] MATOP_MAT_MULT

[Pierre Jolivet]

Hong,
I also removed a couple of deprecated MATOP there 
https://gitlab.com/petsc/petsc/-/merge_requests/2786 

If you git grep MATOP_TRANSPOSE_MAT_MULT, you’ll see that there are still some 
remaining in src/tao/constrained/impls/admm/admm.c
Actually, in this case, it does not come from a MatHasOperation but from a 
MatShellSetOperation.
I don’t think there is any example showing how to add an algorithm inside 
user-code to compute C=A*B. Maybe it would be worth creating such an example? 
(I’m actually not sure how to do it myself)
That way, it would be quite convenient to try out new algorithms, instead of 
having to recompile the library itself.

Thanks,
Pierre

> On 9 May 2020, at 1:36 PM, Stefano Zampini  wrote:
> 
> Hong
> 
> you may want to take a look at this big commit here 
> https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=94a81e695f30a0ff4a14c0eb596e09393f9b329f
>  
> 
> 
> Now that you coded MatProduct, I think it is the case of not playing with the 
> destroy/view routines anymore in the various MatMat operations (you know, I 
> have raised this issue already in the past). I think I got it fixed properly. 
> What I did in short, I have added extra slots to the MatProduct struct to 
> keep track of how to destroy/view the extra information you need for the 
> products.
> Also, I have deprecated MatFreeIntermediateStructures, since we now have 
> MatProductClear
> 
> I have spent quite a bit of time on it, and I personally think it is the 
> proper way to go. I think I took care of all the cases already covered. Tests 
> are clean for me
> I would like to hear your opinion.
> 
> 
> Thanks
> Stefano
> 
> Il giorno sab 9 mag 2020 alle ore 13:45 Stefano Zampini 
> mailto:stefano.zamp...@gmail.com>> ha scritto:
> Jose
> 
> I have just pushed a test 
> https://gitlab.com/petsc/petsc/-/blob/d64c2bc63c8d5d1a8c689f1abc762ae2722bba26/src/mat/tests/ex69.c
>  
> 
> See if it fits your framework, and feel free to modify the test to add more 
> checks
> 
> 
> Il giorno ven 8 mag 2020 alle ore 18:48 Jose E. Roman  > ha scritto:
> Attached. Run with -test 1 or -test 2
> 
> > El 8 may 2020, a las 17:14, Stefano Zampini  > > escribió:
> > 
> > Jose
> > 
> > Just send me a MWE and I’ll fix the case for you
> > 
> > Thanks
> > Stefano
> 
> 
> -- 
> Stefano
> 
> 
> -- 
> Stefano

Re: [petsc-dev] MATOP_MAT_MULT

[Pierre Jolivet]

Stefano,
Is this working for nsize > 1 
https://gitlab.com/petsc/petsc/-/blob/7e88e4dd44e2a5120b858cf9f19502ac359985be/src/mat/tests/ex70.c#L295
 
<https://gitlab.com/petsc/petsc/-/blob/7e88e4dd44e2a5120b858cf9f19502ac359985be/src/mat/tests/ex70.c#L295>
I am now getting (in another example):
[0]PETSC ERROR: Call MatProductSymbolic() first
Instead of the previous:
[0]PETSC ERROR: MatProductSetFromOptions_AB for A mpisbaij and B mpidense is 
not supported

(But my branch is lagging behind maint, so maybe I’m missing some other fixes, 
take this with a grain of salt).
Thanks,
Pierre

> On 6 May 2020, at 4:52 PM, Stefano Zampini  wrote:
> 
> I have working support for MATSHELL here 
> https://gitlab.com/petsc/petsc/-/commit/146e7f1ccf5f267b36079cac494077a23e8bbc45
>  
> <https://gitlab.com/petsc/petsc/-/commit/146e7f1ccf5f267b36079cac494077a23e8bbc45>
> Tested here 
> https://gitlab.com/petsc/petsc/-/commit/c4fcaa45a01cc783c629913983b204a1cbcb3939
>  
> <https://gitlab.com/petsc/petsc/-/commit/c4fcaa45a01cc783c629913983b204a1cbcb3939>
> 
> Jose and Pierre, this code is supposed to work with CUDA, but I haven't 
> tested it yet
> Can you tell me if this fixes the issues for you to not have to loop over the 
> columns of the dense matrix yourself?
> 
> Il giorno mer 6 mag 2020 alle ore 10:09 Stefano Zampini 
> mailto:stefano.zamp...@gmail.com>> ha scritto:
> Hong
> 
> If the product is not supported, the type of C will never be set anyway, so 
> you cannot call MatHasOperation after MatProductSetFromOptions.
> The purpose of MatProductSetFromOptions is to populate the function pointers 
> for symbolic and numeric phases. If not found, they should be set to null 
> instead of erroring as it is now.
> What I propose is to have MatProductHasOperation (not MatHasOperation): this 
> function will be identical to MatHasOperation, with the only difference that 
> does not call PetscValidType on the input mat.
> 
> Meanwhile, I’m coding a basic MatMat (and MatTransposeMat) driver to loop 
> over dense columns and apply MatMult. (Or MatMultTranspose) without memory 
> movement.
> This will be valid for all B matrices being of type dense (and its 
> derivations), with C of type dense too. This in principle will fix Jose and 
> Pierre’s issues (they can correct me if I’m wrong)
> 
> However, we should definitely have a way for the user to enquire if a given 
> operation is supported or not. 
> 
> Thanks
> Stefano
> 
>> On May 6, 2020, at 12:03 AM, Zhang, Hong > <mailto:hzh...@mcs.anl.gov>> wrote:
>> 
>> Stefano:
>> Now, we need address this bug report: enable 
>> MatHasOperation(C,MATOP_MAT_MULT,) for matrix products, e.g., C=A*B, 
>> which is related to your issue https://gitlab.com/petsc/petsc/-/issues/608 
>> <https://gitlab.com/petsc/petsc/-/issues/608>.
>> 
>> In petsc-3.13:
>> 1) MATOP_MAT_MULT, ..., MATOP_MATMAT_MULT are removed from the MATOP table 
>> (they are still listed in petscmat.h -- an overlook, I'll remove them). 
>> MATOP_MAT_MULT_SYMBOLIC/NUMERIC ... are still in the table.
>> 2) MatHasOperation(C,...) must be called for the matrix product C, not 
>> matrix A or B (slepc needs to fix this after this reported bug is fixed).
>> 
>> Like MatSetOption(), MatHasOperation() must be called AFTER MatSetType(). 
>> You moved MatSetType() from MatProductSetFromOptions() back to 
>> MatProductSymbolic() in your latest patch, thus user has to call 
>> MatHasOption() after MatProductSymbolic():
>> 
>> MatProductCreate(A,B,NULL,);
>> MatProductSetType(C,...);
>> ...
>> MatProductSetFromOptions();   //if the product is not supported for the 
>> given mat types, currently petsc crashes here, which we can replace with an 
>> error output
>> 
>> MatProductSymbloc(); -> call MatSetType()
>> MatHasOperation(C,MATOP_MAT_MULT,)
>> 
>> Question: how to call MatHasOperation(C,..) when MatProductSymbloc() is not 
>> supported?
>> 
>> My fix to this bug:
>> Resume MatSetType() in MatProductSetFromOptions(). Then user calls:
>> 
>> MatProductCreate(A,B,NULL,);
>> MatProductSetType(C,...);
>> ...
>> MatProductSetFromOptions(C);  //if the product is not supported for the 
>> given mat types, C->ops->productsymbolic=NULL;
>> MatHasOperation(C,MATOP_PRODUCTSYMBOLIC,);
>> if (flg) { 
>>MatProductSymbolic(C);
>>...
>> } else {
>>MatDestroy();
>>...
>> }
>> 
>> Either you take care of this bug report, or let me know your thoughts about 
>> how to fix this bug.
>> Hong
>> From: Zhang, Hong mailto:hzh...@mcs.anl.gov>>
&g