Re: [petsc-users] Matvecs and KSPSolves with multiple vectors

2023-12-20 Thread Sreeram R Venkat 
Would using the CHOLMOD Cholesky factorization (
https://petsc.org/release/manualpages/Mat/MATSOLVERCHOLMOD/) let us do the
factorization on device as well?



On Wed, Dec 20, 2023 at 1:21 PM Pierre Jolivet  wrote:

>
>
> On 20 Dec 2023, at 8:42 AM, Sreeram R Venkat  wrote:
>
> Ok, I think the error I'm getting has something to do with how the
> multiple solves are being done in succession. I'll try to see if there's
> anything I'm doing wrong there.
>
> One question about the -pc_type lu -ksp_type preonly method: do you know
> which parts of the solve (factorization/triangular solves) are done on host
> and which are done on device?
>
>
> I think only the triangular solves can be done on device.
> Since you have many right-hand sides, it may not be that bad.
> GPU people will hopefully give you a more insightful answer.
>
> Thanks,
> Pierre
>
> Thanks,
> Sreeram
>
> On Sat, Dec 16, 2023 at 10:56 PM Pierre Jolivet  wrote:
>
>> Unfortunately, I am not able to reproduce such a failure with your input
>> matrix.
>> I’ve used ex79 that I linked previously and the system is properly solved.
>> $ ./ex79 -pc_type hypre -ksp_type hpddm -ksp_hpddm_type cg
>> -ksp_converged_reason -ksp_view_mat ascii::ascii_info -ksp_view_rhs
>> ascii::ascii_info
>> Linear solve converged due to CONVERGED_RTOL iterations 6
>> Mat Object: 1 MPI process
>>   type: seqaijcusparse
>>   rows=289, cols=289
>>   total: nonzeros=2401, allocated nonzeros=2401
>>   total number of mallocs used during MatSetValues calls=0
>> not using I-node routines
>> Mat Object: 1 MPI process
>>   type: seqdensecuda
>>   rows=289, cols=10
>>   total: nonzeros=2890, allocated nonzeros=2890
>>   total number of mallocs used during MatSetValues calls=0
>>
>> You mentioned in a subsequent email that you are interested in systems
>> with at most 1E6 unknowns, and up to 1E4 right-hand sides.
>> I’m not sure you can expect significant gains from using GPU for such
>> systems.
>> Probably, the fastest approach would indeed be -pc_type lu -ksp_type
>> preonly -ksp_matsolve_batch_size 100 or something, depending on the memory
>> available on your host.
>>
>> Thanks,
>> Pierre
>>
>> On 15 Dec 2023, at 9:52 PM, Sreeram R Venkat  wrote:
>>
>> Here are the ksp_view files.  I set the options
>> -ksp_error_if_not_converged to try to get the vectors that caused the
>> error. I noticed that some of the KSPMatSolves converge while others don't.
>> In the code, the solves are called as:
>>
>> input vector v --> insert data of v into a dense mat --> KSPMatSolve()
>> --> MatMatMult() --> KSPMatSolve() --> insert data of dense mat into output
>> vector w -- output w
>>
>> The operator used in the KSP is a Laplacian-like operator, and the
>> MatMatMult is with a Mass Matrix. The whole thing is supposed to be a solve
>> with a biharmonic-like operator. I can also run it with only the first
>> KSPMatSolve (i.e. just a Laplacian-like operator). In that case, the KSP
>> reportedly converges after 0 iterations (see the next line), but this
>> causes problems in other parts of the code later on.
>>
>> I saw that sometimes the first KSPMatSolve "converges" after 0 iterations
>> due to CONVERGED_RTOL. Then, the second KSPMatSolve produces a NaN/Inf. I
>> tried setting ksp_min_it, but that didn't seem to do anything.
>>
>> I'll keep trying different options and also try to get the MWE made (this
>> KSPMatSolve is pretty performance critical for us).
>>
>> Thanks for all your help,
>> Sreeram
>>
>> On Fri, Dec 15, 2023 at 1:01 AM Pierre Jolivet  wrote:
>>
>>>
>>> On 14 Dec 2023, at 11:45 PM, Sreeram R Venkat 
>>> wrote:
>>>
>>> Thanks, I will try to create a minimal reproducible example. This may
>>> take me some time though, as I need to figure out how to extract only the
>>> relevant parts (the full program this solve is used in is getting quite
>>> complex).
>>>
>>>
>>> You could just do -ksp_view_mat binary:Amat.bin -ksp_view_pmat
>>> binary:Pmat.bin -ksp_view_rhs binary:rhs.bin and send me those three files
>>> (I’m guessing your are using double-precision scalars with 32-bit PetscInt).
>>>
>>> I'll also try out some of the BoomerAMG options to see if that helps.
>>>
>>>
>>> These should work (this is where all “PCMatApply()-ready” PC are being
>>> tested):
>>> https://petsc.org/release/src/ksp/ksp/tutorials/ex79.c.html#line215
>>> You can see it’s also testing PCHYPRE + KSPHPDDM on device (but not with
>>> HIP).
>>> I’m aware the performance should not be optimal (see your comment about
>>> host/device copies), I’ve money to hire someone to work on this but: a) I
>>> need to find the correct engineer/post-doc, b) I currently don’t have good
>>> use cases (of course, I could generate a synthetic benchmark, for science).
>>> So even if you send me the three Mat, a MWE would be appreciated if the
>>> KSPMatSolve() is performance-critical for you (see point b) from above).
>>>
>>> Thanks,
>>> Pierre
>>>
>>> Thanks,
>>> Sreeram
>>>
>>> On Thu, Dec 14, 2023, 1:12 PM Pierre Jolivet

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Shatanawi, Sawsan Muhammad via petsc-users 
Hello Matthew,

Thank you for your help. I am sorry that I keep coming back with my error 
messages, but I reached a point that I don't know how to fix them, and I don't 
understand them easily.
The list of errors is getting shorter, now I am getting the attached error 
messages

Thank you again,

Sawsan

From: Matthew Knepley 
Sent: Wednesday, December 20, 2023 6:54 PM
To: Shatanawi, Sawsan Muhammad 
Cc: Barry Smith ; petsc-users@mcs.anl.gov 

Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code


[EXTERNAL EMAIL]

On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Do not destroy the PC from KSPGetPC()

  THanks,

 Matt

Bests,
Sawsan

From: Barry Smith mailto:bsm...@petsc.dev>>
Sent: Wednesday, December 20, 2023 6:32 PM
To: Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>>
Cc: Mark Adams mailto:mfad...@lbl.gov>>; 
petsc-users@mcs.anl.gov 
mailto:petsc-users@mcs.anl.gov>>
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code


[EXTERNAL EMAIL]

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr)
call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
call KSPSetPC(ksp, pc,ierr)  ! Associate the preconditioner with the KSP 
solver

do

call KSPGetPC(ksp,pc,ierr)
call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve



On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:

Hello,
I don't think that I set preallocation values when I created the matrix, would 
you please have look at my code. It is just the petsc related part from my code.
I was able to fix some of the error messages. Now I have a new set of error 
messages related to the KSP solver (attached)

I appreciate your help

Sawsan

From: Mark Adams mailto:mfad...@lbl.gov>>
Sent: Wednesday, December 20, 2023 6:44 AM
To: Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>>
Cc: petsc-users@mcs.anl.gov 
mailto:petsc-users@mcs.anl.gov>>
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code

[EXTERNAL EMAIL]
Did you set preallocation values when you created the matrix?
Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>> wrote:
Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) 
then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for 
iOS

From: Mark Adams mailto:mfad...@lbl.gov>>
Sent: Wednesday, December 20, 2023 2:48 AM
To: Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>>
Cc: petsc-users@mcs.anl.gov 
mailto:petsc-users@mcs.anl.gov>>
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code

[EXTERNAL EMAIL]
I am guessing that you are creating a matrix, adding to it, finalizing it 
("assembly"), and then adding to it again, which is fine, but you are adding 
new non-zeros to the sparsity pattern.
If this is what you want then you can tell the matrix to let you do that.
Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Hello everyone,

I hope this email finds you well.

 My Name is Sawsan Shatanawi, and I am currently working on developing a 
Fortran code for simulating groundwater flow in a 3D system. The code involves 
solving a nonlinear system, and I have created the matrix to be solved using 
the PCG solver and Picard iteration. However, when I tried to assign it as a 
PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code 
with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

 Sawsan





--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/
[sawsan.shatanawi@login-p2n04 code_allocate]$ ./compile.sh
mpif90 -fPIC -g -O0

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Matthew Knepley 
On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello Barry,
>
> Thank you a lot for your help, Now I am getting the attached error message.
>

Do not destroy the PC from KSPGetPC()

  THanks,

 Matt


> Bests,
> Sawsan
> --
> *From:* Barry Smith 
> *Sent:* Wednesday, December 20, 2023 6:32 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* Mark Adams ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
> Instead of
>
> call PCCreate(PETSC_COMM_WORLD, pc, ierr)
> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
> call KSPSetPC(ksp, pc,ierr)  ! Associate the preconditioner with the
> KSP solver
>
> do
>
> call KSPGetPC(ksp,pc,ierr)
> call PCSetType(pc, PCILU,ierr)
>
> Do not call KSPSetUp(). It will be taken care of automatically during the
> solve
>
>
>
> On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Hello,
> I don't think that I set preallocation values when I created the matrix,
> would you please have look at my code. It is just the petsc related part
> from my code.
> I was able to fix some of the error messages. Now I have a new set of
> error messages related to the KSP solver (attached)
>
> I appreciate your help
>
> Sawsan
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 6:44 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> Did you set preallocation values when you created the matrix?
> Don't do that.
>
> On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <
> sawsan.shatan...@wsu.edu> wrote:
>
> Hello,
>
> I am trying to create a sparse matrix( which is as I believe a zero
> matrix) then adding some nonzero elements to it over a loop, then
> assembling it
>
> Get Outlook for iOS
> 
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 2:48 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> I am guessing that you are creating a matrix, adding to it, finalizing it
> ("assembly"), and then adding to it again, which is fine, but you are
> adding new non-zeros to the sparsity pattern.
> If this is what you want then you can tell the matrix to let you do that.
> Otherwise you have a bug.
>
> Mark
>
> On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users
>  wrote:
>
> Hello everyone,
>
> I hope this email finds you well.
>
>  My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly asking if someone can help me, I would be happy to share my
> code with him/her.
>
> Please find the attached file contains a list of errors I have gotten
>
> Thank you in advance for your time and assistance.
>
> Best regards,
>
>  Sawsan
>
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Shatanawi, Sawsan Muhammad via petsc-users 
Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Bests,
Sawsan

From: Barry Smith 
Sent: Wednesday, December 20, 2023 6:32 PM
To: Shatanawi, Sawsan Muhammad 
Cc: Mark Adams ; petsc-users@mcs.anl.gov 

Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code


[EXTERNAL EMAIL]

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr)
call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
call KSPSetPC(ksp, pc,ierr)  ! Associate the preconditioner with the KSP 
solver

do

call KSPGetPC(ksp,pc,ierr)
call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve



On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users 
 wrote:

Hello,
I don't think that I set preallocation values when I created the matrix, would 
you please have look at my code. It is just the petsc related part from my code.
I was able to fix some of the error messages. Now I have a new set of error 
messages related to the KSP solver (attached)

I appreciate your help​

Sawsan

From: Mark Adams mailto:mfad...@lbl.gov>>
Sent: Wednesday, December 20, 2023 6:44 AM
To: Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>>
Cc: petsc-users@mcs.anl.gov 
mailto:petsc-users@mcs.anl.gov>>
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code

[EXTERNAL EMAIL]
Did you set preallocation values when you created the matrix?
Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>> wrote:
Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) 
then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for 
iOS

From: Mark Adams mailto:mfad...@lbl.gov>>
Sent: Wednesday, December 20, 2023 2:48 AM
To: Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>>
Cc: petsc-users@mcs.anl.gov 
mailto:petsc-users@mcs.anl.gov>>
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code

[EXTERNAL EMAIL]
I am guessing that you are creating a matrix, adding to it, finalizing it 
("assembly"), and then adding to it again, which is fine, but you are adding 
new non-zeros to the sparsity pattern.
If this is what you want then you can tell the matrix to let you do that.
Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Hello everyone,

I hope this email finds you well.

 My Name is Sawsan Shatanawi, and I am currently working on developing a 
Fortran code for simulating groundwater flow in a 3D system. The code involves 
solving a nonlinear system, and I have created the matrix to be solved using 
the PCG solver and Picard iteration. However, when I tried to assign it as a 
PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code 
with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

 Sawsan



[sawsan.shatanawi@login-p2n04 code_allocate]$ ./compile.sh
mpif90 -fPIC -g -O0 -cpp 
-I/opt/apps/petsc/3.20.0/intel/20.2/mpich/3.3.2/include-c -o 
GW_solver_try.o GW_solver_try.F90
mpif90 -fPIC -g -O0 -cpp 
-I/opt/apps/petsc/3.20.0/intel/20.2/mpich/3.3.2/include   
-L/opt/apps/petsc/3.20.0/intel/20.2/mpich/3.3.2/lib
-Wl,-rpath,/opt/apps/petsc/3.20.0/intel/20.2/mpich/3.3.2/lib  -o GW.exe 
GW_constants.o GW_properties.o GW_initial.o GW_boundary_conditions.o 
GW_elevation.o GW_conductance.o GW_recharge.o GW_pumping.o GW_SW_TEST.o 
GW_AMatrix_RHS_TRY_2.o GW_solver_try.o test_main.o -lpetsc   -lm
  5.200E-03
   50.0
   10.0
  0.000E+00
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: Corrupt argument: https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: Object already free: Parameter # 1
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.20.0, Sep 28, 2023
[0]PETSC ERROR: ./GW.exe on a  named login-p2n04.kamiak.wsu.edu by 
sawsan.shatanawi Wed Dec 20 18:45:25 2023
[0]PETSC ERROR: Configure options 
--prefix=/opt/apps/petsc/3.20.0/intel/20.2/mpich/3.3.2 
--with-blaslapack-dir=/opt/apps/intel/20.2/compilers_and_libraries_2020.2.254/linux/mkl
 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
[0]PETSC ERROR: #1 PCDestroy() at

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Barry Smith 

Instead of 

call PCCreate(PETSC_COMM_WORLD, pc, ierr)
call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
call KSPSetPC(ksp, pc,ierr)  ! Associate the preconditioner with the KSP 
solver

do

call KSPGetPC(ksp,pc,ierr)
call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve



> On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users 
>  wrote:
> 
> Hello, 
> I don't think that I set preallocation values when I created the matrix, 
> would you please have look at my code. It is just the petsc related part from 
> my code.
> I was able to fix some of the error messages. Now I have a new set of error 
> messages related to the KSP solver (attached)
> 
> I appreciate your help​
> 
> Sawsan
> From: Mark Adams mailto:mfad...@lbl.gov>>
> Sent: Wednesday, December 20, 2023 6:44 AM
> To: Shatanawi, Sawsan Muhammad  >
> Cc: petsc-users@mcs.anl.gov  
> mailto:petsc-users@mcs.anl.gov>>
> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran 
> Groundwater Flow Simulation Code
>  
> [EXTERNAL EMAIL]
> Did you set preallocation values when you created the matrix?
> Don't do that.
> 
> On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad 
> mailto:sawsan.shatan...@wsu.edu>> wrote:
> Hello, 
> 
> I am trying to create a sparse matrix( which is as I believe a zero matrix) 
> then adding some nonzero elements to it over a loop, then assembling it 
> 
> Get Outlook for iOS 
> 
> From: Mark Adams mailto:mfad...@lbl.gov>>
> Sent: Wednesday, December 20, 2023 2:48 AM
> To: Shatanawi, Sawsan Muhammad  >
> Cc: petsc-users@mcs.anl.gov  
> mailto:petsc-users@mcs.anl.gov>>
> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran 
> Groundwater Flow Simulation Code
>  
> [EXTERNAL EMAIL]
> I am guessing that you are creating a matrix, adding to it, finalizing it 
> ("assembly"), and then adding to it again, which is fine, but you are adding 
> new non-zeros to the sparsity pattern.
> If this is what you want then you can tell the matrix to let you do that.
> Otherwise you have a bug.
> 
> Mark
> 
> On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users 
> mailto:petsc-users@mcs.anl.gov>> wrote:
> Hello everyone,
> 
> I hope this email finds you well.
> 
>  My Name is Sawsan Shatanawi, and I am currently working on developing a 
> Fortran code for simulating groundwater flow in a 3D system. The code 
> involves solving a nonlinear system, and I have created the matrix to be 
> solved using the PCG solver and Picard iteration. However, when I tried to 
> assign it as a PETSc matrix I started getting a lot of error messages.
> 
> I am kindly asking if someone can help me, I would be happy to share my code 
> with him/her.
> 
> Please find the attached file contains a list of errors I have gotten
> 
> Thank you in advance for your time and assistance.
> Best regards,
> 
>  Sawsan
> 
> 
>

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Shatanawi, Sawsan Muhammad via petsc-users 
Hello,
I don't think that I set preallocation values when I created the matrix, would 
you please have look at my code. It is just the petsc related part from my code.
I was able to fix some of the error messages. Now I have a new set of error 
messages related to the KSP solver (attached)

I appreciate your help​

Sawsan

From: Mark Adams 
Sent: Wednesday, December 20, 2023 6:44 AM
To: Shatanawi, Sawsan Muhammad 
Cc: petsc-users@mcs.anl.gov 
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code


[EXTERNAL EMAIL]

Did you set preallocation values when you created the matrix?
Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>> wrote:
Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) 
then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for 
iOS

From: Mark Adams mailto:mfad...@lbl.gov>>
Sent: Wednesday, December 20, 2023 2:48 AM
To: Shatanawi, Sawsan Muhammad 
mailto:sawsan.shatan...@wsu.edu>>
Cc: petsc-users@mcs.anl.gov 
mailto:petsc-users@mcs.anl.gov>>
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code


[EXTERNAL EMAIL]

I am guessing that you are creating a matrix, adding to it, finalizing it 
("assembly"), and then adding to it again, which is fine, but you are adding 
new non-zeros to the sparsity pattern.
If this is what you want then you can tell the matrix to let you do that.
Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Hello everyone,

I hope this email finds you well.

 My Name is Sawsan Shatanawi, and I am currently working on developing a 
Fortran code for simulating groundwater flow in a 3D system. The code involves 
solving a nonlinear system, and I have created the matrix to be solved using 
the PCG solver and Picard iteration. However, when I tried to assign it as a 
PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code 
with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

 Sawsan



Matrix_RHS.F90
Description: Matrix_RHS.F90
 5.200E-03
   50.0
   10.0
  0.000E+00
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: You requested a vector from a KSP that cannot provide one
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.20.0, Sep 28, 2023
[0]PETSC ERROR: ./GW.exe on a  named login-p2n02.kamiak.wsu.edu by 
sawsan.shatanawi Wed Dec 20 17:44:37 2023
[0]PETSC ERROR: Configure options 
--prefix=/opt/apps/petsc/3.20.0/intel/20.2/mpich/3.3.2 
--with-blaslapack-dir=/opt/apps/intel/20.2/compilers_and_libraries_2020.2.254/linux/mkl
 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
[0]PETSC ERROR: #1 KSPCreateVecs() at 
/weka/data/kamiak/rohit.dhariwal/software_installs/petsc/3.20.0/petsc-3.20.0/src/ksp/ksp/interface/iterativ.c:1785
[0]PETSC ERROR: #2 KSPSetWorkVecs() at 
/weka/data/kamiak/rohit.dhariwal/software_installs/petsc/3.20.0/petsc-3.20.0/src/ksp/ksp/interface/iterativ.c:1853
[0]PETSC ERROR: #3 KSPSetUp_CG() at 
/weka/data/kamiak/rohit.dhariwal/software_installs/petsc/3.20.0/petsc-3.20.0/src/ksp/ksp/impls/cg/cg.c:79
[0]PETSC ERROR: #4 KSPSetUp() at 
/weka/data/kamiak/rohit.dhariwal/software_installs/petsc/3.20.0/petsc-3.20.0/src/ksp/ksp/interface/itfunc.c:377
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR:   It appears a new error in the code was triggered after a 
previous error, possibly because:
[0]PETSC ERROR:   -  The first error was not properly handled via (for example) 
the use of
[0]PETSC ERROR:  PetscCall(TheFunctionThatErrors()); or
[0]PETSC ERROR:   -  The second error was triggered while handling the first 
error.
[0]PETSC ERROR:   Above is the traceback for the previous unhandled error, 
below the traceback for the next error
[0]PETSC ERROR:   ALL ERRORS in the PETSc libraries are fatal, you should add 
the appropriate error checking to the code
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Mat object's type is not set: Argument # 1
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.20.0, Sep 28, 2023
[0]PETSC ERROR: ./GW.exe on a  named login-p2n02.kamiak.wsu.edu by

Re: [petsc-users] SLEPc/NEP for shell matrice T(lambda) and T'(lambda)

2023-12-20 Thread Kenneth C Hall 
Jose,



I have been revisiting the issue of SLEPc/NEP for shell matrices T(lambda) and 
T'(lambda).

I am having problems running SLEPc/NEP with -nep_type nleigs.



I have compiled two versions of PETSc/SLEPc:



petsc-arch-real / slepc-arch-real

./configure --with-cc=gcc-13 --with-cxx=g++-13 --with-fc=gfortran 
--COPTFLAGS='-O3 -fopenmp' --CXXOPTFLAGS='-O3 -fopenmp'

--FOPTFLAGS='-O3 -fopenmp' --with-debugging=1 
--with-logging=1 --with-scalar-type=real --with-precision=double

--download-fblaslapack --with-openmp --with-mpi=0





petsc-arch-complex / slepc-arch-complex

./configure --with-cc=gcc-13 --with-cxx=g++-13 --with-fc=gfortran 
--COPTFLAGS='-O3 -fopenmp' --CXXOPTFLAGS='-O3 -fopenmp'

--FOPTFLAGS='-O3 -fopenmp' --with-debugging=1 
--with-logging=1 --with-scalar-type=complex --with-precision=double

--download-fblaslapack --with-openmp --with-mpi=0



I use gfortran on an Apple Mac Mini M1.  Both the PETSc and SLEPc versions are 
the latest development versions as of today (a6690fd8 and 267bd1cd, 
respectively).



I ran the ex54f90 test cases:



%  main-arch-real -nep_type slp -nep_slp_ksp_type gmres 
-nep_slp_pc_type none

   %  main-arch-complex -nep_type slp -nep_slp_ksp_type gmres 
-nep_slp_pc_type none

   %  main-arch-real  -nep_type nleigs -rg_interval_endpoints 0.2,1.1 
-nep_nleigs_ksp_type gmres -nep_nleigs_pc_type none

   %  main-arch-complex -nep_type nleigs -rg_interval_endpoints 
0.2,1.1,-.1,.1 -nep_nleigs_ksp_type gmres -nep_nleigs_pc_type none



Both the slp cases ran as expected and gave the correct answer.

However, both the real and complex architectures failed for the nleigs case.



For the complex case, none of the callback functions appear to have been called.

For the real case, only the MatMult_A routine appears to be called, 100 times 
and returns each time, sweeping over lambda from 0.2 to 1.1.



Any suggestions would be welcome.



Best regards,

Kenneth Hall





On 10/18/23, 9:16 AM, "Jose E. Roman" mailto:jro...@dsic.upv.es>> wrote:





By the way, the MATOP_DESTROY stuff produced segmentation fault in some 
compilers (in gfortran it worked well). The reason was having the callback 
functions inside CONTAINS, that is why we have removed it and used regular 
subroutines instead.





Jose









> El 18 oct 2023, a las 15:11, Kenneth C Hall  > escribió:

>

> Jose,

>

> Thank you. I have downloaded and will take a look. I will try the new example 
> and then implement in my actual problem. I will keep you posted as to my 
> results.

>

> Thank you and best regards,

> Kenneth

>

> From: Jose E. Roman mailto:jro...@dsic.upv.es>>

> Sent: Tuesday, October 17, 2023 2:31 PM

> To: Kenneth C Hall mailto:kenneth.c.h...@duke.edu>>

> Cc: petsc-users@mcs.anl.gov  
> mailto:petsc-users@mcs.anl.gov>>

> Subject: Re: [petsc-users] SLEPc/NEP for shell matrice T(lambda) and 
> T'(lambda)

>

> Kenneth,

>

> I have worked a bit more on your example and put it in SLEPc 
> https://urldefense.com/v3/__https://gitlab.com/slepc/slepc/-/merge_requests/596__;!!OToaGQ!oSqCpmczx5VDi5025aO5T3WqW-MwGnKUSzxKVkdyXTHo9vuxP4GYnDfMoYxavvWRAA0WdcwX3tiVaiXWT0dh2-o$
>  
> 

> This version also has MATOP_DESTROY to avoid memory leaks.

>

> Thanks.

> Jose

>

>

> > El 12 oct 2023, a las 20:59, Kenneth C Hall  > > escribió:

> >

> > Jose,

> >

> > Thanks very much for this. I will give it a try and let you know how it 
> > works.

> >

> > Best regards,

> > Kenneth

> >

> > From: Jose E. Roman mailto:jro...@dsic.upv.es>>

> > Date: Thursday, October 12, 2023 at 2:12 PM

> > To: Kenneth C Hall  > >

> > Cc: petsc-users@mcs.anl.gov  
> > mailto:petsc-users@mcs.anl.gov>>

> > Subject: Re: [petsc-users] SLEPc/NEP for shell matrice T(lambda) and 
> > T'(lambda)

> >

> > I am attaching your example modified with the context stuff.

> >

> > With the PETSc branch that I indicated, now it works with NLEIGS, for 
> > instance:

> >

> > $ ./test_nep -nep_nleigs_ksp_type gmres -nep_nleigs_pc_type none 
> > -rg_interval_endpoints 0.2,1.1 -nep_target 0.8 -nep_nev 5 -n 400 
> > -nep_monitor -nep_view -nep_error_relative ::ascii_info_detail

> >

> > And also other solvers such as SLP:

> >

> > $ ./test_nep -nep_type slp -nep_slp_ksp_type gmres -nep_slp_pc_type none 
> > -nep_target 0.8 -nep_nev 5 -n 400 -nep_monitor -nep_error_relative 
> > ::ascii_info_detail

> >

> > I will clean the example code an add it as a SLEPc example.

> >

> > Regards,

> > Jose

> >

> >

> > > El 11

Re: [petsc-users] fortran interface to snes matrix-free jacobian

2023-12-20 Thread Barry Smith 

   I apologize; please ignore my answer below. Use MatCreateShell() as 
indicated by Jed.


> On Dec 20, 2023, at 2:14 PM, Barry Smith  wrote:
> 
> 
> 
>> On Dec 20, 2023, at 11:44 AM, Yi Hu  wrote:
>> 
>> Dear Jed,
>> 
>> Thanks for your reply. I have an analytical one to implement.
>> 
>> Best, Yi
>> 
>> -Original Message-
>> From: Jed Brown  
>> Sent: Wednesday, December 20, 2023 5:40 PM
>> To: Yi Hu ; petsc-users@mcs.anl.gov
>> Subject: Re: [petsc-users] fortran interface to snes matrix-free jacobian
>> 
>> Are you wanting an analytic matrix-free operator or one created for you 
>> based on finite differencing? If the latter, just use -snes_mf or 
>> -snes_mf_operator.
>> 
>> https://petsc.org/release/manual/snes/#jacobian-evaluation
>> 
>> Yi Hu  writes:
>> 
>>> Dear PETSc team,
>>> 
>>> My  solution scheme relies on a matrix-free jacobian in the SNES solver. I 
>>> saw the useful C interface like MatCreateSNESMF(), 
>>> DMSNESCreateJacobianMF(). I am wondering if you have the fortran 
>>> equivalence?
> 
>   You can use DMSNESCreateJacobianMF() (MatCreateSNESMF is not appropriate 
> when you are providing the operation).
> 
> 
>>> 
>>> I think for my problem in the main program I need to do 
>>> DMDASNESsetJacobianLocal(DM, INSERT_VALUES, myJacobian, ctx, err_petsc). 
>>> Then in myJacobian() subroutine I have to create the operator from 
>>> DMSNESCreateJacobianMF(), and register my own MATOP_MULT from 
>>> MatShellSetOperation(). Am I correct?
> 
> Not exactly. Do not use DMDASNESsetJacobianLocal() use 
> DMSNESCreateJacobianMF() to create a Mat J where you create the SNES and use 
> SNESSetJacobian()  and pass the J matrix in along with myJacobian().
> 
>>> 
>>> Are these fortran subroutines available? I saw an example in ts module 
>>> as ex22f_mf.F90 which behaves similar as what I would like to do. Because I 
>>> would like to use ngmres, I then need to stay in the SNES.
>>> 
>>> Thanks for your help.
>>> 
>>> Best wishes,
>>> Yi
>>> 
>>> -
>>> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>>> 
>>> Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1
>>> D-40237 Düsseldorf
>>> 
>>> Handelsregister B 2533
>>> Amtsgericht Düsseldorf
>>> 
>>> Geschäftsführung
>>> Prof. Dr. Gerhard Dehm
>>> Prof. Dr. Jörg Neugebauer
>>> Prof. Dr. Dierk Raabe
>>> Dr. Kai de Weldige
>>> 
>>> Ust.-Id.-Nr.: DE 11 93 58 514
>>> Steuernummer: 105 5891 1000
>>> 
>>> 
>>> Please consider that invitations and e-mails of our institute are only 
>>> valid if they end with …@mpie.de.
>>> If you are not sure of the validity please contact r...@mpie.de
>>> 
>>> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails 
>>> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
>>> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
>>> -
>> 
>> 
>> -
>> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>> 
>> Max-Planck-Institut für Eisenforschung GmbH
>> Max-Planck-Straße 1
>> D-40237 Düsseldorf
>> 
>> Handelsregister B 2533 
>> Amtsgericht Düsseldorf
>> 
>> Geschäftsführung
>> Prof. Dr. Gerhard Dehm
>> Prof. Dr. Jörg Neugebauer
>> Prof. Dr. Dierk Raabe
>> Dr. Kai de Weldige
>> 
>> Ust.-Id.-Nr.: DE 11 93 58 514 
>> Steuernummer: 105 5891 1000
>> 
>> 
>> Please consider that invitations and e-mails of our institute are 
>> only valid if they end with …@mpie.de. 
>> If you are not sure of the validity please contact r...@mpie.de
>> 
>> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
>> aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
>> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
>> -

Re: [petsc-users] fortran interface to snes matrix-free jacobian

2023-12-20 Thread Barry Smith 



> On Dec 20, 2023, at 11:44 AM, Yi Hu  wrote:
> 
> Dear Jed,
> 
> Thanks for your reply. I have an analytical one to implement.
> 
> Best, Yi
> 
> -Original Message-
> From: Jed Brown  
> Sent: Wednesday, December 20, 2023 5:40 PM
> To: Yi Hu ; petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] fortran interface to snes matrix-free jacobian
> 
> Are you wanting an analytic matrix-free operator or one created for you based 
> on finite differencing? If the latter, just use -snes_mf or -snes_mf_operator.
> 
> https://petsc.org/release/manual/snes/#jacobian-evaluation
> 
> Yi Hu  writes:
> 
>> Dear PETSc team,
>> 
>> My  solution scheme relies on a matrix-free jacobian in the SNES solver. I 
>> saw the useful C interface like MatCreateSNESMF(), DMSNESCreateJacobianMF(). 
>> I am wondering if you have the fortran equivalence?

   You can use DMSNESCreateJacobianMF() (MatCreateSNESMF is not appropriate 
when you are providing the operation).
 
   
>> 
>> I think for my problem in the main program I need to do 
>> DMDASNESsetJacobianLocal(DM, INSERT_VALUES, myJacobian, ctx, err_petsc). 
>> Then in myJacobian() subroutine I have to create the operator from 
>> DMSNESCreateJacobianMF(), and register my own MATOP_MULT from 
>> MatShellSetOperation(). Am I correct?

Not exactly. Do not use DMDASNESsetJacobianLocal() use DMSNESCreateJacobianMF() 
to create a Mat J where you create the SNES and use SNESSetJacobian()  and pass 
the J matrix in along with myJacobian().

>> 
>> Are these fortran subroutines available? I saw an example in ts module 
>> as ex22f_mf.F90 which behaves similar as what I would like to do. Because I 
>> would like to use ngmres, I then need to stay in the SNES.
>> 
>> Thanks for your help.
>> 
>> Best wishes,
>> Yi
>> 
>> -
>> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>> 
>> Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1
>> D-40237 Düsseldorf
>> 
>> Handelsregister B 2533
>> Amtsgericht Düsseldorf
>> 
>> Geschäftsführung
>> Prof. Dr. Gerhard Dehm
>> Prof. Dr. Jörg Neugebauer
>> Prof. Dr. Dierk Raabe
>> Dr. Kai de Weldige
>> 
>> Ust.-Id.-Nr.: DE 11 93 58 514
>> Steuernummer: 105 5891 1000
>> 
>> 
>> Please consider that invitations and e-mails of our institute are only 
>> valid if they end with …@mpie.de.
>> If you are not sure of the validity please contact r...@mpie.de
>> 
>> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails 
>> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
>> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
>> -
> 
> 
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
> 
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
> 
> Handelsregister B 2533 
> Amtsgericht Düsseldorf
> 
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
> 
> Ust.-Id.-Nr.: DE 11 93 58 514 
> Steuernummer: 105 5891 1000
> 
> 
> Please consider that invitations and e-mails of our institute are 
> only valid if they end with …@mpie.de. 
> If you are not sure of the validity please contact r...@mpie.de
> 
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -
>

Re: [petsc-users] fortran interface to snes matrix-free jacobian

2023-12-20 Thread Jed Brown 
Then just use MatShell. I see the docs need some work to clarify this, but 
MatCreateSNESMF is to specify matrix-free finite differencing from code 
(perhaps where one wants to customize parameters).

Yi Hu  writes:

> Dear Jed,
>
> Thanks for your reply. I have an analytical one to implement.
>
> Best, Yi
>
> -Original Message-
> From: Jed Brown  
> Sent: Wednesday, December 20, 2023 5:40 PM
> To: Yi Hu ; petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] fortran interface to snes matrix-free jacobian
>
> Are you wanting an analytic matrix-free operator or one created for you based 
> on finite differencing? If the latter, just use -snes_mf or -snes_mf_operator.
>
> https://petsc.org/release/manual/snes/#jacobian-evaluation
>
> Yi Hu  writes:
>
>> Dear PETSc team,
>>  
>> My  solution scheme relies on a matrix-free jacobian in the SNES solver. I 
>> saw the useful C interface like MatCreateSNESMF(), DMSNESCreateJacobianMF(). 
>> I am wondering if you have the fortran equivalence?
>>  
>> I think for my problem in the main program I need to do 
>> DMDASNESsetJacobianLocal(DM, INSERT_VALUES, myJacobian, ctx, err_petsc). 
>> Then in myJacobian() subroutine I have to create the operator from 
>> DMSNESCreateJacobianMF(), and register my own MATOP_MULT from 
>> MatShellSetOperation(). Am I correct? 
>>  
>> Are these fortran subroutines available? I saw an example in ts module 
>> as ex22f_mf.F90 which behaves similar as what I would like to do. Because I 
>> would like to use ngmres, I then need to stay in the SNES.
>>  
>> Thanks for your help.
>>  
>> Best wishes,
>> Yi
>>
>> -
>> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>>
>> Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1
>> D-40237 Düsseldorf
>>  
>> Handelsregister B 2533
>> Amtsgericht Düsseldorf
>>  
>> Geschäftsführung
>> Prof. Dr. Gerhard Dehm
>> Prof. Dr. Jörg Neugebauer
>> Prof. Dr. Dierk Raabe
>> Dr. Kai de Weldige
>>  
>> Ust.-Id.-Nr.: DE 11 93 58 514
>> Steuernummer: 105 5891 1000
>>
>>
>> Please consider that invitations and e-mails of our institute are only 
>> valid if they end with …@mpie.de.
>> If you are not sure of the validity please contact r...@mpie.de
>>
>> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails 
>> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
>> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
>> -
>
>
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>  
> Handelsregister B 2533 
> Amtsgericht Düsseldorf
>  
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>  
> Ust.-Id.-Nr.: DE 11 93 58 514 
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are 
> only valid if they end with …@mpie.de. 
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -

Re: [petsc-users] fortran interface to snes matrix-free jacobian

2023-12-20 Thread Yi Hu 
Dear Jed,

Thanks for your reply. I have an analytical one to implement.

Best, Yi

-Original Message-
From: Jed Brown  
Sent: Wednesday, December 20, 2023 5:40 PM
To: Yi Hu ; petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] fortran interface to snes matrix-free jacobian

Are you wanting an analytic matrix-free operator or one created for you based 
on finite differencing? If the latter, just use -snes_mf or -snes_mf_operator.

https://petsc.org/release/manual/snes/#jacobian-evaluation

Yi Hu  writes:

> Dear PETSc team,
>  
> My  solution scheme relies on a matrix-free jacobian in the SNES solver. I 
> saw the useful C interface like MatCreateSNESMF(), DMSNESCreateJacobianMF(). 
> I am wondering if you have the fortran equivalence?
>  
> I think for my problem in the main program I need to do 
> DMDASNESsetJacobianLocal(DM, INSERT_VALUES, myJacobian, ctx, err_petsc). Then 
> in myJacobian() subroutine I have to create the operator from 
> DMSNESCreateJacobianMF(), and register my own MATOP_MULT from 
> MatShellSetOperation(). Am I correct? 
>  
> Are these fortran subroutines available? I saw an example in ts module 
> as ex22f_mf.F90 which behaves similar as what I would like to do. Because I 
> would like to use ngmres, I then need to stay in the SNES.
>  
> Thanks for your help.
>  
> Best wishes,
> Yi
>
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>
> Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1
> D-40237 Düsseldorf
>  
> Handelsregister B 2533
> Amtsgericht Düsseldorf
>  
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>  
> Ust.-Id.-Nr.: DE 11 93 58 514
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are only 
> valid if they end with …@mpie.de.
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails 
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -


-
Stay up to date and follow us on LinkedIn, Twitter and YouTube.

Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf
 
Handelsregister B 2533 
Amtsgericht Düsseldorf
 
Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige
 
Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000


Please consider that invitations and e-mails of our institute are 
only valid if they end with …@mpie.de. 
If you are not sure of the validity please contact r...@mpie.de

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
-

Re: [petsc-users] fortran interface to snes matrix-free jacobian

2023-12-20 Thread Jed Brown 
Are you wanting an analytic matrix-free operator or one created for you based 
on finite differencing? If the latter, just use -snes_mf or -snes_mf_operator.

https://petsc.org/release/manual/snes/#jacobian-evaluation

Yi Hu  writes:

> Dear PETSc team,
>  
> My  solution scheme relies on a matrix-free jacobian in the SNES solver. I 
> saw the useful C interface like MatCreateSNESMF(), DMSNESCreateJacobianMF(). 
> I am wondering if you have the fortran equivalence?
>  
> I think for my problem in the main program I need to do 
> DMDASNESsetJacobianLocal(DM, INSERT_VALUES, myJacobian, ctx, err_petsc). Then 
> in myJacobian() subroutine I have to create the operator from 
> DMSNESCreateJacobianMF(), and register my own MATOP_MULT from 
> MatShellSetOperation(). Am I correct? 
>  
> Are these fortran subroutines available? I saw an example in ts module as 
> ex22f_mf.F90 which behaves similar as what I would like to do. Because I 
> would like to use ngmres, I then need to stay in the SNES.  
>  
> Thanks for your help.
>  
> Best wishes,
> Yi
>
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>  
> Handelsregister B 2533 
> Amtsgericht Düsseldorf
>  
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>  
> Ust.-Id.-Nr.: DE 11 93 58 514 
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are 
> only valid if they end with …@mpie.de. 
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -

[petsc-users] fortran interface to snes matrix-free jacobian

2023-12-20 Thread Yi Hu 
Dear PETSc team,
 
My  solution scheme relies on a matrix-free jacobian in the SNES solver. I saw 
the useful C interface like MatCreateSNESMF(), DMSNESCreateJacobianMF(). I am 
wondering if you have the fortran equivalence?
 
I think for my problem in the main program I need to do 
DMDASNESsetJacobianLocal(DM, INSERT_VALUES, myJacobian, ctx, err_petsc). Then 
in myJacobian() subroutine I have to create the operator from 
DMSNESCreateJacobianMF(), and register my own MATOP_MULT from 
MatShellSetOperation(). Am I correct? 
 
Are these fortran subroutines available? I saw an example in ts module as 
ex22f_mf.F90 which behaves similar as what I would like to do. Because I would 
like to use ngmres, I then need to stay in the SNES.  
 
Thanks for your help.
 
Best wishes,
Yi

-
Stay up to date and follow us on LinkedIn, Twitter and YouTube.

Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf
 
Handelsregister B 2533 
Amtsgericht Düsseldorf
 
Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige
 
Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000


Please consider that invitations and e-mails of our institute are 
only valid if they end with …@mpie.de. 
If you are not sure of the validity please contact r...@mpie.de

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
-

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Mark Adams 
Did you set preallocation values when you created the matrix?
Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <
sawsan.shatan...@wsu.edu> wrote:

> Hello,
>
> I am trying to create a sparse matrix( which is as I believe a zero
> matrix) then adding some nonzero elements to it over a loop, then
> assembling it
>
> Get Outlook for iOS 
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 2:48 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> I am guessing that you are creating a matrix, adding to it, finalizing it
> ("assembly"), and then adding to it again, which is fine, but you are
> adding new non-zeros to the sparsity pattern.
> If this is what you want then you can tell the matrix to let you do that.
> Otherwise you have a bug.
>
> Mark
>
> On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users
>  wrote:
>
>> Hello everyone,
>>
>> I hope this email finds you well.
>>
>>  My Name is Sawsan Shatanawi, and I am currently working on developing a
>> Fortran code for simulating groundwater flow in a 3D system. The code
>> involves solving a nonlinear system, and I have created the matrix to be
>> solved using the PCG solver and Picard iteration. However, when I tried
>> to assign it as a PETSc matrix I started getting a lot of error messages.
>>
>> I am kindly asking if someone can help me, I would be happy to share my
>> code with him/her.
>>
>> Please find the attached file contains a list of errors I have gotten
>>
>> Thank you in advance for your time and assistance.
>>
>> Best regards,
>>
>>  Sawsan
>>
>>

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Shatanawi, Sawsan Muhammad via petsc-users 
Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) 
then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for iOS

From: Mark Adams 
Sent: Wednesday, December 20, 2023 2:48 AM
To: Shatanawi, Sawsan Muhammad 
Cc: petsc-users@mcs.anl.gov 
Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater 
Flow Simulation Code


[EXTERNAL EMAIL]

I am guessing that you are creating a matrix, adding to it, finalizing it 
("assembly"), and then adding to it again, which is fine, but you are adding 
new non-zeros to the sparsity pattern.
If this is what you want then you can tell the matrix to let you do that.
Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Hello everyone,

I hope this email finds you well.

 My Name is Sawsan Shatanawi, and I am currently working on developing a 
Fortran code for simulating groundwater flow in a 3D system. The code involves 
solving a nonlinear system, and I have created the matrix to be solved using 
the PCG solver and Picard iteration. However, when I tried to assign it as a 
PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code 
with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

 Sawsan

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Matthew Knepley 
On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello everyone,
>
> I hope this email finds you well.
>
>  My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly asking if someone can help me, I would be happy to share my
> code with him/her.
>
> Please find the attached file contains a list of errors I have gotten
>

This error indicates that your preallocation is not sufficient for the
values you want to insert. Now in PETSc you can just remove your
preallocation, and PETSc will automatically allocate correctly.

  Thanks,

 Matt


> Thank you in advance for your time and assistance.
>
> Best regards,
>
>  Sawsan
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] [petsc-maint] DMSwarm on multiple processors

2023-12-20 Thread Matthew Knepley 
On Wed, Dec 20, 2023 at 4:12 AM Joauma Marichal <
joauma.maric...@uclouvain.be> wrote:

> Hello,
>
>
>
> I used Address Sanitizer on my laptop and I have no leaks.
>
> I do have access to another machine (managed by the same people as the
> previous one) but I obtain similar errors…
>

Let me understand:

1) You have run the exact same problem on two different parallel machines,
and gotten the same error,
 meaning on the second machine, it printed

malloc_consolidate(): invalid chunk size

Is this true?

 2) You have run the exact same problem on the same number of processes on
your own machine under Address Sanitizer with no errors?

  Thanks,

 Matt


> Thanks again for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *mardi, 19 décembre 2023 à 14:30
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Tue, Dec 19, 2023 at 5:11 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> I have used Address Sanitizer to check any memory errors. On my computer,
> no errors are found. Unfortunately, on the supercomputer that I am using, I
> get lots of errors… I attach my log files (running on 1 and 70 procs).
>
> Do you have any idea of what I could do?
>
>
>
> Run the same parallel configuration as you do on the supercomputer. If
> that is fine, I would suggest Address Sanitizer there. Something is
> corrupting the stack, and it appears that it is connected to that machine,
> rather than the library. Do you have access to a second parallel machine?
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Thanks a lot for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *lundi, 18 décembre 2023 à 12:00
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Mon, Dec 18, 2023 at 5:09 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> Sorry for the delay. I attach the file that I obtain when running the code
> with the debug mode.
>
>
>
> Okay, we can now see where this is happening:
>
>
>
> malloc_consolidate(): invalid chunk size
> [cns263:3265170] *** Process received signal ***
> [cns263:3265170] Signal: Aborted (6)
> [cns263:3265170] Signal code:  (-6)
> [cns263:3265170] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f3bd9148b20]
> [cns263:3265170] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f3bd9148a9f]
> [cns263:3265170] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f3bd911be05]
> [cns263:3265170] [ 3] /lib64/libc.so.6(+0x91037)[0x7f3bd918b037]
> [cns263:3265170] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f3bd919219c]
> [cns263:3265170] [ 5] /lib64/libc.so.6(+0x98b68)[0x7f3bd9192b68]
> [cns263:3265170] [ 6] /lib64/libc.so.6(+0x9af18)[0x7f3bd9194f18]
> [cns263:3265170] [ 7] /lib64/libc.so.6(__libc_malloc+0x1e2)[0x7f3bd9196822]
> [cns263:3265170] [ 8] /lib64/libc.so.6(posix_memalign+0x3c)[0x7f3bd91980fc]
> [cns263:3265170] [ 9]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocAlign+0x45)[0x7f3bda5f1625]
> [cns263:3265170] [10]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocA+0x297)[0x7f3bda5f1b07]
> [cns263:3265170] [11]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMCreate+0x5b)[0x7f3bdaa73c1b]
> [cns263:3265170] [12]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate+0x9)[0x7f3bdab0a2f9]
> [cns263:3265170] [13]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate3d+0x9a)[0x7f3bdab07dea]
> [cns263:3265170] [14] ./cobpor[0x402de8]
> [cns263:3265170] [15]
> /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3bd9134cf3]
> [cns263:3265170] [16] ./cobpor[0x40304e]
> [cns263:3265170] *** End of error message ***
>
>
>
> However, this is not great. First, the amount of memory being allocated is
> quite small, and this does not appear to be an Out of Memory error. Second,
> the error occurs in libc:
>
>
>
>   malloc_consolidate(): invalid chunk size
>
>
>
> which means something is wrong internally. I agree with this analysis (
> https://stackoverflow.com/questions/18760999/sample-example-program-to-get-the-malloc-consolidate-error)
> that says you have probably overwritten memory somewhere in your code. I
> recommend running under valgrind, or using Address Sanitizer from clang.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Thanks for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *jeudi, 23 novembre 2023 à 15:32
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Thu, Nov 23, 2023 at 9:01 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> My problem persists… Is there anything I could try?
>

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

2023-12-20 Thread Mark Adams 
I am guessing that you are creating a matrix, adding to it, finalizing it
("assembly"), and then adding to it again, which is fine, but you are
adding new non-zeros to the sparsity pattern.
If this is what you want then you can tell the matrix to let you do that.
Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello everyone,
>
> I hope this email finds you well.
>
>  My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly asking if someone can help me, I would be happy to share my
> code with him/her.
>
> Please find the attached file contains a list of errors I have gotten
>
> Thank you in advance for your time and assistance.
>
> Best regards,
>
>  Sawsan
>
>