Re: [petsc-users] PCFactorSetShiftType does not work in code but -pc_factor_set_shift_type works

2017-05-25 Thread Danyang Su 

Hi Hong,

It works like a charm. I really appreciate your help.

Regards,

Danyang


On 17-05-25 07:49 AM, Hong wrote:

Danyang:
You must access inner pc, then set shift. See
petsc/src/ksp/ksp/examples/tutorials/ex7.c

For example, I add following to 
petsc/src/ksp/ksp/examples/tutorials/ex2.c, line 191:

  PetscBool isbjacobi;
  PCpc;
  ierr = KSPGetPC(ksp,);CHKERRQ(ierr);
  ierr = 
PetscObjectTypeCompare((PetscObject)pc,PCBJACOBI,);CHKERRQ(ierr);

  if (isbjacobi) {
PetscInt nlocal;
KSP  *subksp;
PC   subpc;

ierr = KSPSetUp(ksp);CHKERRQ(ierr);
ierr = KSPGetPC(ksp,);CHKERRQ(ierr);

/* Extract the array of KSP contexts for the local blocks */
ierr = PCBJacobiGetSubKSP(pc,,NULL,);CHKERRQ(ierr);
printf("isbjacobi, nlocal %D, set option to subpc...\n",nlocal);
for (i=0; i> wrote:

Dear Hong,

I just tested with different number of processors for the
same matrix. It sometimes got "ERROR: Arguments are
incompatible" for different number of processors. It works
fine using 4, 8, or 24 processors, but failed with "ERROR:
Arguments are incompatible" using 16 or 48 processors. The
error information is attached. I tested this on my local
computer with 6 cores 12 threads. Any suggestion on this?

Thanks,

Danyang


On 17-05-24 12:28 PM, Danyang Su wrote:


Hi Hong,

Awesome. Thanks for testing the case. I will try your
options for the code and get back to you later.

Re: [petsc-users] PCFactorSetShiftType does not work in code but -pc_factor_set_shift_type works

2017-05-25 Thread Hong 
Danyang:
You must access inner pc, then set shift. See
petsc/src/ksp/ksp/examples/tutorials/ex7.c

For example, I add following to petsc/src/ksp/ksp/examples/tutorials/ex2.c,
line 191:
  PetscBool isbjacobi;
  PCpc;
  ierr = KSPGetPC(ksp,);CHKERRQ(ierr);
  ierr =
PetscObjectTypeCompare((PetscObject)pc,PCBJACOBI,);CHKERRQ(ierr);
  if (isbjacobi) {
PetscInt nlocal;
KSP  *subksp;
PC   subpc;

ierr = KSPSetUp(ksp);CHKERRQ(ierr);
ierr = KSPGetPC(ksp,);CHKERRQ(ierr);

/* Extract the array of KSP contexts for the local blocks */
ierr = PCBJacobiGetSubKSP(pc,,NULL,);CHKERRQ(ierr);
printf("isbjacobi, nlocal %D, set option to subpc...\n",nlocal);
for (i=0; i
> I have implemented this option in the code, as we also need to use
> configuration from file for convenience. When I run the code using options,
> it works fine, however, when I run the code using configuration file, it
> does not work. The code has two set of equations, flow and reactive, with
> prefix been set to "flow_" and "react_". When I run the code using
>
> mpiexec -n 4 ../executable -flow_sub_pc_factor_shift_type nonzero
> -react_sub_pc_factor_shift_type nonzero
>
> it works. However, if I run using
>
> mpiexec -n 4 ../executable
>
> and let the executable file read the options from file, it just does not
> work at "call PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO, ierr)  or
> none, positive_definite ...". Do I miss something here?
>
> Below is the pseudo code I have used for flow equations, similar for
> reactive equations.
>
>   call MatCreateAIJ(Petsc_Comm_World,nndof,nndof,nngbldof, &
> nngbldof,d_nz,PETSC_NULL_INTEGER,o_nz, &
> PETSC_NULL_INTEGER,a_flow,ierr)
>   CHKERRQ(ierr)
>
> call MatSetFromOptions(a_flow,ierr)
> CHKERRQ(ierr)
>
> call KSPCreate(Petsc_Comm_World, ksp_flow, ierr)
> CHKERRQ(ierr)
>
> call KSPAppendOptionsPrefix(ksp_flow,"flow_",ierr)
> CHKERRQ(ierr)
>
> call KSPSetInitialGuessNonzero(ksp_flow,   &
> b_initial_guess_nonzero_flow, ierr)
> CHKERRQ(ierr)
>
> call KSPSetInitialGuessNonzero(ksp_flow,   &
> b_initial_guess_nonzero_flow, ierr)
> CHKERRQ(ierr)
>
> call KSPSetDM(ksp_flow,dmda_flow%da,ierr)
> CHKERRQ(ierr)
> call KSPSetDMActive(ksp_flow,PETSC_FALSE,ierr)
> CHKERRQ(ierr)
>
> *CHECK IF READ OPTION FROM FILE*
> if (read_option_from_file) then
>
>   call KSPSetType(ksp_flow, KSPGMRES, ierr)!or KSPBCGS or
> others...
>   CHKERRQ(ierr)
>
>   call KSPGetPC(ksp_flow, pc_flow, ierr)
>   CHKERRQ(ierr)
>
>   call PCSetType(pc_flow,PCBJACOBI, ierr)   !or PCILU or
> PCJACOBI or PCHYPRE ...
>   CHKERRQ(ierr)
>
>   call PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO, ierr)  or
> none, positive_definite ...
>   CHKERRQ(ierr)
>
> end if
>
> call PCFactorGetMatSolverPackage(pc_flow,solver_pkg_flow,ierr)
> CHKERRQ(ierr)
>
> call compute_jacobian(rank,dmda_flow%da,   &
>   a_flow,a_in,ia_in,ja_in,nngl_in, &
>   row_idx_l2pg,col_idx_l2pg,   &
>   b_non_interlaced)
> call KSPSetFromOptions(ksp_flow,ierr)
> CHKERRQ(ierr)
>
> call KSPSetUp(ksp_flow,ierr)
> CHKERRQ(ierr)
>
> call KSPSetUpOnBlocks(ksp_flow,ierr)
> CHKERRQ(ierr)
>
> call KSPSolve(ksp_flow,b_flow,x_flow,ierr)
> CHKERRQ(ierr)
>
>
> Thanks and Regards,
>
> Danyang
> On 17-05-24 06:32 PM, Hong wrote:
>
> Remove your option '-vecload_block_size 10'.
> Hong
>
> On Wed, May 24, 2017 at 3:06 PM, Danyang Su  wrote:
>
>> Dear Hong,
>>
>> I just tested with different number of processors for the same matrix. It
>> sometimes got "ERROR: Arguments are incompatible" for different number of
>> processors. It works fine using 4, 8, or 24 processors, but failed with
>> "ERROR: Arguments are incompatible" using 16 or 48 processors. The error
>> information is attached. I tested this on my local computer with 6 cores 12
>> threads. Any suggestion on this?
>>
>> Thanks,
>>
>> Danyang
>>
>> On 17-05-24 12:28 PM, Danyang Su wrote:
>>
>> Hi Hong,
>>
>> Awesome. Thanks for testing the case. I will try your options for the
>> code and get back to you later.
>>
>> Regards,
>>
>> Danyang
>>
>> On 17-05-24 12:21 PM, Hong wrote:
>>
>> Danyang :
>> I tested your data.
>> Your matrices encountered zero pivots, e.g.
>> petsc/src/ksp/ksp/examples/tutorials (master)
>> $ mpiexec -n

[petsc-users] PCFactorSetShiftType does not work in code but -pc_factor_set_shift_type works

2017-05-25 Thread Danyang Su 

Dear Hong and Barry,

I have implemented this option in the code, as we also need to use 
configuration from file for convenience. When I run the code using 
options, it works fine, however, when I run the code using configuration 
file, it does not work. The code has two set of equations, flow and 
reactive, with prefix been set to "flow_" and "react_". When I run the 
code using


mpiexec -n 4 ../executable -flow_sub_pc_factor_shift_type nonzero 
-react_sub_pc_factor_shift_type nonzero


it works. However, if I run using

mpiexec -n 4 ../executable

and let the executable file read the options from file, it just does not 
work at "call PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO, ierr)  or 
none, positive_definite ...". Do I miss something here?


Below is the pseudo code I have used for flow equations, similar for 
reactive equations.


  call MatCreateAIJ(Petsc_Comm_World,nndof,nndof,nngbldof, &
nngbldof,d_nz,PETSC_NULL_INTEGER,o_nz, &
PETSC_NULL_INTEGER,a_flow,ierr)
  CHKERRQ(ierr)

call MatSetFromOptions(a_flow,ierr)
CHKERRQ(ierr)

call KSPCreate(Petsc_Comm_World, ksp_flow, ierr)
CHKERRQ(ierr)

call KSPAppendOptionsPrefix(ksp_flow,"flow_",ierr)
CHKERRQ(ierr)

call KSPSetInitialGuessNonzero(ksp_flow,   &
b_initial_guess_nonzero_flow, ierr)
CHKERRQ(ierr)

call KSPSetInitialGuessNonzero(ksp_flow,   &
b_initial_guess_nonzero_flow, ierr)
CHKERRQ(ierr)

call KSPSetDM(ksp_flow,dmda_flow%da,ierr)
CHKERRQ(ierr)
call KSPSetDMActive(ksp_flow,PETSC_FALSE,ierr)
CHKERRQ(ierr)

*CHECK IF READ OPTION FROM FILE*
if (read_option_from_file) then

  call KSPSetType(ksp_flow, KSPGMRES, ierr)!or KSPBCGS or 
others...

  CHKERRQ(ierr)

  call KSPGetPC(ksp_flow, pc_flow, ierr)
  CHKERRQ(ierr)

  call PCSetType(pc_flow,PCBJACOBI, ierr)   !or PCILU or 
PCJACOBI or PCHYPRE ...

  CHKERRQ(ierr)

  call PCFactorSetShiftType(pc_flow,MAT_SHIFT_NONZERO, ierr)  
or none, positive_definite ...

  CHKERRQ(ierr)

end if

call PCFactorGetMatSolverPackage(pc_flow,solver_pkg_flow,ierr)
CHKERRQ(ierr)

call compute_jacobian(rank,dmda_flow%da,   &
a_flow,a_in,ia_in,ja_in,nngl_in, &
row_idx_l2pg,col_idx_l2pg,   &
  b_non_interlaced)
call KSPSetFromOptions(ksp_flow,ierr)
CHKERRQ(ierr)

call KSPSetUp(ksp_flow,ierr)
CHKERRQ(ierr)

call KSPSetUpOnBlocks(ksp_flow,ierr)
CHKERRQ(ierr)

call KSPSolve(ksp_flow,b_flow,x_flow,ierr)
CHKERRQ(ierr)


Thanks and Regards,

Danyang

On 17-05-24 06:32 PM, Hong wrote:

Remove your option '-vecload_block_size 10'.
Hong

On Wed, May 24, 2017 at 3:06 PM, Danyang Su > wrote:


Dear Hong,

I just tested with different number of processors for the same
matrix. It sometimes got "ERROR: Arguments are incompatible" for
different number of processors. It works fine using 4, 8, or 24
processors, but failed with "ERROR: Arguments are incompatible"
using 16 or 48 processors. The error information is attached. I
tested this on my local computer with 6 cores 12 threads. Any
suggestion on this?

Thanks,

Danyang


On 17-05-24 12:28 PM, Danyang Su wrote:


Hi Hong,

Awesome. Thanks for testing the case. I will try your options for
the code and get back to you later.

Regards,

Danyang


On 17-05-24 12:21 PM, Hong wrote:

Danyang :
I tested your data.
Your matrices encountered zero pivots, e.g.
petsc/src/ksp/ksp/examples/tutorials (master)
$ mpiexec -n 24 ./ex10 -f0 a_react_in_2.bin -rhs
b_react_in_2.bin -ksp_monitor -ksp_error_if_not_converged

[15]PETSC ERROR: Zero pivot in LU factorization:
http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot

[15]PETSC ERROR: Zero pivot row 1249 value 2.05808e-14 tolerance
2.22045e-14
...

Adding option '-sub_pc_factor_shift_type nonzero', I got
mpiexec -n 24 ./ex10 -f0 a_react_in_2.bin -rhs b_react_in_2.bin
-ksp_monitor -ksp_error_if_not_converged
-sub_pc_factor_shift_type nonzero -mat_view ascii::ascii_info

Mat Object: 24 MPI processes
  type: mpiaij
  rows=45, cols=45
  total: nonzeros=6991400, allocated nonzeros=6991400
  total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
  0 KSP Residual norm 5.84911755e+01
  1 KSP Residual norm 6.824179430230e-01
  2 KSP Residual norm 3.994483555787e-02
  3 KSP Residual norm