On Mon, Feb 6, 2023 at 11:45 AM Nicolas Barnafi
<[email protected]> wrote:
Thank you Matt,
Again, at the bottom of this message you will find the -info
output. I don't see any output related to the fields,
If the splits were done automatically, you would see an info message
from here:
https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/fieldsplit/fieldsplit.c#L1595
Thus it must be setup here
https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/fieldsplit/fieldsplit.c#L380
There are info statements, but you do not see them, I do not see a way
around using a small example
to understand how you are setting up the system, since it is working
as expected in the PETSc examples.
Thanks,
Matt
Best
------ -info
[0] <sys> PetscDetermineInitialFPTrap(): Floating point trapping
is on by default 13
[0] <sys> PetscDeviceInitializeTypeFromOptions_Private():
PetscDeviceType host available, initializing
[0] <sys> PetscDeviceInitializeTypeFromOptions_Private():
PetscDevice host initialized, default device id 0, view FALSE,
init type lazy
[0] <sys> PetscDeviceInitializeTypeFromOptions_Private():
PetscDeviceType cuda not available
[0] <sys> PetscDeviceInitializeTypeFromOptions_Private():
PetscDeviceType hip not available
[0] <sys> PetscDeviceInitializeTypeFromOptions_Private():
PetscDeviceType sycl not available
[0] <sys> PetscInitialize_Common(): PETSc successfully started:
number of processors = 1
[0] <sys> PetscGetHostName(): Rejecting domainname, likely is NIS
nico-santech.(none)
[0] <sys> PetscInitialize_Common(): Running on machine: nico-santech
[0] <sys> SlepcInitialize(): SLEPc successfully started
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066936960 94770087780768 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770087780768
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066936960 is being unlinked from inner PETSc comm 94770087780768
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770087780768
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770087780768
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066936960 94770087780768 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770087780768
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066936960 is being unlinked from inner PETSc comm 94770087780768
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770087780768
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770087780768
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066936960 94770087780768 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770087780768
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066936960 is being unlinked from inner PETSc comm 94770087780768
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770087780768
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770087780768
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066936960 94770087780768 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770087780768
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066936960 is being unlinked from inner PETSc comm 94770087780768
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770087780768
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770087780768
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066936960 94770087780768 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770087780768
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066936960 is being unlinked from inner PETSc comm 94770087780768
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770087780768
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770087780768
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066936960 94770087780768 max tags = 2147483647
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066936960 94770087780768
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 1219 X 1219;
storage space: 0 unneeded,26443 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 150
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 1219) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 1160 nodes out of 1219
rows. Not using Inode routines
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066936960 94770087780768
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066936960 94770087780768
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066936960 94770087780768
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066936960 94770087780768
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066936960 94770087780768
[0] <sys> PetscGetHostName(): Rejecting domainname, likely is NIS
nico-santech.(none)
[0] <pc> PCSetUp(): Setting up PC for first time
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 615 X 615; storage
space: 0 unneeded,9213 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 117
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 615) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 561 nodes out of 615 rows.
Not using Inode routines
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066934048 94770110251424 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770110251424
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066934048 is being unlinked from inner PETSc comm 94770110251424
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770110251424
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770110251424
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 64 X 64; storage
space: 0 unneeded,0 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
64)/(num_localrows 64) > 0.6. Use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 13 nodes of 64. Limit used:
5. Using Inode routines
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066934048 94770100861088 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770100861088
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066934048 is being unlinked from inner PETSc comm 94770100861088
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770100861088
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770100861088
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 240 X 240; storage
space: 0 unneeded,2140 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 240) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 235 nodes out of 240 rows.
Not using Inode routines
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066934048 94770100861088 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770100861088
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066934048 is being unlinked from inner PETSc comm 94770100861088
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770100861088
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770100861088
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 300 X 300; storage
space: 0 unneeded,2292 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 33
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 300) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 300 nodes out of 300 rows.
Not using Inode routines
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066934048 94770100861088 max tags = 2147483647
[0] <sys> Petsc_OuterComm_Attr_Delete_Fn(): Removing reference to
PETSc communicator embedded in a user MPI_Comm 94770100861088
[0] <sys> Petsc_InnerComm_Attr_Delete_Fn(): User MPI_Comm
94770066934048 is being unlinked from inner PETSc comm 94770100861088
[0] <sys> PetscCommDestroy(): Deleting PETSc MPI_Comm 94770100861088
[0] <sys> Petsc_Counter_Attr_Delete_Fn(): Deleting counter data in
an MPI_Comm 94770100861088
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 615 X 1219;
storage space: 0 unneeded,11202 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 150
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 615) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 561 nodes out of 615 rows.
Not using Inode routines
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 64 X 1219; storage
space: 0 unneeded,288 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 6
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 64) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 64 nodes out of 64 rows.
Not using Inode routines
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 240 X 1219;
storage space: 0 unneeded,8800 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 78
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 240) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 235 nodes out of 240 rows.
Not using Inode routines
[0] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 300 X 1219;
storage space: 0 unneeded,6153 used
[0] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during
MatSetValues() is 0
[0] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 89
[0] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 300) < 0.6. Do not use CompressedRow routines.
[0] <mat> MatSeqAIJCheckInode(): Found 300 nodes out of 300 rows.
Not using Inode routines
[0] <sys> PetscCommDuplicate(): Duplicating a communicator
94770066934048 94770100861088 max tags = 2147483647
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
0 KSP Residual norm 2.541418258630e+01
[0] <ksp> KSPConvergedDefault(): user has provided nonzero initial
guess, computing 2-norm of RHS
[0] <pc> PCSetUp(): Leaving PC with identical preconditioner since
operator is unchanged
[0] <pc> PCSetUp(): Leaving PC with identical preconditioner since
operator is unchanged
[0] <pc> PCSetUp(): Setting up PC for first time
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
[0] <pc> PCSetUp(): Leaving PC with identical preconditioner since
operator is unchanged
[0] <pc> PCSetUp(): Setting up PC for first time
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 94770100861088
[0] <sys> PetscCommDuplicate(): Using internal PETSc communicator
94770066934048 947701008610882
On 03/02/23 21:11, Matthew Knepley wrote:
On Fri, Feb 3, 2023 at 3:03 PM Nicolas Barnafi
<[email protected]> wrote:
> There are a number of common errors:
>
> 1) Your PC has a prefix
>
> 2) You have not called KSPSetFromOptions() here
>
> Can you send the -ksp_view output?
The PC at least has no prefix. I had to set ksp_rtol to 1 to
get through
the solution process, you will find both the petsc_rc and the
ksp_view
at the bottom of this message.
Options are indeed being set from the options file, but there
must be
something missing at a certain level. Thanks for looking into
this.
Okay, the next step is to pass
-info
and send the output. This will tell us how the default splits
were done. If that
is not conclusive, we will have to use the debugger.
Thanks,
Matt
Best
---- petsc_rc file
-ksp_monitor
-ksp_type gmres
-ksp_view
-mat_type aij
-ksp_norm_type unpreconditioned
-ksp_atol 1e-14
-ksp_rtol 1
-pc_type fieldsplit
-pc_fieldsplit_type multiplicative
---- ksp_view
KSP Object: 1 MPI process
type: gmres
restart=500, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
happy breakdown tolerance 1e-30
maximum iterations=10000, nonzero initial guess
tolerances: relative=1., absolute=1e-14, divergence=10000.
right preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI process
type: fieldsplit
FieldSplit with MULTIPLICATIVE composition: total splits = 4
Solver info for each split is in the following KSP objects:
Split number 0 Defined by IS
KSP Object: (fieldsplit_0_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000.
left preconditioning
using DEFAULT norm type for convergence test
PC Object: (fieldsplit_0_) 1 MPI process
type: ilu
PC has not been set up so information may be incomplete
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
matrix solver type: petsc
matrix not yet factored; no additional information
available
linear system matrix = precond matrix:
Mat Object: (fieldsplit_0_) 1 MPI process
type: seqaij
rows=615, cols=615
total: nonzeros=9213, allocated nonzeros=9213
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Split number 1 Defined by IS
KSP Object: (fieldsplit_1_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000.
left preconditioning
using DEFAULT norm type for convergence test
PC Object: (fieldsplit_1_) 1 MPI process
type: ilu
PC has not been set up so information may be incomplete
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
matrix solver type: petsc
matrix not yet factored; no additional information
available
linear system matrix = precond matrix:
Mat Object: (fieldsplit_1_) 1 MPI process
type: seqaij
rows=64, cols=64
total: nonzeros=0, allocated nonzeros=0
total number of mallocs used during MatSetValues calls=0
using I-node routines: found 13 nodes, limit used is 5
Split number 2 Defined by IS
KSP Object: (fieldsplit_2_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000.
left preconditioning
using DEFAULT norm type for convergence test
PC Object: (fieldsplit_2_) 1 MPI process
type: ilu
PC has not been set up so information may be incomplete
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
matrix solver type: petsc
matrix not yet factored; no additional information
available
linear system matrix = precond matrix:
Mat Object: (fieldsplit_2_) 1 MPI process
type: seqaij
rows=240, cols=240
total: nonzeros=2140, allocated nonzeros=2140
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Split number 3 Defined by IS
KSP Object: (fieldsplit_3_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50,
divergence=10000.
left preconditioning
using DEFAULT norm type for convergence test
PC Object: (fieldsplit_3_) 1 MPI process
type: ilu
PC has not been set up so information may be incomplete
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
matrix solver type: petsc
matrix not yet factored; no additional information
available
linear system matrix = precond matrix:
Mat Object: (fieldsplit_3_) 1 MPI process
type: seqaij
rows=300, cols=300
total: nonzeros=2292, allocated nonzeros=2292
total number of mallocs used during MatSetValues calls=0
not using I-node routines
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=1219, cols=1219
total: nonzeros=26443, allocated nonzeros=26443
total number of mallocs used during MatSetValues calls=0
not using I-node routines
solving time: 0.00449609
iterations: 0
estimated error: 25.4142
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to
which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
<http://www.cse.buffalo.edu/~knepley/>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
<http://www.cse.buffalo.edu/~knepley/>
