Thank you Jed and Barry,
First, attached are the logs from the benchmark runs I did without
(log_std.txt) and with MF method (log_mf.txt). It took me some trouble to get
the -log_view to work because I'm using push and pop for the options which
means that PETSc is initialized with no argument so the command line argument
was not taken into account, but I guess this is for a separate discussion.
To answer questions about the current per-conditioners:
* I used the same pre-conditioner options as listed in my previous email
when I added the -snes_mf option; I did try to remove all the PC related
options at one point with the MF method but didn't see a change in runtime so I
put them back in
* this benchmark is for a 1D DMDA using 20 grid points; when running in 2D
or 3D I switch the PC options to: -pc_type fieldsplit -fieldsplit_0_pc_type sor
-fieldsplit_1_pc_type gamg -fieldsplit_1_ksp_type gmres -ksp_type fgmres
-fieldsplit_1_pc_gamg_threshold -1
I haven't tried a Jacobi PC instead of SOR, I will run a set of more realistic
runs (1D and 2D) without MF but with Jacobi and report on it next week. When
you say "iterations" do you mean what is given by -ksp_monitor?
Cheers,
Sophie
________________________________
De : Barry Smith <[email protected]>
Envoyé : vendredi 28 août 2020 12:12
À : Blondel, Sophie <[email protected]>
Cc : [email protected] <[email protected]>;
[email protected]
<[email protected]>
Objet : Re: [petsc-users] Matrix Free Method questions
[External Email]
Sophie,
This is exactly what i would expect. If you run with -ksp_monitor you will
see the -snes_mf run takes many more iterations.
I am puzzled that the argument -pc_type fieldsplit did not stop the run
since this is under normal circumstances not a viable preconditioner with
-snes_mf. Did you also remove the -pc_type fieldsplit argument?
In order to see how one can avoid forming the entire matrix and use
matrix-free to do the matrix-vector but still have an effective preconditioner
let's look at what the current preconditioner options do.
-pc_fieldsplit_detect_coupling
creates two sub-preconditioners, the first for all the variables and the second
for those that are coupled by the matrix to variables in neighboring cells
Since only the smallest cluster sizes have diffusion/advection this second set
contains only the cluster size one variables.
-fieldsplit_0_pc_type sor
Runs SOR on all the variables; you can think of this as running SOR on the
reactions, it is a pretty good preconditioner for the reactions since the
reactions are local, per cell.
-fieldsplit_1_pc_type redundant
This runs the default preconditioner (ILU) on just the variables that diffuse,
i.e. the elliptic part. For smallish problems this is fine, for larger problems
and 2d and 3d presumably you have also -redundant_pc_type gamg to use algebraic
multigrid for the diffusion. This part of the matrix will always need to be
formed and used in the preconditioner. It is very important since the
diffusion is what brings in most of the ill-conditioning for larger problems
into the linear system. Note that it only needs the matrix entries for the
cluster size of 1 so it is very small compared to the entire sparse matrix.
----
The first preconditioner SOR requires ALL the matrix entries which are almost
all (except for the diffusion terms) the coupling between different size
clusters within a cell. Especially each cell has its own sparse matrix of the
size of total number of clusters, it is sparse but not super sparse.
So the to significantly lower memory usage we need to remove the SOR and the
storing of all the matrix entries but still have an efficient preconditioner
for the "reaction" terms.
The simplest thing would be to use Jacobi instead of SOR for the first
subpreconditioner since it only requires the diagonal entries in the matrix.
But Jacobi is a worse preconditioner than SOR (since it totally ignores the
matrix coupling) and sometimes can be much worse.
Before anyone writes additional code we need to know if doing something along
these lines does not ruin the convergence that.
Have you used the same options as before but with -fieldsplit_0_pc_type
jacobi ? (Not using any matrix free). We need to get an idea of how many more
linear iterations it requires (not time, comparing time won't be helpful for
this exercise.) We also need this information for realistic size problems in 2
or 3 dimensions that you really want to run; for small problems this approach
will work ok and give misleading information about what happens for large
problems.
I suspect the iteration counts will shot up. Can you run some cases and see
how the iteration counts change?
Based on that we can decide if we still retain "good convergence" by changing
the SOR to Jacobi and then change the code to make this change efficient
(basically by skipping the explicit computation of the reaction Jacobian terms
and using matrix-free on the outside of the PCFIELDSPLIT.)
Barry
On Aug 28, 2020, at 9:49 AM, Blondel, Sophie via petsc-users
<[email protected]<mailto:[email protected]>> wrote:
Hi everyone,
I have been using PETSc for a few years with a fully implicit TS ARKIMEX method
and am now exploring the matrix free method option. Here is the list of PETSc
options I typically use: -ts_dt 1.0e-12 -ts_adapt_time_step_increase_delay 5
-snes_force_iteration -ts_max_time 1000.0 -ts_adapt_dt_max 2.0e-3
-ts_adapt_wnormtype INFINITY -ts_exact_final_time stepover
-fieldsplit_0_pc_type sor -ts_max_snes_failures -1
-pc_fieldsplit_detect_coupling -ts_monitor -pc_type fieldsplit
-fieldsplit_1_pc_type redundant -ts_max_steps 100
I started to compare the performance of the code without changing anything of
the executable and simply adding "-snes_mf", I see a reduction of memory usage
as expected and a benchmark that would usually take ~5min to run now takes
~50min. Reading the documentation I saw that there are a few option to play
with the matrix free method like -snes_mf_err, -snes_mf_umin, or switching to
-snes_mf_type wp. I used and modified the values of each of these options
separately but never saw a sizable change in runtime, is it expected?
And are there other ways to make the matrix free method faster? I saw in the
documentation that you can define your own per-conditioner for instance. Let me
know if you need additional information about the PETSc setup in the
application I use.
Best,
Sophie
Starting Xolotl Plasma-Surface Interactions Simulator
Fri Aug 28 15:52:54 2020
Loaded network of size 12302.
-1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5
0 TS dt 1e-12 time 0.
Time: 0
Helium content = 0
Deuterium content = 0
Tritium content = 0
Vacancy content = 0
Interstitial content = 0
Fluence = 0
1 TS dt 1.87486e-12 time 1e-12
Time: 1e-12
Helium content = 3.998268973002881e-08
Deuterium content = 1.799760296381141e-07
Tritium content = 1.799814376083755e-07
Vacancy content = 5.54221024241515e-86
Interstitial content = 1.076083851388208e-09
Fluence = 4e-07
2 TS dt 9.33506e-12 time 2.87486e-12
Time: 2.874863371692955e-12
Helium content = 1.148621169190103e-07
Deuterium content = 5.172733926277329e-07
Tritium content = 5.173129029911139e-07
Vacancy content = 4.658476854195309e-78
Interstitial content = 2.786274004228506e-09
Fluence = 1.149945348677182e-06
3 TS dt 8.31605e-11 time 1.22099e-11
Time: 1.220992231043647e-11
Helium content = 4.860866926971022e-07
Deuterium content = 2.194127143629604e-06
Tritium content = 2.194804298858191e-06
Vacancy content = 2.239663947526481e-63
Interstitial content = 8.661082643970004e-09
Fluence = 4.883968924174588e-06
4 TS dt 8.31605e-10 time 9.53705e-11
Time: 9.537046352538632e-11
Helium content = 3.678776568415282e-06
Deuterium content = 1.69451969981092e-05
Tritium content = 1.698554014545955e-05
Vacancy content = -3.419344797889238e-49
Interstitial content = 3.117542829612831e-08
Fluence = 3.814818541015453e-05
5 TS dt 8.31605e-09 time 9.26976e-10
Time: 9.269758756748847e-10
Helium content = 2.707166607834261e-05
Deuterium content = 0.0001480748666772199
Tritium content = 0.0001512025198850674
Vacancy content = 1.874072027347334e-35
Interstitial content = 4.784869796591994e-08
Fluence = 0.0003707903502699539
6 TS dt 3.67703e-08 time 9.24303e-09
Time: 9.243029997169868e-09
Helium content = 5.981727029232682e-05
Deuterium content = 0.0006649850032406357
Tritium content = 0.0007604914245556196
Vacancy content = 1.242134501693808e-25
Interstitial content = 4.587227649229742e-08
Fluence = 0.003697211998867947
7 TS dt 1.70705e-07 time 4.60134e-08
Time: 4.601337569781458e-08
Helium content = 5.608141577731466e-05
Deuterium content = 0.0007229862566200205
Tritium content = 0.0008927171367993695
Vacancy content = 8.821578752783583e-23
Interstitial content = 4.592631030278966e-08
Fluence = 0.01840535027912583
8 TS dt 1.27967e-06 time 2.16718e-07
Time: 2.167182119912474e-07
Helium content = 5.636796729822405e-05
Deuterium content = 0.0007159591775982223
Tritium content = 0.0008765901992898002
Vacancy content = 6.163530024811232e-21
Interstitial content = 4.592588203947602e-08
Fluence = 0.08668728479649895
9 TS dt 1.27967e-05 time 1.49639e-06
Time: 1.496391882028352e-06
Helium content = 5.636167377621148e-05
Deuterium content = 0.0007163342298980516
Tritium content = 0.0008775936745554613
Vacancy content = 6.76351397219519e-20
Interstitial content = 4.592586036660697e-08
Fluence = 0.5985567528113409
10 TS dt 0.000127967 time 1.42931e-05
Time: 1.42931285823994e-05
Helium content = 5.636170958070677e-05
Deuterium content = 0.0007163283813925675
Tritium content = 0.0008775743701864409
Vacancy content = 1.378329880888954e-19
Interstitial content = 4.592586130914882e-08
Fluence = 5.717251432959761
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r
-fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary:
----------------------------------------------
Unknown Name on a 20200828 named sophie-Precision-5530 with 4 processors, by
sophie Fri Aug 28 16:47:41 2020
Using Petsc Development GIT revision: v3.13.4-851-gde18fec8da GIT Date:
2020-08-28 16:47:50 +0000
Max Max/Min Avg Total
Time (sec): 3.091e+03 1.000 3.091e+03
Objects: 9.100e+01 1.000 9.100e+01
Flop: 1.625e+11 1.000 1.625e+11 6.501e+11
Flop/sec: 5.258e+07 1.000 5.258e+07 2.103e+08
MPI Messages: 7.187e+04 2.000 5.390e+04 2.156e+05
MPI Message Lengths: 7.073e+09 2.000 9.842e+04 2.122e+10
MPI Reductions: 7.206e+04 1.000
Flop counting convention: 1 flop = 1 real number operation of type
(multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N
flop
and VecAXPY() for complex vectors of length N -->
8N flop
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- --
Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total
Avg %Total Count %Total
0: Main Stage: 3.0909e+03 100.0% 6.5013e+11 100.0% 2.156e+05 100.0%
9.842e+04 100.0% 7.205e+04 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting
output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and
PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in
this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all
processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop
--- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct
%T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 1 1.0 1.8411e-05 1.7 0.00e+00 0.0 6.0e+00 4.0e+00
1.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetGraph 1 1.0 4.2300e-07 1.8 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 1 1.0 3.7780e-05 1.2 0.00e+00 0.0 1.2e+01 4.0e+00
1.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFBcastOpBegin 35933 1.0 1.0361e+01 1.4 0.00e+00 0.0 2.2e+05 9.8e+04
0.0e+00 0 0100100 0 0 0100100 0 0
SFBcastOpEnd 35933 1.0 6.3830e+00 3.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFPack 35933 1.0 1.5615e+00285.6 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFUnpack 35933 1.0 1.1998e+00183.6 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecDot 47 1.0 1.9313e+00273.0 5.78e+06 1.0 0.0e+00 0.0e+00
4.7e+01 0 0 0 0 0 0 0 0 0 0 12
VecMDot 34677 1.0 1.5005e+0321.7 6.57e+10 1.0 0.0e+00 0.0e+00
3.5e+04 17 40 0 0 48 17 40 0 0 48 175
VecNorm 37045 1.0 6.0632e+01 8.0 4.56e+09 1.0 0.0e+00 0.0e+00
3.7e+04 1 3 0 0 51 1 3 0 0 51 301
VecScale 107594 1.0 4.0805e+00 1.3 4.41e+09 1.0 0.0e+00 0.0e+00
0.0e+00 0 3 0 0 0 0 3 0 0 0 4324
VecCopy 37244 1.0 1.5880e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 37200 1.0 1.6951e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 38137 1.0 3.9457e+00 1.3 4.69e+09 1.0 0.0e+00 0.0e+00
0.0e+00 0 3 0 0 0 0 3 0 0 0 4757
VecAYPX 35933 1.0 3.9184e+00 1.2 2.21e+09 1.0 0.0e+00 0.0e+00
0.0e+00 0 1 0 0 0 0 1 0 0 0 2256
VecAXPBYCZ 35953 1.0 1.4487e+01 1.1 6.63e+09 1.0 0.0e+00 0.0e+00
0.0e+00 0 4 0 0 0 0 4 0 0 0 1832
VecWAXPY 35893 1.0 1.1341e+01 1.2 4.41e+09 1.0 0.0e+00 0.0e+00
0.0e+00 0 3 0 0 0 0 3 0 0 0 1556
VecMAXPY 35926 1.0 7.5672e+01 1.1 6.99e+10 1.0 0.0e+00 0.0e+00
0.0e+00 2 43 0 0 0 2 43 0 0 0 3697
VecScatterBegin 35933 1.0 1.0453e+01 1.4 0.00e+00 0.0 2.2e+05 9.8e+04
0.0e+00 0 0100100 0 0 0100100 0 0
VecScatterEnd 35933 1.0 6.4492e+00 3.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecReduceArith 94 1.0 1.2670e-02 1.1 1.16e+07 1.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 3651
VecReduceComm 47 1.0 5.5436e-0229.3 0.00e+00 0.0 0.0e+00 0.0e+00
4.7e+01 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 35846 1.0 5.7913e+01 6.1 6.61e+09 1.0 0.0e+00 0.0e+00
3.6e+04 1 4 0 0 50 1 4 0 0 50 457
TSStep 10 1.0 3.0906e+03 1.0 1.63e+11 1.0 2.2e+05 9.8e+04
7.2e+04100100100100100 100100100100100 210
TSFunctionEval 35933 1.0 2.9025e+03 2.0 2.21e+09 1.0 2.2e+05 9.8e+04
0.0e+00 79 1100100 0 79 1100100 0 3
SNESSolve 30 1.0 3.0901e+03 1.0 1.63e+11 1.0 2.2e+05 9.8e+04
7.2e+04100100100100100 100100100100100 210
SNESSetUp 1 1.0 3.0985e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
4.0e+00 0 0 0 0 0 0 0 0 0 0 0
SNESFunctionEval 35923 1.0 2.9163e+03 2.0 8.84e+09 1.0 2.2e+05 9.8e+04
0.0e+00 79 5100100 0 79 5100100 0 12
SNESJacobianEval 47 1.0 7.7553e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
4.0e+00 0 0 0 0 0 0 0 0 0 0 0
SNESLineSearch 47 1.0 7.8849e+00 1.0 6.93e+07 1.0 5.6e+02 9.8e+04
1.9e+02 0 0 0 0 0 0 0 0 0 0 35
MatMult MF 35846 1.0 2.9273e+03 2.0 2.00e+10 1.0 2.2e+05 9.8e+04
1.1e+03 80 12100100 2 80 12100100 2 27
MatMult 35846 1.0 2.9274e+03 2.0 2.00e+10 1.0 2.2e+05 9.8e+04
1.1e+03 80 12100100 2 80 12100100 2 27
MatAssemblyBegin 47 1.0 7.9460e-06 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 47 1.0 7.5770e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
4.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 47 1.0 1.0295e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
1.0e+01 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 47 1.0 3.0794e+03 1.0 1.62e+11 1.0 2.1e+05 9.8e+04
7.2e+04100100100100 99 100100100100 99 211
KSPGMRESOrthog 34677 1.0 1.5676e+0311.2 1.31e+11 1.0 0.0e+00 0.0e+00
3.5e+04 19 81 0 0 48 19 81 0 0 48 335
PCSetUp 47 1.0 1.0055e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 35846 1.0 1.4963e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Distributed Mesh 2 2 10636 0.
Index Set 4 4 3628 0.
IS L to G Mapping 1 1 708 0.
Star Forest Graph 5 5 5392 0.
Discrete System 2 2 1920 0.
Vec Scatter 1 1 816 0.
Vector 62 62 28844704 0.
TSAdapt 1 1 1272 0.
TS 1 1 2480 0.
DMTS 1 1 768 0.
SNES 1 1 1412 0.
DMSNES 3 3 2040 0.
SNESLineSearch 1 1 1008 0.
Krylov Solver 1 1 18656 0.
DMKSP interface 1 1 664 0.
MatMFFD 1 1 752 0.
Matrix 1 1 3080 0.
Preconditioner 1 1 1120 0.
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 4.35e-08
Average time for MPI_Barrier(): 1.3176e-06
Average time for zero size MPI_Send(): 2.5725e-06
#No PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: PETSC_DIR=/home/sophie/Code/petsc PETSC_ARCH=20200828
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no
--with-shared-libraries
-----------------------------------------
Libraries compiled on 2020-08-28 19:04:45 on sophie-Precision-5530
Machine characteristics: Linux-5.4.0-42-generic-x86_64-with-glibc2.29
Using PETSc directory: /home/sophie/Code/petsc
Using PETSc arch: 20200828
-----------------------------------------
Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O
Using Fortran compiler: mpif77 -fPIC -Wall -ffree-line-length-0
-Wno-unused-dummy-argument -g -O
-----------------------------------------
Using include paths: -I/home/sophie/Code/petsc/include
-I/home/sophie/Code/petsc/20200828/include
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif77
Using libraries: -Wl,-rpath,/home/sophie/Code/petsc/20200828/lib
-L/home/sophie/Code/petsc/20200828/lib -lpetsc
-Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
-L/usr/lib/x86_64-linux-gnu/openmpi/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9
-Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu
-Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -llapack -lblas -lX11
-lm -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
-lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lquadmath -lstdc++
-ldl
-----------------------------------------
Timers:
name: Flux
process_count: 4
min: 1314.986373829005
max: 2710.160338506994
average: 2269.130352842237
stdev: 564.8118469801432
name: Partial Derivatives
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: RHSFunctionTimer
process_count: 4
min: 1421.154035243007
max: 2898.291923265016
average: 2434.779431158001
stdev: 603.2482947852172
name: RHSJacobianTimer
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: SolveTimer
process_count: 4
min: 3090.769498707
max: 3090.776894364
average: 3090.77344393525
stdev: 0.002733325756942514
name: monitor1D:checkNeg
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:event
process_count: 4
min: 0.000265262
max: 0.000280984
average: 0.00027393375
stdev: 5.619976618056686e-06
name: monitor1D:heRet
process_count: 4
min: 0.163278166
max: 0.164602271
average: 0.163611881
stdev: 0.0005718095422132117
name: monitor1D:init
process_count: 4
min: 0.000323421
max: 0.00048071
average: 0.000363009
stdev: 6.795505535646326e-05
name: monitor1D:postEvent
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:scatter
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:series
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:startStop
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:tridyn
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:xeRet
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
Counters:
name: Flux
process_count: 4
min: 71866
max: 179665
average: 143732
stdev: 44008.75746371397
name: Partial Derivatives
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
HardwareCounters:
Starting Xolotl Plasma-Surface Interactions Simulator
Fri Aug 28 15:40:46 2020
Loaded network of size 12302.
-1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5
0 TS dt 1e-12 time 0.
Time: 0
Helium content = 0
Deuterium content = 0
Tritium content = 0
Vacancy content = 0
Interstitial content = 0
Fluence = 0
1 TS dt 1.87486e-12 time 1e-12
Time: 1e-12
Helium content = 3.998453355656088e-08
Deuterium content = 1.799754050011278e-07
Tritium content = 1.799799241815265e-07
Vacancy content = 5.183275479630833e-86
Interstitial content = 1.085277712412299e-09
Fluence = 4e-07
2 TS dt 9.33506e-12 time 2.87486e-12
Time: 2.874863371648933e-12
Helium content = 1.148666308125159e-07
Deuterium content = 5.172716233423786e-07
Tritium content = 5.17309064862937e-07
Vacancy content = 7.529087673238805e-79
Interstitial content = 2.794423493510334e-09
Fluence = 1.149945348659573e-06
3 TS dt 8.31605e-11 time 1.22099e-11
Time: 1.220992231427955e-11
Helium content = 4.860946003548312e-07
Deuterium content = 2.194114458139551e-06
Tritium content = 2.194788782571064e-06
Vacancy content = 5.88846442845581e-68
Interstitial content = 8.661671344256942e-09
Fluence = 4.883968925711822e-06
4 TS dt 8.31605e-10 time 9.53705e-11
Time: 9.537046345911179e-11
Helium content = 3.678429468697813e-06
Deuterium content = 1.694343578854218e-05
Tritium content = 1.698409072971427e-05
Vacancy content = 8.199076747767878e-53
Interstitial content = 3.117599405335611e-08
Fluence = 3.814818538364472e-05
5 TS dt 8.31605e-09 time 9.26976e-10
Time: 9.269758749074341e-10
Helium content = 2.704843186609527e-05
Deuterium content = 0.0001480489510446246
Tritium content = 0.0001511966396747453
Vacancy content = 3.17763401489997e-37
Interstitial content = 4.785787897242143e-08
Fluence = 0.0003707903499629736
6 TS dt 3.67778e-08 time 9.24303e-09
Time: 9.243029989390658e-09
Helium content = 5.981804087746346e-05
Deuterium content = 0.0006649926823889447
Tritium content = 0.0007604993902267464
Vacancy content = -2.440075099058014e-26
Interstitial content = 4.586783835841149e-08
Fluence = 0.003697211995756263
7 TS dt 1.70247e-07 time 4.60208e-08
Time: 4.602082733157514e-08
Helium content = 5.608817965934341e-05
Deuterium content = 0.0007228336282560409
Tritium content = 0.000892435974583055
Vacancy content = 2.800714813815969e-23
Interstitial content = 4.592624813261854e-08
Fluence = 0.01840833093263006
8 TS dt 1.26255e-06 time 2.16268e-07
Time: 2.162675511088579e-07
Helium content = 5.636705290898551e-05
Deuterium content = 0.0007160168831708334
Tritium content = 0.0008767183714606261
Vacancy content = 7.196478669185629e-20
Interstitial content = 4.592583806232926e-08
Fluence = 0.08650702044354315
9 TS dt 1.26255e-05 time 1.47882e-06
Time: 1.47881741173335e-06
Helium content = 5.636169192941323e-05
Deuterium content = 0.0007163310946872647
Tritium content = 0.0008775850103291917
Vacancy content = 5.852389689608021e-17
Interstitial content = 4.59258386244094e-08
Fluence = 0.5915269646933402
10 TS dt 0.000126255 time 1.41043e-05
Time: 1.410431601797827e-05
Helium content = 5.63616933608036e-05
Deuterium content = 0.000716330972295392
Tritium content = 0.0008775829798637923
Vacancy content = 1.928147214843122e-15
Interstitial content = 4.592583862447052e-08
Fluence = 5.641726407191309
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r
-fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary:
----------------------------------------------
Unknown Name on a 20200828 named sophie-Precision-5530 with 4 processors, by
sophie Fri Aug 28 15:46:04 2020
Using Petsc Development GIT revision: v3.13.4-851-gde18fec8da GIT Date:
2020-08-28 16:47:50 +0000
Max Max/Min Avg Total
Time (sec): 1.188e+02 1.000 1.188e+02
Objects: 1.410e+02 1.000 1.410e+02
Flop: 1.048e+09 1.049 1.031e+09 4.124e+09
Flop/sec: 8.821e+06 1.049 8.683e+06 3.473e+07
MPI Messages: 8.510e+02 1.522 7.050e+02 2.820e+03
MPI Message Lengths: 2.433e+07 1.947 2.611e+04 7.364e+07
MPI Reductions: 8.000e+02 1.000
Flop counting convention: 1 flop = 1 real number operation of type
(multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N
flop
and VecAXPY() for complex vectors of length N -->
8N flop
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- --
Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total
Avg %Total Count %Total
0: Main Stage: 1.1876e+02 100.0% 4.1244e+09 100.0% 2.820e+03 100.0%
2.611e+04 100.0% 7.930e+02 99.1%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting
output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and
PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in
this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all
processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop
--- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct
%T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 127 1.0 5.4117e+01364.6 0.00e+00 0.0 3.6e+01 4.0e+00
1.3e+02 15 0 1 0 16 15 0 1 0 16 0
BuildTwoSidedF 119 1.0 5.4117e+01364.3 0.00e+00 0.0 0.0e+00 0.0e+00
1.2e+02 15 0 0 0 15 15 0 0 0 15 0
DMCreateMat 1 1.0 2.3224e-01 1.0 0.00e+00 0.0 1.2e+01 3.6e+01
8.0e+00 0 0 0 0 1 0 0 0 0 1 0
SFSetGraph 8 1.0 6.0344e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 8 1.0 9.8374e-04 1.1 0.00e+00 0.0 7.2e+01 7.6e+01
8.0e+00 0 0 3 0 1 0 0 3 0 1 0
SFBcastOpBegin 604 1.0 5.5006e-02 1.1 0.00e+00 0.0 2.7e+03 2.7e+04
0.0e+00 0 0 94 97 0 0 0 94 97 0 0
SFBcastOpEnd 604 1.0 6.1392e-0117.8 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFReduceBegin 160 1.0 1.4311e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFReduceEnd 160 1.0 1.6915e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFPack 764 1.0 9.4400e-0346.5 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFUnpack 764 1.0 4.2959e-0337.5 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecDot 40 1.0 1.1605e-02 1.2 4.92e+06 1.0 0.0e+00 0.0e+00
4.0e+01 0 0 0 0 5 0 0 0 0 5 1696
VecMDot 40 1.0 1.0304e-02 1.0 4.92e+06 1.0 0.0e+00 0.0e+00
4.0e+01 0 0 0 0 5 0 0 0 0 5 1910
VecNorm 186 1.0 2.9879e+0046.0 2.29e+07 1.0 0.0e+00 0.0e+00
1.9e+02 1 2 0 0 23 1 2 0 0 23 31
VecScale 206 1.0 8.0727e-03 1.7 5.54e+06 1.0 0.0e+00 0.0e+00
0.0e+00 0 1 0 0 0 0 1 0 0 0 2747
VecCopy 202 1.0 3.1186e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 542 1.0 1.3831e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 40 1.0 7.3390e-03 1.8 4.92e+06 1.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 2682
VecAYPX 80 1.0 8.1422e-03 1.1 4.92e+06 1.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 2418
VecAXPBYCZ 100 1.0 3.7710e-02 1.3 1.85e+07 1.0 0.0e+00 0.0e+00
0.0e+00 0 2 0 0 0 0 2 0 0 0 1958
VecWAXPY 40 1.0 1.1588e-02 1.0 2.46e+06 1.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 849
VecMAXPY 160 1.0 3.9864e-02 1.1 3.44e+07 1.0 0.0e+00 0.0e+00
0.0e+00 0 3 0 0 0 0 3 0 0 0 3457
VecScatterBegin 764 1.0 7.0007e-02 1.1 0.00e+00 0.0 2.7e+03 2.7e+04
0.0e+00 0 0 94 97 0 0 0 94 97 0 0
VecScatterEnd 764 1.0 6.1507e-0117.4 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecReduceArith 80 1.0 1.1059e-02 1.1 9.84e+06 1.0 0.0e+00 0.0e+00
0.0e+00 0 1 0 0 0 0 1 0 0 0 3560
VecReduceComm 40 1.0 7.3553e-0323.5 0.00e+00 0.0 0.0e+00 0.0e+00
4.0e+01 0 0 0 0 5 0 0 0 0 5 0
VecNormalize 80 1.0 1.8563e-02 1.1 1.48e+07 1.0 0.0e+00 0.0e+00
8.0e+01 0 1 0 0 10 0 1 0 0 10 3181
TSStep 10 1.0 1.1845e+02 1.0 1.05e+09 1.0 2.8e+03 2.6e+04
6.5e+02100100100100 81 100100100100 82 35
TSFunctionEval 80 1.0 6.4416e+00 1.8 4.92e+06 1.0 4.8e+02 9.8e+04
0.0e+00 5 0 17 64 0 5 0 17 64 0 3
TSJacobianEval 40 1.0 1.0902e+02 1.0 1.67e+08 1.4 2.4e+02 9.8e+04
4.0e+01 92 15 9 32 5 92 15 9 32 5 6
SNESSolve 30 1.0 1.1772e+02 1.0 1.02e+09 1.1 2.7e+03 2.5e+04
6.2e+02 99 97 97 92 78 99 97 97 92 79 34
SNESSetUp 1 1.0 2.3252e-01 1.0 0.00e+00 0.0 1.2e+01 3.6e+01
1.0e+01 0 0 0 0 1 0 0 0 0 1 0
SNESFunctionEval 70 1.0 5.6662e+00 1.8 1.72e+07 1.0 4.2e+02 9.8e+04
0.0e+00 4 2 15 56 0 4 2 15 56 0 12
SNESJacobianEval 40 1.0 1.0902e+02 1.0 1.67e+08 1.4 2.4e+02 9.8e+04
4.0e+01 92 15 9 32 5 92 15 9 32 5 6
SNESLineSearch 40 1.0 3.5682e+00 1.0 1.99e+08 1.0 4.8e+02 4.9e+04
1.6e+02 3 19 17 32 20 3 19 17 32 20 223
MatMult 160 1.0 8.6618e-01 1.2 4.22e+08 1.0 9.6e+02 1.4e+02
0.0e+00 1 41 34 0 0 1 41 34 0 0 1949
MatSolve 80 1.0 2.4229e-03 1.0 2.17e+06 1.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 3581
MatSOR 80 1.0 1.1120e+00 1.1 3.32e+08 1.0 0.0e+00 0.0e+00
0.0e+00 1 32 0 0 0 1 32 0 0 0 1193
MatLUFactorSym 1 1.0 2.1282e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatLUFactorNum 40 1.0 1.1040e-02 1.1 1.14e+07 1.0 0.0e+00 0.0e+00
0.0e+00 0 1 0 0 0 0 1 0 0 0 4124
MatCopy 39 1.0 1.2534e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatConvert 1 1.0 1.9312e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 40 1.0 3.2254e-01 1.0 8.23e+07 1.0 0.0e+00 0.0e+00
0.0e+00 0 8 0 0 0 0 8 0 0 0 1021
MatAssemblyBegin 324 1.0 5.4119e+01360.6 0.00e+00 0.0 0.0e+00 0.0e+00
1.2e+02 15 0 0 0 15 15 0 0 0 15 0
MatAssemblyEnd 324 1.0 5.3798e-03 1.0 0.00e+00 0.0 3.6e+01 3.6e+01
1.5e+01 0 0 1 0 2 0 0 1 0 2 0
MatGetRowIJ 1 1.0 1.4130e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMats 40 1.0 1.0297e-02 5.3 0.00e+00 0.0 0.0e+00 0.0e+00
4.3e+01 0 0 0 0 5 0 0 0 0 5 0
MatCreateSubMat 80 1.0 5.8790e-01 1.0 0.00e+00 0.0 4.8e+01 8.6e+01
1.0e+02 0 0 2 0 13 0 0 2 0 13 0
MatGetOrdering 1 1.0 1.1928e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatZeroEntries 40 1.0 1.5298e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatRedundantMat 40 1.0 1.0710e-02 4.5 0.00e+00 0.0 0.0e+00 0.0e+00
4.3e+01 0 0 0 0 5 0 0 0 0 5 0
MatMPIConcateSeq 40 1.0 1.7883e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 160 1.0 6.2801e-04 2.5 0.00e+00 0.0 0.0e+00 0.0e+00
1.0e+01 0 0 0 0 1 0 0 0 0 1 0
KSPSolve 40 1.0 2.2788e+00 1.0 6.39e+08 1.0 1.8e+03 1.5e+03
4.0e+02 2 62 65 4 49 2 62 65 4 50 1123
KSPGMRESOrthog 40 1.0 1.7794e-02 1.1 9.84e+06 1.0 0.0e+00 0.0e+00
4.0e+01 0 1 0 0 5 0 1 0 0 5 2212
PCSetUp 120 1.0 6.2884e-01 1.0 1.14e+07 1.0 1.6e+02 1.3e+04
2.6e+02 1 1 6 3 33 1 1 6 3 33 72
PCApply 80 1.0 1.3403e+00 1.0 4.43e+08 1.0 1.5e+03 4.8e+02
6.3e+01 1 43 54 1 8 1 43 54 1 8 1322
KSPSolve_FS_0 80 1.0 1.1169e+00 1.1 3.32e+08 1.0 0.0e+00 0.0e+00
0.0e+00 1 32 0 0 0 1 32 0 0 0 1188
KSPSolve_FS_1 80 1.0 3.2080e-02 1.5 1.36e+07 1.0 1.0e+03 6.4e+02
6.3e+01 0 1 37 1 8 0 1 37 1 8 1690
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Distributed Mesh 2 2 10636 0.
Index Set 31 31 282924 0.
IS L to G Mapping 1 1 708 0.
Star Forest Graph 12 12 13344 0.
Discrete System 2 2 1920 0.
Vec Scatter 8 8 6528 0.
Vector 53 53 17028544 0.
TSAdapt 1 1 1272 0.
TS 1 1 2480 0.
DMTS 1 1 768 0.
SNES 1 1 1412 0.
DMSNES 3 3 2040 0.
SNESLineSearch 1 1 1008 0.
Krylov Solver 4 4 22904 0.
DMKSP interface 1 1 664 0.
Matrix 14 14 36300856 0.
Preconditioner 4 4 3960 0.
Viewer 1 0 0 0.
========================================================================================================================
Average time to get PetscTime(): 3.99e-08
Average time for MPI_Barrier(): 1.0242e-06
Average time for zero size MPI_Send(): 1.6415e-06
#No PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: PETSC_DIR=/home/sophie/Code/petsc PETSC_ARCH=20200828
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no
--with-shared-libraries
-----------------------------------------
Libraries compiled on 2020-08-28 19:04:45 on sophie-Precision-5530
Machine characteristics: Linux-5.4.0-42-generic-x86_64-with-glibc2.29
Using PETSc directory: /home/sophie/Code/petsc
Using PETSc arch: 20200828
-----------------------------------------
Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O
Using Fortran compiler: mpif77 -fPIC -Wall -ffree-line-length-0
-Wno-unused-dummy-argument -g -O
-----------------------------------------
Using include paths: -I/home/sophie/Code/petsc/include
-I/home/sophie/Code/petsc/20200828/include
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif77
Using libraries: -Wl,-rpath,/home/sophie/Code/petsc/20200828/lib
-L/home/sophie/Code/petsc/20200828/lib -lpetsc
-Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
-L/usr/lib/x86_64-linux-gnu/openmpi/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9
-Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu
-Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -llapack -lblas -lX11
-lm -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
-lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lquadmath -lstdc++
-ldl
-----------------------------------------
Timers:
name: Flux
process_count: 4
min: 2.753409014999998
max: 5.608755317999998
average: 4.643186591499999
stdev: 1.16397305514639
name: Partial Derivatives
process_count: 4
min: 52.42340589499997
max: 103.594970247
average: 87.42253989125001
stdev: 20.93637460508896
name: RHSFunctionTimer
process_count: 4
min: 3.481893305999999
max: 6.432799519
average: 5.486161613249999
stdev: 1.20606437014616
name: RHSJacobianTimer
process_count: 4
min: 108.440422635
max: 108.440765729
average: 108.440608359
stdev: 0.0001449835906141654
name: SolveTimer
process_count: 4
min: 118.630885263
max: 118.640783686
average: 118.63561294775
stdev: 0.003589092694791343
name: monitor1D:checkNeg
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:event
process_count: 4
min: 0.000238582
max: 0.0002488880000000001
average: 0.0002434565
stdev: 3.909336868317932e-06
name: monitor1D:heRet
process_count: 4
min: 0.171773307
max: 0.173038982
average: 0.172093411
stdev: 0.0005459390859497897
name: monitor1D:init
process_count: 4
min: 0.000391785
max: 0.00062706
average: 0.00045116375
stdev: 0.0001015550578587178
name: monitor1D:postEvent
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:scatter
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:series
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:startStop
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:tridyn
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
name: monitor1D:xeRet
process_count: 4
min: 0
max: 0
average: 0
stdev: 0
Counters:
name: Flux
process_count: 4
min: 160
max: 400
average: 320
stdev: 97.97958971132712
name: Partial Derivatives
process_count: 4
min: 80
max: 200
average: 160
stdev: 48.98979485566356
HardwareCounters:
