Hello,
The log I attached was a very small test case (1 iteration) : I just wanted to
get some information about PETSc log printings (time, flops...). I profiled the
code over "big" test case with scalasca : I know I spend 95% of the time
calling PETSc (on realistic test cases).
Barry,
I have attached 2 logs over 1 and 2 procs running a intermediate test case (not
small but not big).
>From PETSc_1proc.log, I get :
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- --
Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total
Avg %Total counts %Total
0: Main Stage: 2.6379e+02 100.0% 9.8232e+10 100.0% 0.000e+00 0.0%
0.000e+00 0.0% 2.943e+03 100.0%
My understanding is that the relevant figure is 9.8232e+10 (flop) / 2.6379e+02
(sec) = 3.72e12 flops = 3723 gflops for 1 proc.
>From PETSc_2procs.log, I get :
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- --
Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total
Avg %Total counts %Total
0: Main Stage: 1.6733e+02 100.0% 1.0348e+11 100.0% 3.427e+04 100.0%
2.951e+04 100.0% 1.355e+04 100.0%
My understanding is that the relevant figure is 1.0348e+11 (flop) / 1.6733e+02
(sec) = 6.18e12 flops = 6184 gflops for 2 procs.
Am I correct ?
My understanding is that if the code scales "well", when I double the number of
MPI processes, I should double the flops which is not the case (I get 6184
gflops instead of 2*3723) : right ? wrong ?
>From this, how can I know if I am at the bandwidth peak ?
If I compare 6184 gflops to the (computer characteristic computed by Matt =
12.9 GB/s * 1 flops/12 bytes =) 1.1 GF/s, I get a huge difference that I am not
sure to understand... I am not sure I can compare these numbers : I guess no ?!
Did I miss something ?
More over I realize that I get :
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops
--- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct
%T %f %M %L %R %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00
0.0e+00 2 7 0 0 0 2 7 0 0 0 1206 (in PETSc_1proc.log => 1.2 gflops
which is superior to the 1.1 gflops computed by Matt)
VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00
0.0e+00 3 6 0 0 0 3 6 0 0 0 1520 (in PETSc_2procs.log => 1.5 gflops
which is superior to the 1.1 gflops computed by Matt)
My understanding is that I can conclude I am at the bandwidth peak if I rely
on :
VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00
0.0e+00 2 7 0 0 0 2 7 0 0 0 1206 (in PETSc_1proc.log => 1.2 gflops
which is superior to the 1.1 gflops computed by Matt)
VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00
0.0e+00 3 6 0 0 0 3 6 0 0 0 1520 (in PETSc_2procs.log => 1.5 gflops
which is superior to the 1.1 gflops computed by Matt)
But I can not conclude anything when I rely on 3723 gflops for 1 proc / 6184
gflops for 2 procs.
Can somebody help me to see clear on that ?!
Thanks,
FH
________________________________________
De : Barry Smith [[email protected]]
Date d'envoi : jeudi 20 juin 2013 23:20
À : HOUSSEN Franck
Cc: [email protected]
Objet : Re: [petsc-users] How to get bandwidth peak out of PETSc log ?
Please send also the -log_summary for 1 process.
Note that in the run you provided the time spent in PETSc is about 25
percent of the total run. So how the "other" portion of the code scales will
make a large difference for speedup, hence we need to see with a different
number of processes.
Barry
On Jun 20, 2013, at 3:24 PM, HOUSSEN Franck <[email protected]> wrote:
> Hello,
>
> I am new to PETSc.
>
> I have written a (MPI) PETSc code to solve an AX=B system : how to know the
> bandwidth peak for a given run ? The code does not scale as I would expect
> (doubling the number of MPI processes does not half the elapsed time) : I
> would like to understand if this behavior is related to a bad MPI
> parallelization (that I may be able to improve), or, to the fact that the
> bandwidth limit as been reached (and in this case, my understanding is that I
> can not do anything to improve neither the performance, nor the scaling). I
> would like to know what's going on and why !
>
> Concerning the computer, I have tried to estimate the bandwidth peak with the
> "stream benchmark" : www.streambench.org/index.html. I get this :
> ~>./stream_c.exe
> ...
> Function Best Rate MB/s Avg time Min time Max time
> Copy: 11473.8 0.014110 0.013945 0.015064
> Scale: 11421.2 0.014070 0.014009 0.014096
> Add: 12974.4 0.018537 0.018498 0.018590
> Triad: 12964.6 0.018683 0.018512 0.019277
> As a conclusion, my understanding is that the bandwidth peak of my computer
> is about 12 200 MB / s (= average between 11 400 and 12 900) which is about
> 12200 / 1024 = 11.9 GB/s.
>
> Concerning PETSc, I tried to find (without success) a figure to compare to
> 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile (I run
> petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log (using
> -log_summary and -ksp_view) but I was not able to find out an estimate of the
> bandwidth out of it (I get information about time and flops but not about the
> amount of data transfered between MPI processes in MB). How can I get
> bandwidth peak out of PETSc log ? Is there a specific option for that or is
> this not possible ?
>
> I have attached the PETSc log (MPI run over 2 processes).
>
> Thanks,
>
> FH
> <PETSc.log>
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
../bin/NSIBM on a arch-linux2-c-opt named mdlslx07 with 1 processor, by fhoussen Fri Jun 21 07:45:55 2013
Using Petsc Release Version 3.4.1, Jun, 10, 2013
Max Max/Min Avg Total
Time (sec): 2.638e+02 1.00000 2.638e+02
Objects: 1.332e+03 1.00000 1.332e+03
Flops: 9.823e+10 1.00000 9.823e+10 9.823e+10
Flops/sec: 3.724e+08 1.00000 3.724e+08 3.724e+08
MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00
MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00
MPI Reductions: 2.944e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 2.6379e+02 100.0% 9.8232e+10 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.943e+03 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %f - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
KSPGMRESOrthog 20 1.0 5.0569e-02 1.0 1.01e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2002
KSPSetUp 408 1.0 3.1576e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.3e+01 0 0 0 0 1 0 0 0 0 1 0
KSPSolve 400 1.0 1.1151e+02 1.0 9.78e+10 1.0 0.0e+00 0.0e+00 1.2e+03 42100 0 0 41 42100 0 0 41 877
PCSetUp 4 1.0 3.8332e+00 1.0 4.45e+08 1.0 0.0e+00 0.0e+00 1.2e+02 1 0 0 0 4 1 0 0 0 4 116
PCSetUpOnBlocks 1200 1.0 1.0758e-01 1.0 3.60e+07 1.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 335
PCApply 2796 1.0 8.7881e+01 1.0 7.34e+10 1.0 0.0e+00 0.0e+00 9.0e+00 33 75 0 0 0 33 75 0 0 0 835
MatMult 13216 1.0 5.4614e+01 1.0 5.52e+10 1.0 0.0e+00 0.0e+00 0.0e+00 21 56 0 0 0 21 56 0 0 0 1010
MatMultAdd 1800 1.0 5.1904e+00 1.0 2.89e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 3 0 0 0 2 3 0 0 0 558
MatMultTranspose 1800 1.0 5.9022e+00 1.0 2.89e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 3 0 0 0 2 3 0 0 0 490
MatSolve 2796 1.0 1.4970e+01 1.0 1.06e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 11 0 0 0 6 11 0 0 0 705
MatLUFactorSym 1 1.0 4.3559e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatLUFactorNum 2 1.0 6.9269e-02 1.0 3.60e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 520
MatILUFactorSym 1 1.0 2.9269e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatConvert 2 1.0 2.8107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 6 1.0 2.1034e-02 1.0 1.15e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 545
MatAssemblyBegin 418 1.0 8.0109e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 418 1.0 6.1482e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0
MatGetRow 1380261 1.0 9.8042e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 2 1.0 1.4806e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 2 1.0 3.8633e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCoarsen 2 1.0 6.1192e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAXPY 2 1.0 1.0645e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMatMult 2 1.0 1.4493e-01 1.0 9.86e+06 1.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 68
MatMatMultSym 2 1.0 1.0025e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMatMultNum 2 1.0 4.4658e-02 1.0 9.86e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 221
MatPtAP 2 1.0 7.1076e-01 1.0 5.82e+07 1.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 82
MatPtAPSymbolic 2 1.0 2.6459e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatPtAPNumeric 2 1.0 4.4617e-01 1.0 5.82e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 130
MatTrnMatMult 2 1.0 1.8477e+00 1.0 1.20e+08 1.0 0.0e+00 0.0e+00 8.0e+00 1 0 0 0 0 1 0 0 0 0 65
MatGetSymTrans 4 1.0 4.1037e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecDot 2396 1.0 1.5234e+00 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1327
VecDotNorm2 1198 1.0 2.0391e+00 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 1.2e+03 1 2 0 0 41 1 2 0 0 41 991
VecMDot 20 1.0 2.2134e-02 1.0 5.06e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2286
VecNorm 1620 1.0 9.7562e-01 1.0 1.36e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1392
VecScale 7222 1.0 1.8196e+00 1.0 1.66e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 913
VecCopy 2600 1.0 9.1264e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 9485 1.0 1.1512e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 7 0 0 0 2 7 0 0 0 1206
VecAYPX 14400 1.0 6.2547e+00 1.0 4.14e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 662
VecAXPBYCZ 2396 1.0 2.8533e+00 1.0 4.04e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 1417
VecWAXPY 2396 1.0 2.3100e+00 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 875
VecMAXPY 22 1.0 3.3529e-02 1.0 5.98e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1784
VecAssemblyBegin 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAssemblyEnd 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 10822 1.0 5.3550e+00 1.0 2.49e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 3 0 0 0 2 3 0 0 0 465
VecSetRandom 2 1.0 8.3539e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 22 1.0 1.2867e-02 1.0 1.52e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1180
PCGAMGgraph_AGG 2 1.0 4.5846e-01 1.0 8.26e+06 1.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 18
PCGAMGcoarse_AGG 2 1.0 1.9906e+00 1.0 1.20e+08 1.0 0.0e+00 0.0e+00 1.4e+01 1 0 0 0 0 1 0 0 0 0 60
PCGAMGProl_AGG 2 1.0 9.0512e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCGAMGPOpt_AGG 2 1.0 4.5364e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 5.0e+01 0 0 0 0 2 0 0 0 0 2 491
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Krylov Solver 9 9 85984 0
Preconditioner 9 9 9140 0
Matrix 416 416 27690164964 0
Matrix Coarsen 2 2 1272 0
Vector 885 885 152281104 0
Index Set 8 8 1698460 0
PetscRandom 2 2 1264 0
Viewer 1 0 0 0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Tue Jun 18 18:24:43 2013
Configure options: --prefix=/home/fhoussen/Programs/PETSc/petsc-3.4.1/local --download-f-blas-lapack=yes --with-shared-libraries=0 --with-debugging=0
-----------------------------------------
Libraries compiled on Tue Jun 18 18:24:43 2013 on mdlslx07
Machine characteristics: Linux-3.2.0-30-generic-x86_64-with-Ubuntu-12.04-precise
Using PETSc directory: /home/fhoussen/Programs/PETSc/petsc-3.4.1
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------
Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lflapack -lfblas -lX11 -lpthread -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl
-----------------------------------------
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
../bin/NSIBM on a arch-linux2-c-opt named mdlslx07 with 2 processors, by fhoussen Fri Jun 21 07:50:43 2013
Using Petsc Release Version 3.4.1, Jun, 10, 2013
Max Max/Min Avg Total
Time (sec): 1.673e+02 1.00000 1.673e+02
Objects: 4.271e+03 1.00000 4.271e+03
Flops: 5.192e+10 1.00708 5.174e+10 1.035e+11
Flops/sec: 3.103e+08 1.00708 3.092e+08 6.184e+08
MPI Messages: 1.714e+04 1.00000 1.714e+04 3.427e+04
MPI Message Lengths: 5.056e+08 1.00000 2.951e+04 1.011e+09
MPI Reductions: 1.355e+04 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 1.6733e+02 100.0% 1.0348e+11 100.0% 3.427e+04 100.0% 2.951e+04 100.0% 1.355e+04 100.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %f - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
KSPGMRESOrthog 20 1.0 3.3943e-02 1.0 5.07e+07 1.0 0.0e+00 0.0e+00 2.0e+01 0 0 0 0 0 0 0 0 0 0 2983
KSPSetUp 408 1.0 1.2255e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 400 1.0 7.7299e+01 1.0 5.17e+10 1.0 3.2e+04 3.0e+04 5.9e+03 46100 95 97 43 46100 95 97 43 1333
PCSetUp 4 1.0 3.1902e+00 1.0 2.39e+08 1.1 2.1e+02 5.8e+04 4.8e+02 2 0 1 1 4 2 0 1 1 4 141
PCSetUpOnBlocks 1200 1.0 7.0002e-02 1.8 3.40e+07 7.6 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 550
PCApply 3130 1.0 5.9343e+01 1.0 3.78e+10 1.0 2.7e+04 2.7e+04 9.0e+00 35 73 79 72 0 35 73 79 72 0 1268
MatMult 13550 1.0 3.8204e+01 1.0 2.85e+10 1.0 2.7e+04 3.5e+04 0.0e+00 23 55 79 93 0 23 55 79 93 0 1494
MatMultAdd 1800 1.0 3.5612e+00 1.0 1.46e+09 1.0 2.7e+03 7.7e+03 0.0e+00 2 3 8 2 0 2 3 8 2 0 815
MatMultTranspose 1800 1.0 3.7446e+00 1.0 1.46e+09 1.0 2.7e+03 7.7e+03 0.0e+00 2 3 8 2 0 2 3 8 2 0 775
MatSolve 3130 1.4 9.1963e+00 1.0 6.32e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 12 0 0 0 5 12 0 0 0 1340
MatLUFactorSym 1 1.0 4.5419e-03453.6 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatLUFactorNum 2 1.0 4.7811e-02 2.1 3.40e+07 7.6 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 806
MatILUFactorSym 1 1.0 1.4721e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatConvert 2 1.0 1.1778e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 6 1.0 1.4802e-02 1.0 5.75e+06 1.0 4.0e+00 3.2e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 777
MatAssemblyBegin 432 1.0 3.1621e-01 2.2 0.00e+00 0.0 1.2e+01 6.0e+03 8.3e+02 0 0 0 0 6 0 0 0 0 6 0
MatAssemblyEnd 432 1.0 5.3555e+00 1.0 0.00e+00 0.0 1.6e+03 1.1e+04 3.3e+03 3 0 5 2 24 3 0 5 2 24 0
MatGetRow 690903 1.0 8.1701e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 2 2.0 1.6189e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 2 2.0 2.2869e-03 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCoarsen 2 1.0 4.5821e-02 1.0 0.00e+00 0.0 1.6e+01 6.4e+04 3.4e+01 0 0 0 0 0 0 0 0 0 0 0
MatAXPY 2 1.0 6.3078e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMatMult 2 1.0 1.3996e-01 1.0 4.14e+06 1.0 2.4e+01 1.6e+04 4.8e+01 0 0 0 0 0 0 0 0 0 0 59
MatMatMultSym 2 1.0 9.7141e-02 1.0 0.00e+00 0.0 2.0e+01 1.3e+04 4.4e+01 0 0 0 0 0 0 0 0 0 0 0
MatMatMultNum 2 1.0 4.2821e-02 1.0 4.14e+06 1.0 4.0e+00 3.2e+04 4.0e+00 0 0 0 0 0 0 0 0 0 0 193
MatPtAP 2 1.0 4.5969e-01 1.0 2.95e+07 1.0 3.6e+01 4.0e+04 5.0e+01 0 0 0 0 0 0 0 0 0 0 128
MatPtAPSymbolic 2 1.0 2.0768e-01 1.0 0.00e+00 0.0 2.4e+01 4.3e+04 3.0e+01 0 0 0 0 0 0 0 0 0 0 0
MatPtAPNumeric 2 1.0 2.5202e-01 1.0 2.95e+07 1.0 1.2e+01 3.2e+04 2.0e+01 0 0 0 0 0 0 0 0 0 0 233
MatTrnMatMult 2 1.0 1.7820e+00 1.0 6.10e+07 1.0 2.4e+01 2.5e+05 5.8e+01 1 0 0 1 0 1 0 0 1 0 68
MatGetLocalMat 10 1.0 4.9279e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 6 1.0 6.8142e-03 1.0 0.00e+00 0.0 2.8e+01 4.3e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetSymTrans 4 1.0 5.5912e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecDot 2730 1.0 1.1804e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 2.7e+03 1 2 0 0 20 1 2 0 0 20 1951
VecDotNorm2 1365 1.0 1.2446e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 1.4e+03 1 2 0 0 10 1 2 0 0 10 1851
VecMDot 20 1.0 1.5607e-02 1.0 2.53e+07 1.0 0.0e+00 0.0e+00 2.0e+01 0 0 0 0 0 0 0 0 0 0 3244
VecNorm 1787 1.0 6.5676e-01 1.1 7.51e+08 1.0 0.0e+00 0.0e+00 1.8e+03 0 1 0 0 13 0 1 0 0 13 2283
VecScale 7222 1.0 1.1229e+00 1.0 8.32e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1480
VecCopy 2600 1.0 7.1086e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 11963 1.0 7.7005e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 6 0 0 0 3 6 0 0 0 1520
VecAYPX 14400 1.0 4.4027e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 4 0 0 0 3 4 0 0 0 941
VecAXPBYCZ 2730 1.0 2.0359e+00 1.0 2.31e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 2263
VecWAXPY 2730 1.0 2.0002e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1152
VecMAXPY 22 1.0 2.1713e-02 1.0 2.99e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2756
VecAssemblyBegin 23 1.0 2.1164e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.6e+01 0 0 0 0 0 0 0 0 0 0 0
VecAssemblyEnd 23 1.0 1.9312e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 10822 1.0 4.3510e+00 1.0 1.25e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 2 0 0 0 3 2 0 0 0 572
VecScatterBegin 17175 1.0 9.1976e-01 1.1 0.00e+00 0.0 3.3e+04 3.0e+04 0.0e+00 1 0 95 97 0 1 0 95 97 0 0
VecScatterEnd 17175 1.0 1.5751e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
VecSetRandom 2 1.0 4.2439e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 22 1.0 7.4840e-03 1.1 7.60e+06 1.0 0.0e+00 0.0e+00 2.2e+01 0 0 0 0 0 0 0 0 0 0 2030
PCGAMGgraph_AGG 2 1.0 3.2885e-01 1.0 4.14e+06 1.0 1.2e+01 1.6e+04 3.8e+01 0 0 0 0 0 0 0 0 0 0 25
PCGAMGcoarse_AGG 2 1.0 1.9040e+00 1.0 6.10e+07 1.0 6.0e+01 1.3e+05 1.3e+02 1 0 0 1 1 1 0 0 1 1 64
PCGAMGProl_AGG 2 1.0 7.5517e-02 1.0 0.00e+00 0.0 2.8e+01 2.1e+04 5.6e+01 0 0 0 0 0 0 0 0 0 0 0
PCGAMGPOpt_AGG 2 1.0 3.2388e-01 1.0 1.11e+08 1.0 6.4e+01 2.6e+04 1.1e+02 0 0 0 0 1 0 0 0 0 1 683
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Krylov Solver 10 10 87136 0
Preconditioner 10 10 9804 0
Matrix 1260 1260 20815152180 0
Matrix Coarsen 2 2 1272 0
Vector 1729 1729 104655936 0
Vector Scatter 415 415 446540 0
Index Set 842 842 10725048 0
PetscRandom 2 2 1264 0
Viewer 1 0 0 0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 4.00543e-06
Average time for zero size MPI_Send(): 1.06096e-05
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Tue Jun 18 18:24:43 2013
Configure options: --prefix=/home/fhoussen/Programs/PETSc/petsc-3.4.1/local --download-f-blas-lapack=yes --with-shared-libraries=0 --with-debugging=0
-----------------------------------------
Libraries compiled on Tue Jun 18 18:24:43 2013 on mdlslx07
Machine characteristics: Linux-3.2.0-30-generic-x86_64-with-Ubuntu-12.04-precise
Using PETSc directory: /home/fhoussen/Programs/PETSc/petsc-3.4.1
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------
Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lflapack -lfblas -lX11 -lpthread -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl
-----------------------------------------