Hello,

The log I attached was a very small test case (1 iteration) : I just wanted to 
get some information about PETSc log printings (time, flops...). I profiled the 
code over "big" test case with scalasca : I know I spend 95% of the time 
calling PETSc (on realistic test cases).

Barry,
I have attached 2 logs over 1 and 2 procs running a intermediate test case (not 
small but not big).
>From PETSc_1proc.log, I get :
Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- 
Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     
Avg         %Total   counts   %Total 
 0:      Main Stage: 2.6379e+02 100.0%  9.8232e+10 100.0%  0.000e+00   0.0%  
0.000e+00        0.0%  2.943e+03 100.0% 
My understanding is that the relevant figure is 9.8232e+10 (flop) / 2.6379e+02 
(sec) = 3.72e12 flops = 3723 gflops for 1 proc.
>From PETSc_2procs.log, I get :
Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- 
Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     
Avg         %Total   counts   %Total 
 0:      Main Stage: 1.6733e+02 100.0%  1.0348e+11 100.0%  3.427e+04 100.0%  
2.951e+04      100.0%  1.355e+04 100.0% 
My understanding is that the relevant figure is 1.0348e+11 (flop) / 1.6733e+02 
(sec) = 6.18e12 flops = 6184 gflops for 2 procs.

Am I correct ?

My understanding is that if the code scales "well", when I double the number of 
MPI processes, I should double the flops which is not the case (I get 6184 
gflops instead of 2*3723) : right ? wrong ?
>From this, how can I know if I am at the bandwidth peak ?

If I compare 6184 gflops to the (computer characteristic computed by Matt = 
12.9 GB/s * 1 flops/12 bytes =) 1.1 GF/s, I get a huge difference that I am not 
sure to understand... I am not sure I can compare these numbers : I guess no ?! 
Did I miss something ?

More over I realize that I get :
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                            
 --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct 
 %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
VecAXPY            14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 
0.0e+00  2  7  0  0  0   2  7  0  0  0  1206 (in PETSc_1proc.log => 1.2 gflops 
which is superior to the 1.1 gflops computed by Matt)
VecAXPY            14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 
0.0e+00  3  6  0  0  0   3  6  0  0  0  1520 (in PETSc_2procs.log => 1.5 gflops 
which is superior to the 1.1 gflops computed by Matt)

My understanding is that I can conclude I  am at the bandwidth peak if I rely 
on :
VecAXPY            14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 
0.0e+00  2  7  0  0  0   2  7  0  0  0  1206 (in PETSc_1proc.log => 1.2 gflops 
which is superior to the 1.1 gflops computed by Matt)
VecAXPY            14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 
0.0e+00  3  6  0  0  0   3  6  0  0  0  1520 (in PETSc_2procs.log => 1.5 gflops 
which is superior to the 1.1 gflops computed by Matt)

But I can not conclude anything when I rely on 3723 gflops for 1 proc /  6184 
gflops for 2 procs.

Can somebody help me to see clear on that ?!

Thanks,

FH
 
________________________________________
De : Barry Smith [[email protected]]
Date d'envoi : jeudi 20 juin 2013 23:20
À : HOUSSEN Franck
Cc: [email protected]
Objet : Re: [petsc-users] How to get bandwidth peak out of PETSc log ?

   Please send also the -log_summary for 1 process.

   Note that in the run you provided the time spent in PETSc is about 25 
percent of the total run. So how the "other" portion of the code scales will 
make a large difference for speedup, hence we need to see with a different 
number of processes.

   Barry

On Jun 20, 2013, at 3:24 PM, HOUSSEN Franck <[email protected]> wrote:

> Hello,
>
> I am new to PETSc.
>
> I have written a (MPI) PETSc code to solve an AX=B system : how to know the 
> bandwidth peak for a given run ? The code does not scale as I would expect 
> (doubling the number of MPI processes does not half the elapsed time) : I 
> would like to understand if this behavior is related to a bad MPI 
> parallelization (that I may be able to improve), or, to the fact that the 
> bandwidth limit as been reached (and in this case, my understanding is that I 
> can not do anything to improve neither the performance, nor the scaling). I 
> would like to know what's going on and why !
>
> Concerning the computer, I have tried to estimate the bandwidth peak with the 
> "stream benchmark" : www.streambench.org/index.html. I get this :
> ~>./stream_c.exe
> ...
> Function    Best Rate MB/s  Avg time     Min time     Max time
> Copy:           11473.8     0.014110     0.013945     0.015064
> Scale:          11421.2     0.014070     0.014009     0.014096
> Add:            12974.4     0.018537     0.018498     0.018590
> Triad:          12964.6     0.018683     0.018512     0.019277
> As a conclusion, my understanding is that the bandwidth peak of my computer 
> is about 12 200 MB / s (= average between 11 400 and 12 900) which is about 
> 12200 / 1024 = 11.9 GB/s.
>
> Concerning PETSc, I tried to find (without success) a figure to compare to 
> 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile (I run 
> petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log (using 
> -log_summary and -ksp_view) but I was not able to find out an estimate of the 
> bandwidth out of it (I get information about time and flops but not about the 
> amount of data transfered between MPI processes in MB). How can I get 
> bandwidth peak out of PETSc log ? Is there a specific option for that or is 
> this not possible ?
>
> I have attached the PETSc log (MPI run over 2 processes).
>
> Thanks,
>
> FH
> <PETSc.log>

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

../bin/NSIBM on a arch-linux2-c-opt named mdlslx07 with 1 processor, by fhoussen Fri Jun 21 07:45:55 2013
Using Petsc Release Version 3.4.1, Jun, 10, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.638e+02      1.00000   2.638e+02
Objects:              1.332e+03      1.00000   1.332e+03
Flops:                9.823e+10      1.00000   9.823e+10  9.823e+10
Flops/sec:            3.724e+08      1.00000   3.724e+08  3.724e+08
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       2.944e+03      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.6379e+02 100.0%  9.8232e+10 100.0%  0.000e+00   0.0%  0.000e+00        0.0%  2.943e+03 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

KSPGMRESOrthog        20 1.0 5.0569e-02 1.0 1.01e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2002
KSPSetUp             408 1.0 3.1576e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.3e+01  0  0  0  0  1   0  0  0  0  1     0
KSPSolve             400 1.0 1.1151e+02 1.0 9.78e+10 1.0 0.0e+00 0.0e+00 1.2e+03 42100  0  0 41  42100  0  0 41   877
PCSetUp                4 1.0 3.8332e+00 1.0 4.45e+08 1.0 0.0e+00 0.0e+00 1.2e+02  1  0  0  0  4   1  0  0  0  4   116
PCSetUpOnBlocks     1200 1.0 1.0758e-01 1.0 3.60e+07 1.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0   335
PCApply             2796 1.0 8.7881e+01 1.0 7.34e+10 1.0 0.0e+00 0.0e+00 9.0e+00 33 75  0  0  0  33 75  0  0  0   835
MatMult            13216 1.0 5.4614e+01 1.0 5.52e+10 1.0 0.0e+00 0.0e+00 0.0e+00 21 56  0  0  0  21 56  0  0  0  1010
MatMultAdd          1800 1.0 5.1904e+00 1.0 2.89e+09 1.0 0.0e+00 0.0e+00 0.0e+00  2  3  0  0  0   2  3  0  0  0   558
MatMultTranspose    1800 1.0 5.9022e+00 1.0 2.89e+09 1.0 0.0e+00 0.0e+00 0.0e+00  2  3  0  0  0   2  3  0  0  0   490
MatSolve            2796 1.0 1.4970e+01 1.0 1.06e+10 1.0 0.0e+00 0.0e+00 0.0e+00  6 11  0  0  0   6 11  0  0  0   705
MatLUFactorSym         1 1.0 4.3559e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         2 1.0 6.9269e-02 1.0 3.60e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   520
MatILUFactorSym        1 1.0 2.9269e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatConvert             2 1.0 2.8107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               6 1.0 2.1034e-02 1.0 1.15e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   545
MatAssemblyBegin     418 1.0 8.0109e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd       418 1.0 6.1482e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
MatGetRow        1380261 1.0 9.8042e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            2 1.0 1.4806e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         2 1.0 3.8633e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             2 1.0 6.1192e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAXPY                2 1.0 1.0645e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             2 1.0 1.4493e-01 1.0 9.86e+06 1.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0    68
MatMatMultSym          2 1.0 1.0025e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMultNum          2 1.0 4.4658e-02 1.0 9.86e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   221
MatPtAP                2 1.0 7.1076e-01 1.0 5.82e+07 1.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0    82
MatPtAPSymbolic        2 1.0 2.6459e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatPtAPNumeric         2 1.0 4.4617e-01 1.0 5.82e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   130
MatTrnMatMult          2 1.0 1.8477e+00 1.0 1.20e+08 1.0 0.0e+00 0.0e+00 8.0e+00  1  0  0  0  0   1  0  0  0  0    65
MatGetSymTrans         4 1.0 4.1037e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecDot              2396 1.0 1.5234e+00 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  1327
VecDotNorm2         1198 1.0 2.0391e+00 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 1.2e+03  1  2  0  0 41   1  2  0  0 41   991
VecMDot               20 1.0 2.2134e-02 1.0 5.06e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2286
VecNorm             1620 1.0 9.7562e-01 1.0 1.36e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  1392
VecScale            7222 1.0 1.8196e+00 1.0 1.66e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0   913
VecCopy             2600 1.0 9.1264e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              9485 1.0 1.1512e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  2  7  0  0  0   2  7  0  0  0  1206
VecAYPX            14400 1.0 6.2547e+00 1.0 4.14e+09 1.0 0.0e+00 0.0e+00 0.0e+00  2  4  0  0  0   2  4  0  0  0   662
VecAXPBYCZ          2396 1.0 2.8533e+00 1.0 4.04e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  1417
VecWAXPY            2396 1.0 2.3100e+00 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0   875
VecMAXPY              22 1.0 3.3529e-02 1.0 5.98e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1784
VecAssemblyBegin       2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd         2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult   10822 1.0 5.3550e+00 1.0 2.49e+09 1.0 0.0e+00 0.0e+00 0.0e+00  2  3  0  0  0   2  3  0  0  0   465
VecSetRandom           2 1.0 8.3539e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          22 1.0 1.2867e-02 1.0 1.52e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1180
PCGAMGgraph_AGG        2 1.0 4.5846e-01 1.0 8.26e+06 1.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0    18
PCGAMGcoarse_AGG       2 1.0 1.9906e+00 1.0 1.20e+08 1.0 0.0e+00 0.0e+00 1.4e+01  1  0  0  0  0   1  0  0  0  0    60
PCGAMGProl_AGG         2 1.0 9.0512e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCGAMGPOpt_AGG         2 1.0 4.5364e-01 1.0 2.23e+08 1.0 0.0e+00 0.0e+00 5.0e+01  0  0  0  0  2   0  0  0  0  2   491
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

       Krylov Solver     9              9        85984     0
      Preconditioner     9              9         9140     0
              Matrix   416            416  27690164964     0
      Matrix Coarsen     2              2         1272     0
              Vector   885            885    152281104     0
           Index Set     8              8      1698460     0
         PetscRandom     2              2         1264     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Tue Jun 18 18:24:43 2013
Configure options: --prefix=/home/fhoussen/Programs/PETSc/petsc-3.4.1/local --download-f-blas-lapack=yes --with-shared-libraries=0 --with-debugging=0
-----------------------------------------
Libraries compiled on Tue Jun 18 18:24:43 2013 on mdlslx07 
Machine characteristics: Linux-3.2.0-30-generic-x86_64-with-Ubuntu-12.04-precise
Using PETSc directory: /home/fhoussen/Programs/PETSc/petsc-3.4.1
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------

Using C compiler: mpicc  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: mpif90  -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi
-----------------------------------------

Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lflapack -lfblas -lX11 -lpthread -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl 
-----------------------------------------

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

../bin/NSIBM on a arch-linux2-c-opt named mdlslx07 with 2 processors, by fhoussen Fri Jun 21 07:50:43 2013
Using Petsc Release Version 3.4.1, Jun, 10, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           1.673e+02      1.00000   1.673e+02
Objects:              4.271e+03      1.00000   4.271e+03
Flops:                5.192e+10      1.00708   5.174e+10  1.035e+11
Flops/sec:            3.103e+08      1.00708   3.092e+08  6.184e+08
MPI Messages:         1.714e+04      1.00000   1.714e+04  3.427e+04
MPI Message Lengths:  5.056e+08      1.00000   2.951e+04  1.011e+09
MPI Reductions:       1.355e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 1.6733e+02 100.0%  1.0348e+11 100.0%  3.427e+04 100.0%  2.951e+04      100.0%  1.355e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

KSPGMRESOrthog        20 1.0 3.3943e-02 1.0 5.07e+07 1.0 0.0e+00 0.0e+00 2.0e+01  0  0  0  0  0   0  0  0  0  0  2983
KSPSetUp             408 1.0 1.2255e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
KSPSolve             400 1.0 7.7299e+01 1.0 5.17e+10 1.0 3.2e+04 3.0e+04 5.9e+03 46100 95 97 43  46100 95 97 43  1333
PCSetUp                4 1.0 3.1902e+00 1.0 2.39e+08 1.1 2.1e+02 5.8e+04 4.8e+02  2  0  1  1  4   2  0  1  1  4   141
PCSetUpOnBlocks     1200 1.0 7.0002e-02 1.8 3.40e+07 7.6 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0   550
PCApply             3130 1.0 5.9343e+01 1.0 3.78e+10 1.0 2.7e+04 2.7e+04 9.0e+00 35 73 79 72  0  35 73 79 72  0  1268
MatMult            13550 1.0 3.8204e+01 1.0 2.85e+10 1.0 2.7e+04 3.5e+04 0.0e+00 23 55 79 93  0  23 55 79 93  0  1494
MatMultAdd          1800 1.0 3.5612e+00 1.0 1.46e+09 1.0 2.7e+03 7.7e+03 0.0e+00  2  3  8  2  0   2  3  8  2  0   815
MatMultTranspose    1800 1.0 3.7446e+00 1.0 1.46e+09 1.0 2.7e+03 7.7e+03 0.0e+00  2  3  8  2  0   2  3  8  2  0   775
MatSolve            3130 1.4 9.1963e+00 1.0 6.32e+09 1.1 0.0e+00 0.0e+00 0.0e+00  5 12  0  0  0   5 12  0  0  0  1340
MatLUFactorSym         1 1.0 4.5419e-03453.6 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         2 1.0 4.7811e-02 2.1 3.40e+07 7.6 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   806
MatILUFactorSym        1 1.0 1.4721e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatConvert             2 1.0 1.1778e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               6 1.0 1.4802e-02 1.0 5.75e+06 1.0 4.0e+00 3.2e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0   777
MatAssemblyBegin     432 1.0 3.1621e-01 2.2 0.00e+00 0.0 1.2e+01 6.0e+03 8.3e+02  0  0  0  0  6   0  0  0  0  6     0
MatAssemblyEnd       432 1.0 5.3555e+00 1.0 0.00e+00 0.0 1.6e+03 1.1e+04 3.3e+03  3  0  5  2 24   3  0  5  2 24     0
MatGetRow         690903 1.0 8.1701e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            2 2.0 1.6189e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         2 2.0 2.2869e-03 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             2 1.0 4.5821e-02 1.0 0.00e+00 0.0 1.6e+01 6.4e+04 3.4e+01  0  0  0  0  0   0  0  0  0  0     0
MatAXPY                2 1.0 6.3078e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             2 1.0 1.3996e-01 1.0 4.14e+06 1.0 2.4e+01 1.6e+04 4.8e+01  0  0  0  0  0   0  0  0  0  0    59
MatMatMultSym          2 1.0 9.7141e-02 1.0 0.00e+00 0.0 2.0e+01 1.3e+04 4.4e+01  0  0  0  0  0   0  0  0  0  0     0
MatMatMultNum          2 1.0 4.2821e-02 1.0 4.14e+06 1.0 4.0e+00 3.2e+04 4.0e+00  0  0  0  0  0   0  0  0  0  0   193
MatPtAP                2 1.0 4.5969e-01 1.0 2.95e+07 1.0 3.6e+01 4.0e+04 5.0e+01  0  0  0  0  0   0  0  0  0  0   128
MatPtAPSymbolic        2 1.0 2.0768e-01 1.0 0.00e+00 0.0 2.4e+01 4.3e+04 3.0e+01  0  0  0  0  0   0  0  0  0  0     0
MatPtAPNumeric         2 1.0 2.5202e-01 1.0 2.95e+07 1.0 1.2e+01 3.2e+04 2.0e+01  0  0  0  0  0   0  0  0  0  0   233
MatTrnMatMult          2 1.0 1.7820e+00 1.0 6.10e+07 1.0 2.4e+01 2.5e+05 5.8e+01  1  0  0  1  0   1  0  0  1  0    68
MatGetLocalMat        10 1.0 4.9279e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          6 1.0 6.8142e-03 1.0 0.00e+00 0.0 2.8e+01 4.3e+04 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetSymTrans         4 1.0 5.5912e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecDot              2730 1.0 1.1804e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 2.7e+03  1  2  0  0 20   1  2  0  0 20  1951
VecDotNorm2         1365 1.0 1.2446e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 1.4e+03  1  2  0  0 10   1  2  0  0 10  1851
VecMDot               20 1.0 1.5607e-02 1.0 2.53e+07 1.0 0.0e+00 0.0e+00 2.0e+01  0  0  0  0  0   0  0  0  0  0  3244
VecNorm             1787 1.0 6.5676e-01 1.1 7.51e+08 1.0 0.0e+00 0.0e+00 1.8e+03  0  1  0  0 13   0  1  0  0 13  2283
VecScale            7222 1.0 1.1229e+00 1.0 8.32e+08 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  1480
VecCopy             2600 1.0 7.1086e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             11963 1.0 7.7005e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00  3  6  0  0  0   3  6  0  0  0  1520
VecAYPX            14400 1.0 4.4027e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00  3  4  0  0  0   3  4  0  0  0   941
VecAXPBYCZ          2730 1.0 2.0359e+00 1.0 2.31e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  4  0  0  0   1  4  0  0  0  2263
VecWAXPY            2730 1.0 2.0002e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  2  0  0  0   1  2  0  0  0  1152
VecMAXPY              22 1.0 2.1713e-02 1.0 2.99e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2756
VecAssemblyBegin      23 1.0 2.1164e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.6e+01  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd        23 1.0 1.9312e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult   10822 1.0 4.3510e+00 1.0 1.25e+09 1.0 0.0e+00 0.0e+00 0.0e+00  3  2  0  0  0   3  2  0  0  0   572
VecScatterBegin    17175 1.0 9.1976e-01 1.1 0.00e+00 0.0 3.3e+04 3.0e+04 0.0e+00  1  0 95 97  0   1  0 95 97  0     0
VecScatterEnd      17175 1.0 1.5751e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
VecSetRandom           2 1.0 4.2439e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          22 1.0 7.4840e-03 1.1 7.60e+06 1.0 0.0e+00 0.0e+00 2.2e+01  0  0  0  0  0   0  0  0  0  0  2030
PCGAMGgraph_AGG        2 1.0 3.2885e-01 1.0 4.14e+06 1.0 1.2e+01 1.6e+04 3.8e+01  0  0  0  0  0   0  0  0  0  0    25
PCGAMGcoarse_AGG       2 1.0 1.9040e+00 1.0 6.10e+07 1.0 6.0e+01 1.3e+05 1.3e+02  1  0  0  1  1   1  0  0  1  1    64
PCGAMGProl_AGG         2 1.0 7.5517e-02 1.0 0.00e+00 0.0 2.8e+01 2.1e+04 5.6e+01  0  0  0  0  0   0  0  0  0  0     0
PCGAMGPOpt_AGG         2 1.0 3.2388e-01 1.0 1.11e+08 1.0 6.4e+01 2.6e+04 1.1e+02  0  0  0  0  1   0  0  0  0  1   683
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

       Krylov Solver    10             10        87136     0
      Preconditioner    10             10         9804     0
              Matrix  1260           1260  20815152180     0
      Matrix Coarsen     2              2         1272     0
              Vector  1729           1729    104655936     0
      Vector Scatter   415            415       446540     0
           Index Set   842            842     10725048     0
         PetscRandom     2              2         1264     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 4.00543e-06
Average time for zero size MPI_Send(): 1.06096e-05
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Tue Jun 18 18:24:43 2013
Configure options: --prefix=/home/fhoussen/Programs/PETSc/petsc-3.4.1/local --download-f-blas-lapack=yes --with-shared-libraries=0 --with-debugging=0
-----------------------------------------
Libraries compiled on Tue Jun 18 18:24:43 2013 on mdlslx07 
Machine characteristics: Linux-3.2.0-30-generic-x86_64-with-Ubuntu-12.04-precise
Using PETSc directory: /home/fhoussen/Programs/PETSc/petsc-3.4.1
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------

Using C compiler: mpicc  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: mpif90  -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/include -I/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi
-----------------------------------------

Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -L/home/fhoussen/Programs/PETSc/petsc-3.4.1/arch-linux2-c-opt/lib -lflapack -lfblas -lX11 -lpthread -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl 
-----------------------------------------

Reply via email to