Hi Matt, thanks for all the help so far. The -info option is really very helpful. So i think i straightened the actual errors out. However, now i am back to the original question i had. That is why it takes so much longer on 4 procs than on 1 proc.
I profiled the KSPSolve(...) as stage 2: For 1 proc i have: --- Event Stage 2: Stage 2 of ChomboPetscInterface VecDot 4000 1.0 4.9158e-02 1.0 4.74e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 2 18 0 0 0 474 VecNorm 8000 1.0 2.1798e-01 1.0 2.14e+08 1.0 0.0e+00 0.0e+00 4.0e+03 1 36 0 0 28 7 36 0 0 33 214 VecAYPX 4000 1.0 1.3449e-01 1.0 1.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 5 18 0 0 0 173 MatMult 4000 1.0 3.6004e-01 1.0 3.24e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 9 0 0 0 12 9 0 0 0 32 MatSolve 8000 1.0 1.0620e+00 1.0 2.19e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 18 0 0 0 36 18 0 0 0 22 KSPSolve 4000 1.0 2.8338e+00 1.0 4.52e+07 1.0 0.0e+00 0.0e+00 1.2e+04 7100 0 0 84 97100 0 0100 45 PCApply 8000 1.0 1.1133e+00 1.0 2.09e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 18 0 0 0 38 18 0 0 0 21 for 4 procs i have : --- Event Stage 2: Stage 2 of ChomboPetscInterface VecDot 4000 1.0 3.5884e+01133.7 2.17e+07133.7 0.0e+00 0.0e+00 4.0e+03 8 18 0 0 5 9 18 0 0 14 1 VecNorm 8000 1.0 3.4986e-01 1.3 4.43e+07 1.3 0.0e+00 0.0e+00 8.0e+03 0 36 0 0 10 0 36 0 0 29 133 VecSet 8000 1.0 3.5024e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAYPX 4000 1.0 5.6790e-02 1.3 1.28e+08 1.3 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 0 18 0 0 0 410 VecScatterBegin 4000 1.0 6.0042e+01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 38 0 0 0 0 45 0 0 0 0 0 VecScatterEnd 4000 1.0 5.9364e+01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 37 0 0 0 0 44 0 0 0 0 0 MatMult 4000 1.0 1.1959e+02 1.4 3.46e+04 1.4 0.0e+00 0.0e+00 0.0e+00 75 9 0 0 0 89 9 0 0 0 0 MatSolve 8000 1.0 2.8150e-01 1.0 2.16e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 0 18 0 0 0 83 MatLUFactorNum 1 1.0 1.3685e-04 1.1 5.64e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 21 MatILUFactorSym 1 1.0 2.3389e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 9.6083e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetup 1 1.0 2.1458e-06 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 4000 1.0 1.2200e+02 1.0 2.63e+05 1.0 0.0e+00 0.0e+00 2.8e+04 84100 0 0 34 100100 0 0100 1 PCSetUp 1 1.0 5.0187e-04 1.2 1.68e+06 1.2 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 6 PCSetUpOnBlocks 4000 1.0 1.2104e-02 2.2 1.34e+05 2.2 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 8000 1.0 8.4254e-01 1.2 8.27e+06 1.2 0.0e+00 0.0e+00 8.0e+03 1 18 0 0 10 1 18 0 0 29 28 ------------------------------------------------------------------------------------------------------------------------ Now if i understand it right, all these calls summarize all calls between the pop and push commands. That would mean that the majority of the time is spend in the MatMult and in within that the VecScatterBegin and VecScatterEnd commands (if i understand it right). My problem size is really small. So i was wondering if the problem lies in that (namely that the major time is simply spend communicating between processors, or whether there is still something wrong with how i wrote the code?) thanks mat On Tuesday 01 August 2006 18:28, Matthew Knepley wrote: > On 8/1/06, Matt Funk <mafunk at nmsu.edu> wrote: > > Actually the errors occur on my calls to a PETSc functions after calling > > PETSCInitialize. > > Yes, it is the error I pointed out in the last message. > > Matt > > > mat
