On Wed, Sep 29, 2021 at 5:52 AM Karthikeyan Chockalingam - STFC UKRI < [email protected]> wrote:
> That was helpful. I would like to provide some additional details of my > run on cpus and gpus. Please find the following attachments: > > > > 1. graph.pdf a plot showing overall time and various petsc events. > 2. ksp_ex45_N511_cpu_6.txt data file of the log_summary > 3. ksp_ex45_N511_gpu_2.txt data file of the log_summary > > > > I used the following petsc options for cpu > > > > mpirun -n 6 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z > 511 -dm_mat_type mpiaij -dm_vec_type mpi -ksp_type cg -pc_type bjacobi > -ksp_monitor > > > > and for gpus > > > > mpirun -n 1 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z > 511 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type cg -pc_type > bjacobi -ksp_monitor > > > > to run the following problem > > > > https://petsc.org/release/src/ksp/ksp/tutorials/ex45.c.html > > > > From the above code, I see is there no individual function called KSPSetUp(), > so I gather KSPSetDM, KSPSetComputeInitialGuess, KSPSetComputeRHS, > kSPSetComputeOperators all are timed together as KSPSetUp. For this > example, is KSPSetUp time and KSPSolve time mutually exclusive? > No, KSPSetUp() will be contained in KSPSolve() if it is called automatically. > In your response you said that > > > > “PCSetUp() time may be in KSPSetUp() or it maybe in PCApply() it > depends on how much of the preconditioner construction can take place > early, so depends exactly on the preconditioner used.” > > > > I don’t see a explicit call to PCSetUp() or PCApply() in ex45; so for > this particular preconditioner (bjacobi) how can I tell how they are timed? > They are all inside KSPSolve(). If you have a preconditioned linear solve, the oreconditioning happens during the iteration. So an iteration would mostly consist of MatMult + PCApply, with some vector work. > I am hoping to time KSP solving and preconditioning mutually exclusively. > I am not sure that concept makes sense here. See above. Thanks, Matt > > > Kind regards, > > Karthik. > > > > > > *From: *Barry Smith <[email protected]> > *Date: *Tuesday, 28 September 2021 at 19:19 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > [email protected]> > *Cc: *"[email protected]" <[email protected]> > *Subject: *Re: [petsc-users] %T (percent time in this phase) > > > > > > > > On Sep 28, 2021, at 12:11 PM, Karthikeyan Chockalingam - STFC UKRI < > [email protected]> wrote: > > > > Thanks for Barry for your response. > > > > I was just benchmarking the problem with various preconditioner on cpu and > gpu. I understand, it is not possible to get mutually exclusive timing. > > However, can you tell if KSPSolve time includes both PCSetup and PCApply? > And if KSPSolve and KSPSetup are mutually exclusive? Likewise for PCSetUp > and PCApply. > > > > If you do not call KSPSetUp() separately from KSPSolve() then its time > is included with KSPSolve(). > > > > PCSetUp() time may be in KSPSetUp() or it maybe in PCApply() it depends > on how much of the preconditioner construction can take place early, so > depends exactly on the preconditioner used. > > > > So yes the answer is not totally satisfying. The one thing I would > recommend is to not call KSPSetUp() directly and then KSPSolve() will > always include the total time of the solve plus all setup time. PCApply > will contain all the time to apply the preconditioner but may also include > some setup time. > > > > Barry > > > > > > Best, > > Karthik. > > > > > > > > > > *From: *Barry Smith <[email protected]> > *Date: *Tuesday, 28 September 2021 at 16:56 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > [email protected]> > *Cc: *"[email protected]" <[email protected]> > *Subject: *Re: [petsc-users] %T (percent time in this phase) > > > > > > > > > On Sep 28, 2021, at 10:55 AM, Karthikeyan Chockalingam - STFC UKRI < > [email protected]> wrote: > > > > Hello, > > > > I ran ex45 in the KPS tutorial, which is a 3D finite-difference Poisson > problem. I noticed from the output from using the flag -log_summary that > for various events their respective %T (percent time in this phase) do not > add up to 100 but rather exceeds 100. So, I gather there is some overlap > among these events. I am primarily looking at the events KSPSetUp, > KSPSolve, PCSetUp and PCSolve. Is it possible to get a mutually exclusive > %T or Time for these individual events? I have attached the log_summary > output file from my run for your reference. > > > > > > For nested solvers it is tricky to get the times to be mutually > exclusive because some parts of the building of the preconditioner is for > some preconditioners delayed until the solve has started. > > > > It looks like you are using the default preconditioner options which for > this example are taking more or less no time since so many iterations are > needed. It is best to use -pc_type mg to use geometric multigrid on this > problem. > > > > Barry > > > > > > > > > Thanks! > > Karthik. > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > <ksp_ex45_N511_cpu_6.txt> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
