Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < [email protected]> wrote:
> I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams <[email protected]> > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > [email protected]> > *Cc: *"[email protected]" <[email protected]> > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > [email protected]> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > [email protected]> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams <[email protected]> > *Cc: *"[email protected]" <[email protected]> > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams <[email protected]> > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > [email protected]> > *Cc: *"[email protected]" <[email protected]> > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > [email protected]> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > >
