Re: [petsc-users] Scaling problem when cores > 600

KSPSolve would include all the KSP solves You can use PetscLogStageRegister(), PetscLogStagePush(), PetscLogStagePop(), to have the information about each solve displayed separately. Barry > On Apr 22, 2018, at 11:42 AM, TAY wee-beng wrote: > > > On 22/4/2018 5:22 AM, Smith, Barry

Re: [petsc-users] Scaling problem when cores > 600

On 22/4/2018 5:22 AM, Smith, Barry F. wrote: Comparing the time in KSPSolve() 1.8116e+02/3.5276e+01 5.135502891484295 to the increase in number of processes one sees 1440/288. 5.0 so very very good speedup. So the linear solver is working very well. The problem is the rest of the

Re: [petsc-users] Scaling problem when cores > 600

Comparing the time in KSPSolve() >>> 1.8116e+02/3.5276e+01 5.135502891484295 to the increase in number of processes one sees >>> 1440/288. 5.0 so very very good speedup. So the linear solver is working very well. The problem is the rest of the code. On 288 processes the code is spending

Re: [petsc-users] Scaling problem when cores > 600

looks fine to me. On Sat, Apr 21, 2018 at 11:34 AM, TAY wee-beng wrote: > Hi, > > I have found some time to work on this scaling problem again. I am now > using: > > mpirun ./a.out -log_view -poisson_pc_type gamg > -poisson_pc_gamg_agg_nsmooths 1 > > I have attached the log_view output for 288,

Re: [petsc-users] Scaling problem when cores > 600

Hi, I have found some time to work on this scaling problem again. I am now using: mpirun ./a.out -log_view -poisson_pc_type gamg -poisson_pc_gamg_agg_nsmooths 1 I have attached the log_view output for 288, 600, 960, 1440 procs for comparison. Please give some comments. Thank you very m

Re: [petsc-users] Scaling problem when cores > 600

What are you using for Poisson log. If it is a Poisson problem then almost for sure you should be using Hypre BoomerAMG?. It sounds like your matrix does not change. You will need to discuss the scaling with the hypre people. Barry > On Mar 7, 2018, at 5:38 AM, TAY wee-beng wro

Re: [petsc-users] Scaling problem when cores > 600

On Mon, 5 Mar 2018, Smith, Barry F. wrote: > > My Poisson eqn solving should be the biggest culprit. I call it thru: > > > > call hypre_solver(p_array,q_p_array) > > > > So in this case, I first create a global variable with > > PetscLogEventRegister, and then before/after the subroutine, I cal

Re: [petsc-users] Scaling problem when cores > 600

> On Mar 5, 2018, at 9:59 AM, TAY wee-beng wrote: > > > On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >> 360 process >> >> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 >> 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >> >> 1920 processes >> >> KSPSolve

Re: [petsc-users] Scaling problem when cores > 600

On 5/3/2018 11:43 AM, Smith, Barry F. wrote: 360 process KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 1920 processes KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 410

Re: [petsc-users] Scaling problem when cores > 600

360 process KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 1920 processes KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 Ratio of number of processes

Re: [petsc-users] Scaling problem when cores > 600

On 1/3/2018 12:14 PM, Smith, Barry F. wrote: On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: On 1/3/2018 12:10 AM, Matthew Knepley wrote: On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: Hi, I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid si

Re: [petsc-users] Scaling problem when cores > 600

> On Feb 28, 2018, at 10:59 PM, TAY wee-beng wrote: > > > On 1/3/2018 10:07 AM, Matthew Knepley wrote: >> On Wed, Feb 28, 2018 at 9:01 PM, TAY wee-beng wrote: >> >> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: >>> Hi, >>> >>> I have

Re: [petsc-users] Scaling problem when cores > 600

On 1/3/2018 10:07 AM, Matthew Knepley wrote: On Wed, Feb 28, 2018 at 9:01 PM, TAY wee-beng > wrote: On 1/3/2018 12:10 AM, Matthew Knepley wrote: On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng mailto:zon...@gmail.com>> wrote: Hi, I have a CFD

Re: [petsc-users] Scaling problem when cores > 600

> On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: > > > On 1/3/2018 12:10 AM, Matthew Knepley wrote: >> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: >> Hi, >> >> I have a CFD code which uses PETSc and HYPRE. I found that for a certain >> case with grid size of 192,570,048, I encou

Re: [petsc-users] Scaling problem when cores > 600

> > > Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? > However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't > converge. > As Matt said GMRES is not great for symmetric operators like Poisson and you can use CG for the KSP method. HYPRE and GAMG are both

Re: [petsc-users] Scaling problem when cores > 600

On Wed, Feb 28, 2018 at 9:01 PM, TAY wee-beng wrote: > > On 1/3/2018 12:10 AM, Matthew Knepley wrote: > > On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: > >> Hi, >> >> I have a CFD code which uses PETSc and HYPRE. I found that for a certain >> case with grid size of 192,570,048, I encount

Re: [petsc-users] Scaling problem when cores > 600

On 1/3/2018 12:10 AM, Matthew Knepley wrote: On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng > wrote: Hi, I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cor

Re: [petsc-users] Scaling problem when cores > 600

On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: > Hi, > > I have a CFD code which uses PETSc and HYPRE. I found that for a certain > case with grid size of 192,570,048, I encounter scaling problem when my > cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, > 1440 an