On Thu, Jan 1, 2015 at 2:15 AM, TAY wee-beng <[email protected]> wrote:
> > On 1/1/2015 2:06 PM, Matthew Knepley wrote: > > On Wed, Dec 31, 2014 at 11:20 PM, TAY wee-beng <[email protected]> wrote: > >> Hi, >> >> I used to run my CFD code with 96 procs, with a grid size of 231 x 461 x >> 368. >> >> I used MPI and partition my grid in the z direction. Hence with 96 procs >> (8 nodes, each 12 procs), each procs has a size of 231 x 461 x 3 or 231 x >> 461 x 4. >> >> It worked fine. >> >> Now I modified the code and added some more routines which increases the >> fixed memory requirement per procs. However, the grid size is still the >> same. But the code aborts while solving the Poisson eqn, saying: >> >> Out of memory trying to allocate XXX bytes >> >> I'm using PETSc with HYPRE boomeramg to solve the linear Poisson eqn. I >> am guessing that now the amt of memory per procs is less because I added >> some routines which uses some memory. The result is less memory available >> for the solving of the Poisson eqn. >> > > I would try GAMG instead of Hypre. It tends to be memory light compared > to it. > > -pc_type gamg > > Thanks, > > Matt > > Hi Matt, > > To use gamg, must I use use DMDA to partition the grid? > No. > Also, does MPI partitioning in only the z direction affect parallel > performance? Since the MPI partition grid is almost like 2D plane with 3-4 > cell thickness. > It could affect load balance. > Lastly, using 10x10 = 100 procs seems to work for now, although there's a > wastage of 20 procs since each node has 12 procs. > PETSc attempts to make square domains, but you can prescribe your own domains if that helps. Thanks, Matt > Thanks! > > > >> I'm now changing to KSPBCGS but it seems to take forever. When I abort >> it, the error msg is: >> >> Out of memory. This could be due to allocating >> [10]PETSC ERROR: too large an object or bleeding by not properly >> [10]PETSC ERROR: destroying unneeded objects. >> [10]PETSC ERROR: Memory allocated 0 Memory used by process 4028370944 >> [10]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. >> >> I can't use more procs because some procs will have a size of 231 x 461 x >> 2 (or even 1). This will give error since I need to reference the nearby >> values along the z direction. >> >> So what options do I have? I'm thinking of these at the moment: >> >> 1. Remove as much fixed overhead memory per procs as possible so that >> there's enough memory for each procs. >> >> 2. Re-partition my grid in both x,y direction or x,y,z direction so I >> will not encounter extremely skew grid dimensions per procs. Btw, does >> having extremely skew grid dimensions affect the performance in solving the >> linear eqn? >> >> Are there other feasible options >> >> -- >> Thank you. >> >> Yours sincerely, >> >> TAY wee-beng >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
