In one of the earlier discussions with Andreas, we touched upon how much memory
is necessary for a 500x500x500 run.
Since we have a scaling problem with parallel solver, I abadoned that route for
now (waiting for later development).
With 1p box problems, here is a simple estimate,
100 cube, requires about 1.7GB memory.
Memory scales pretty linearly with number of element, so,
200 cube, we need about 1.7x8 = 13.6GB memory (which is roughtly the largest
problem I can run before my workstation ran out of memory).
500 cube, we need about 1.7x125 = 212.5GB memory
For parallel run and more complex models, the memory requirement will be larger
-- to be tested.
I hope this will be helpful for some of you.
Best,
-Shawn
From: Andreas Lauser <[email protected]>
To: Shawn Zhang <[email protected]>
Sent: Tuesday, March 20, 2012 11:37 AM
Subject: Re: Parallel dumux
Hi Shawn,
On Dienstag, 20. März 2012 you wrote:
> Hi Andreas,
>
> I just plan to use cartisian grid, however, I will need both 1p and 2p
> model to the minimum. Do you think your patch can work for both cases? If
> so, it will definitely be appreciated.
well, for the 1p model most time is spend when calculating the finite volume
gemetries for the elements and the resulting linearization is symmetric if the
fluid is incompressible which allows to use conjugated gradients instead of Bi-
conjugate gradients as linear solver.
for the 2p model it's a different story: here, evaluating the local jacobian is
quite a bit more expensive because that comes with a cost of
O((numPrimaryVars*numVerticesInElement)^2) and also the material laws are more
expensive. This means that you gain almost nothing if you hack your way
around recalculating the finite volume geometries (which can be done assuming
that all elements look the same). Also for 2p, the resulting linearization is
always unsymmetric, which means that a more expensive linear solver has to be
used in any case...
> I am also facinated by your memory estimate.
>
> So in my failed case, I have 100x100x100 = 1M vertices. If these vertices
> are in 2D, with my 4 core machine, I should be able to handle 40M (10M per
> core)?
yes, but there is some duplication between the cores. usually, the grid is
initially present on all cores and later distributed. I might be wrong here,
though...
> How come this constraint is related to number of cores, not memory? I must
> have misunderstood something.
> My plan is to get a 500x500x500 grid, 2p model running on a 8-core, 12GB
> RAM machine. What is your rough estimate on memory requirement and
> computational time?
actually, I don't know. You should probably find out yourself and inform me
about the results ;). 500x500x500 sounds quite too large for a 2p model,
though. (even if you get this into memory, you will probably run into
convergence problems for the linear solver and the time steps will also
probably be quite small.)
cheers
Andreas
> Again, any advices/inputs will be greatly appreciated.
>
> -Shawn
>
>
> From: Andreas Lauser <[email protected]>
> To: Shawn Zhang <[email protected]>
> Sent: Thursday, March 15, 2012 5:42 AM
> Subject: Re: Parallel dumux
>
> Hi Shawn,
>
> I can confirm points 1 and 2. The second issue is probably due to the fact
> that the overlap is limited the boundary of the sub-domains. We can't
> increase it though, since we have a bug with larger overlaps. I will have
> to look into this in the near to medium term future...
>
> The third issue is strange. I suppose there might be a memory leak
> somewhere in the linear algebra infrastructure. How many vertices do you
> assign to each Core? I think for the isothermal single-phase model you
> should be able to go up to roughly 10 megavertices per Core on a 2D grid
> on your laptop (on 3D grids the number is lower) before you should run out
> of memory. Another possibility is that the MPI library leaks some memory.
> Which one are you using? Also, do you intend to only use cartesian grids
> and incompressible fluids with the 1p model? If yes, I can prepare a small
> patch for you which speeds things up by about factor 5...
>
> On a different note, I would recomend you to run your simulations natively
> on linux if you care about performance. If you use "VirtualBox" as your
> hypervisor, it slows you down by appoximately factor 2. Also, try to use a
> recent compiler (i.e. GCC 4.6 or newer) since you get decent speedups just
> by using them.
>
> cheers
> Andreas
>
> On Monday, March 12, 2012 08:33:31 PM you wrote:
>
> Hi Andreas,
>
> After upgrade and patch, I got the following performance results on some
> test runs. In summary,
>
> 1. Parallel solver runs, and the results appears to be correct.
> 2. I observed very bad scaling.
> 3. when problem size is big, the parallel solver crashed. This may due to
> memory limitation. I am using a vbox Debian virtual machine on a Win7
> laptop, with 4 CPU and 4GB allocated for the virtual machine.
>
> Any suggestions on the latter two issues?
>
> ****************
> Dumux performance:
>
> 1p, 3D, 40x40x40,
> # CPUs, Time
> 1, 74 sec
> 2, 91sec
> 4, 109sec
>
> 1p, 3D, 100x100x100
> 1, 1567.9
> 2, with the following error,
> Welcome aboard DuMuX airlines. Please fasten your seatbelts! Emergency
> exits are near the time integration.
> Initializing problem '1ptest'
> Writing result file for "1ptest"
> Writing result file for "1ptest"
> Assemble: r(x^k) = dS/dt + div F - q; M = grad rAssemble: r(x^k) = dS/dt +
> divSolve: M deltax^k =
> r--------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 30539 on node szdebian exited
> on signal 9 (Killed).
> --------------------------------------------------------------------------
>
>
>
> From: Andreas Lauser <[email protected]>
> To: Shawn Zhang <[email protected]>
> Sent: Thursday, March 8, 2012 12:13 PM
> Subject: Re: Parallel dumux
>
> > By the way, this patch works with v2.1, right?
>
> yep.
>
> > I am still on v2.0.1, so I need to upgrade first.
>
> If you do not have deep changes like custom models or fluid systems
> implemended on top of 2.0.1, I would strongly recommend to update since
> there are a _lot_ of improvments in the new release. If I did not forget
> something, it is possible to just copy your old problems. (After changing
> the way how the type of the material-laws is picked by the spatial
> parameter class, but that's just a excercise in copy-and-paste.)
>
> have fun
> Andreas
>
> > From: Andreas Lauser <[email protected]>
> > To: Shawn Zhang <[email protected]>
> > Sent: Thursday, March 8, 2012 11:13 AM
> > Subject: Parallel dumux
> >
> > Hi Shawn,
> >
> > could you be so kind and retry parallel execution with the latest Dumux
> > trunk version? I've found and fixed a bug when loading grids from DGF
> > files. (which most test problems do.)
> >
> > Alternatively, you can also go to the Dumux root directory and apply the
> > attached patch using
> >
> > patch -p1 < $PATH_TO_PATCH_FILE
> >
> > cheers
> >
> > Andreas
--
Andreas Lauser
Department of Hydromechanics and Modelling of Hydrosystems
University of Stuttgart
Pfaffenwaldring 61
D-70569 Stuttgart
Phone: (+49) 711 685-64719
Fax: (+49) 711 685-60430
www.hydrosys.uni-stuttgart.de
_______________________________________________
Dumux mailing list
[email protected]
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
