Hi Andreas,
That's great info. So what I am seeing is more than two times (13.6GB versus
5.7GB) of your estimate?
I am using the standard Cartesian grid used by boxmodel tests.
-Shawn
From: Andreas Lauser <[email protected]>
To: Shawn Zhang <[email protected]>; [email protected]
Sent: Tuesday, April 3, 2012 7:23 AM
Subject: Re: Memory requirement for 500x500x500 dumux run
Hi Shawn,
Thanks for your results, I found them quite interesting. what kind of grid do
you use? unstructured grids like ALU or UG require a lot more memory than the
cartesian grids like YaspGrid.
for the memory, my detailed estimate for the mem besides the grid would be the
following:
- 27 * N * 8 bytes for the contents of the matrix (N=number of vertices,
27=number of neighbors in 3D, 8=sizeof(Scalar))
- 28 * N * 4 bytes for storing the sparsity pattern of the matrix
- currently dumux copies the matrix before the linear solve so the total size
for storing the matrices would be 27*N*24 + 2*N*4 bytes
- additionally, there are approximately 7 vectors (but that's just my gut-
feeling) which store solutions, i.e. 7*N*8 bytes
- for a 200x200x200 grid N is 8M, so the total would be at approximately 5.7
GB.
cheers
Andreas
On Montag, 2. April 2012, 13:33:18 you wrote:
> In one of the earlier discussions with Andreas, we touched upon how much
> memory is necessary for a 500x500x500 run.
> Since we have a scaling problem with parallel solver, I abadoned that route
> for now (waiting for later development).
> With 1p box problems, here is a simple estimate,
>
> 100 cube, requires about 1.7GB memory.
>
> Memory scales pretty linearly with number of element, so,
>
> 200 cube, we need about 1.7x8 = 13.6GB memory (which is roughtly the largest
> problem I can run before my workstation ran out of memory). 500 cube, we
> need about 1.7x125 = 212.5GB memory
>
> For parallel run and more complex models, the memory requirement will be
> larger -- to be tested.
> I hope this will be helpful for some of you.
>
> Best,
> -Shawn
>
>
> From: Andreas Lauser <[email protected]>
> To: Shawn Zhang <[email protected]>
> Sent: Tuesday, March 20, 2012 11:37 AM
> Subject: Re: Parallel dumux
>
> Hi Shawn,
>
> On Dienstag, 20. März 2012 you wrote:
> > Hi Andreas,
> >
> >
> >
> > I just plan to use cartisian grid, however, I will need both 1p and 2p
> > model to the minimum. Do you think your patch can work for both cases? If
> > so, it will definitely be appreciated.
>
> well, for the 1p model most time is spend when calculating the finite volume
> gemetries for the elements and the resulting linearization is symmetric if
> the fluid is incompressible which allows to use conjugated gradients
> instead of Bi- conjugate gradients as linear solver.
>
> for the 2p model it's a different story: here, evaluating the local jacobian
> is quite a bit more expensive because that comes with a cost of
> O((numPrimaryVars*numVerticesInElement)^2) and also the material laws are
> more expensive. This means that you gain almost nothing if you hack your
> way around recalculating the finite volume geometries (which can be done
> assuming that all elements look the same). Also for 2p, the resulting
> linearization is always unsymmetric, which means that a more expensive
> linear solver has to be used in any case...
>
> > I am also facinated by your memory estimate.
> >
> >
> >
> > So in my failed case, I have 100x100x100 = 1M vertices. If these vertices
> > are in 2D, with my 4 core machine, I should be able to handle 40M (10M per
> > core)?
>
> yes, but there is some duplication between the cores. usually, the grid is
> initially present on all cores and later distributed. I might be wrong here,
> though...
>
> > How come this constraint is related to number of cores, not memory? I must
> > have misunderstood something.
> > My plan is to get a 500x500x500 grid, 2p model running on a 8-core, 12GB
> > RAM machine. What is your rough estimate on memory requirement and
> > computational time?
>
> actually, I don't know. You should probably find out yourself and inform me
> about the results ;). 500x500x500 sounds quite too large for a 2p model,
> though. (even if you get this into memory, you will probably run into
> convergence problems for the linear solver and the time steps will also
> probably be quite small.)
>
> cheers
> Andreas
>
> > Again, any advices/inputs will be greatly appreciated.
> >
> >
> >
> > -Shawn
> >
> >
> >
> >
> > From: Andreas Lauser <[email protected]>
> > To: Shawn Zhang <[email protected]>
> > Sent: Thursday, March 15, 2012 5:42 AM
> > Subject: Re: Parallel dumux
> >
> > Hi Shawn,
> >
> > I can confirm points 1 and 2. The second issue is probably due to the fact
> > that the overlap is limited the boundary of the sub-domains. We can't
> > increase it though, since we have a bug with larger overlaps. I will have
> > to look into this in the near to medium term future...
> >
> > The third issue is strange. I suppose there might be a memory leak
> > somewhere in the linear algebra infrastructure. How many vertices do you
> > assign to each Core? I think for the isothermal single-phase model you
> > should be able to go up to roughly 10 megavertices per Core on a 2D grid
> > on your laptop (on 3D grids the number is lower) before you should run out
> > of memory. Another possibility is that the MPI library leaks some memory.
> > Which one are you using? Also, do you intend to only use cartesian grids
> > and incompressible fluids with the 1p model? If yes, I can prepare a small
> > patch for you which speeds things up by about factor 5...
> >
> > On a different note, I would recomend you to run your simulations natively
> > on linux if you care about performance. If you use "VirtualBox" as your
> > hypervisor, it slows you down by appoximately factor 2. Also, try to use a
> > recent compiler (i.e. GCC 4.6 or newer) since you get decent speedups just
> > by using them.
> >
> > cheers
> >
> > Andreas
> >
> > On Monday, March 12, 2012 08:33:31 PM you wrote:
> >
> > Hi Andreas,
> >
> > After upgrade and patch, I got the following performance results on some
> > test runs. In summary,
> >
> > 1. Parallel solver runs, and the results appears to be correct.
> > 2. I observed very bad scaling.
> > 3. when problem size is big, the parallel solver crashed. This may due to
> > memory limitation. I am using a vbox Debian virtual machine on a Win7
> > laptop, with 4 CPU and 4GB allocated for the virtual machine.
> >
> > Any suggestions on the latter two issues?
> >
> > ****************
> > Dumux performance:
> >
> > 1p, 3D, 40x40x40,
> > # CPUs, Time
> > 1, 74 sec
> > 2, 91sec
> > 4, 109sec
> >
> > 1p, 3D, 100x100x100
> > 1, 1567.9
> > 2, with the following error,
> > Welcome aboard DuMuX airlines. Please fasten your seatbelts! Emergency
> > exits are near the time integration.
> > Initializing problem '1ptest'
> > Writing result file for "1ptest"
> > Writing result file for "1ptest"
> > Assemble: r(x^k) = dS/dt + div F - q; M = grad rAssemble: r(x^k) = dS/dt
> > +
> > divSolve: M deltax^k =
> > r-------------------------------------------------------------------------
> > -
> > mpirun noticed that process rank 0 with PID 30539 on node szdebian exited
> > on signal 9 (Killed).
> > --------------------------------------------------------------------------
> >
> >
> >
> > From: Andreas Lauser <[email protected]>
> > To: Shawn Zhang <[email protected]>
> > Sent: Thursday, March 8, 2012 12:13 PM
> > Subject: Re: Parallel dumux
> >
> > > By the way, this patch works with v2.1, right?
> >
> > yep.
> >
> > > I am still on v2.0.1, so I need to upgrade first.
> >
> > If you do not have deep changes like custom models or fluid systems
> > implemended on top of 2.0.1, I would strongly recommend to update since
> > there are a _lot_ of improvments in the new release. If I did not forget
> > something, it is possible to just copy your old problems. (After changing
> > the way how the type of the material-laws is picked by the spatial
> > parameter class, but that's just a excercise in copy-and-paste.)
> >
> > have fun
> >
> > Andreas
> >
> > > From: Andreas Lauser <[email protected]>
> > > To: Shawn Zhang <[email protected]>
> > > Sent: Thursday, March 8, 2012 11:13 AM
> > > Subject: Parallel dumux
> > >
> > > Hi Shawn,
> > >
> > > could you be so kind and retry parallel execution with the latest Dumux
> > > trunk version? I've found and fixed a bug when loading grids from DGF
> > > files. (which most test problems do.)
> > >
> > > Alternatively, you can also go to the Dumux root directory and apply the
> > > attached patch using
> > >
> > > patch -p1 < $PATH_TO_PATCH_FILE
> > >
> > > cheers
> > >
> > > Andreas
--
Andreas Lauser
Department of Hydromechanics and Modelling of Hydrosystems
University of Stuttgart
Pfaffenwaldring 61
D-70569 Stuttgart
Phone: (+49) 711 685-64719
Fax: (+49) 711 685-60430
www.hydrosys.uni-stuttgart.de
_______________________________________________
Dumux mailing list
[email protected]
https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
