Hi Shawn, yes, maybe the linear solver (or the preconditioner) creates additional copies of the global jacobian matrix. Since I suppose that the DUNE-ISTL solvers have never really been tested with constraint memory, that might be the case. Do you see memory consumption go up and down during the simulation? If yes this would be a hint in this direction...
cheers Andreas Am Dienstag, 3. April 2012, 06:20:32 schrieb Shawn Zhang: > Hi Andreas, > > That's great info. So what I am seeing is more than two times (13.6GB versus > 5.7GB) of your estimate? > I am using the standard Cartesian grid used by boxmodel tests. > > -Shawn > > From: Andreas Lauser <[email protected]> > To: Shawn Zhang <[email protected]>; [email protected] > Sent: Tuesday, April 3, 2012 7:23 AM > Subject: Re: Memory requirement for 500x500x500 dumux run > > Hi Shawn, > > Thanks for your results, I found them quite interesting. what kind of grid > do you use? unstructured grids like ALU or UG require a lot more memory > than the cartesian grids like YaspGrid. > > for the memory, my detailed estimate for the mem besides the grid would be > the following: > > - 27 * N * 8 bytes for the contents of the matrix (N=number of vertices, > 27=number of neighbors in 3D, 8=sizeof(Scalar)) > - 28 * N * 4 bytes for storing the sparsity pattern of the matrix > - currently dumux copies the matrix before the linear solve so the total > size for storing the matrices would be 27*N*24 + 2*N*4 bytes > - additionally, there are approximately 7 vectors (but that's just my gut- > feeling) which store solutions, i.e. 7*N*8 bytes > - for a 200x200x200 grid N is 8M, so the total would be at approximately 5.7 > GB. > > cheers > Andreas > > On Montag, 2. April 2012, 13:33:18 you wrote: > > In one of the earlier discussions with Andreas, we touched upon how much > > memory is necessary for a 500x500x500 run. > > Since we have a scaling problem with parallel solver, I abadoned that > > route > > for now (waiting for later development). > > With 1p box problems, here is a simple estimate, > > > > > > > > 100 cube, requires about 1.7GB memory. > > > > > > > > Memory scales pretty linearly with number of element, so, > > > > > > > > 200 cube, we need about 1.7x8 = 13.6GB memory (which is roughtly the > > largest problem I can run before my workstation ran out of memory). 500 > > cube, we need about 1.7x125 = 212.5GB memory > > > > > > > > For parallel run and more complex models, the memory requirement will be > > larger -- to be tested. > > I hope this will be helpful for some of you. > > > > > > > > Best, > > -Shawn > > > > > > > > From: Andreas Lauser <[email protected]> > > To: Shawn Zhang <[email protected]> > > Sent: Tuesday, March 20, 2012 11:37 AM > > Subject: Re: Parallel dumux > > > > Hi Shawn, > > > > On Dienstag, 20. März 2012 you wrote: > > > Hi Andreas, > > > > > > > > > > > > I just plan to use cartisian grid, however, I will need both 1p and 2p > > > model to the minimum. Do you think your patch can work for both cases? > > > If > > > so, it will definitely be appreciated. > > > > well, for the 1p model most time is spend when calculating the finite > > volume gemetries for the elements and the resulting linearization is > > symmetric if the fluid is incompressible which allows to use conjugated > > gradients instead of Bi- conjugate gradients as linear solver. > > > > for the 2p model it's a different story: here, evaluating the local > > jacobian is quite a bit more expensive because that comes with a cost of > > O((numPrimaryVars*numVerticesInElement)^2) and also the material laws are > > more expensive. This means that you gain almost nothing if you hack your > > way around recalculating the finite volume geometries (which can be done > > assuming that all elements look the same). Also for 2p, the resulting > > linearization is always unsymmetric, which means that a more expensive > > linear solver has to be used in any case... > > > > > I am also facinated by your memory estimate. > > > > > > > > > > > > So in my failed case, I have 100x100x100 = 1M vertices. If these > > > vertices > > > are in 2D, with my 4 core machine, I should be able to handle 40M (10M > > > per > > > core)? > > > > yes, but there is some duplication between the cores. usually, the grid is > > initially present on all cores and later distributed. I might be wrong > > here, though... > > > > > How come this constraint is related to number of cores, not memory? I > > > must > > > have misunderstood something. > > > My plan is to get a 500x500x500 grid, 2p model running on a 8-core, 12GB > > > RAM machine. What is your rough estimate on memory requirement and > > > computational time? > > > > actually, I don't know. You should probably find out yourself and inform > > me > > about the results ;). 500x500x500 sounds quite too large for a 2p model, > > though. (even if you get this into memory, you will probably run into > > convergence problems for the linear solver and the time steps will also > > probably be quite small.) > > > > cheers > > > > Andreas > > > > > Again, any advices/inputs will be greatly appreciated. > > > > > > > > > > > > -Shawn > > > > > > > > > > > > > > > From: Andreas Lauser <[email protected]> > > > To: Shawn Zhang <[email protected]> > > > Sent: Thursday, March 15, 2012 5:42 AM > > > Subject: Re: Parallel dumux > > > > > > Hi Shawn, > > > > > > I can confirm points 1 and 2. The second issue is probably due to the > > > fact > > > that the overlap is limited the boundary of the sub-domains. We can't > > > increase it though, since we have a bug with larger overlaps. I will > > > have > > > to look into this in the near to medium term future... > > > > > > The third issue is strange. I suppose there might be a memory leak > > > somewhere in the linear algebra infrastructure. How many vertices do you > > > assign to each Core? I think for the isothermal single-phase model you > > > should be able to go up to roughly 10 megavertices per Core on a 2D grid > > > on your laptop (on 3D grids the number is lower) before you should run > > > out > > > of memory. Another possibility is that the MPI library leaks some > > > memory. > > > Which one are you using? Also, do you intend to only use cartesian grids > > > and incompressible fluids with the 1p model? If yes, I can prepare a > > > small > > > patch for you which speeds things up by about factor 5... > > > > > > On a different note, I would recomend you to run your simulations > > > natively > > > on linux if you care about performance. If you use "VirtualBox" as your > > > hypervisor, it slows you down by appoximately factor 2. Also, try to use > > > a > > > recent compiler (i.e. GCC 4.6 or newer) since you get decent speedups > > > just > > > by using them. > > > > > > cheers > > > > > > Andreas > > > > > > On Monday, March 12, 2012 08:33:31 PM you wrote: > > > > > > Hi Andreas, > > > > > > After upgrade and patch, I got the following performance results on some > > > test runs. In summary, > > > > > > 1. Parallel solver runs, and the results appears to be correct. > > > 2. I observed very bad scaling. > > > 3. when problem size is big, the parallel solver crashed. This may due > > > to > > > memory limitation. I am using a vbox Debian virtual machine on a Win7 > > > laptop, with 4 CPU and 4GB allocated for the virtual machine. > > > > > > Any suggestions on the latter two issues? > > > > > > **************** > > > Dumux performance: > > > > > > 1p, 3D, 40x40x40, > > > # CPUs, Time > > > 1, 74 sec > > > 2, 91sec > > > 4, 109sec > > > > > > 1p, 3D, 100x100x100 > > > 1, 1567.9 > > > 2, with the following error, > > > Welcome aboard DuMuX airlines. Please fasten your seatbelts! Emergency > > > exits are near the time integration. > > > Initializing problem '1ptest' > > > Writing result file for "1ptest" > > > Writing result file for "1ptest" > > > Assemble: r(x^k) = dS/dt + div F - q; M = grad rAssemble: r(x^k) = > > > dS/dt > > > + > > > divSolve: M deltax^k = > > > r----------------------------------------------------------------------- > > > -- > > > - > > > mpirun noticed that process rank 0 with PID 30539 on node szdebian > > > exited > > > on signal 9 (Killed). > > > ------------------------------------------------------------------------ > > > -- > > > > > > > > > > > > From: Andreas Lauser <[email protected]> > > > To: Shawn Zhang <[email protected]> > > > Sent: Thursday, March 8, 2012 12:13 PM > > > Subject: Re: Parallel dumux > > > > > > > By the way, this patch works with v2.1, right? > > > > > > yep. > > > > > > > I am still on v2.0.1, so I need to upgrade first. > > > > > > If you do not have deep changes like custom models or fluid systems > > > implemended on top of 2.0.1, I would strongly recommend to update since > > > there are a _lot_ of improvments in the new release. If I did not forget > > > something, it is possible to just copy your old problems. (After > > > changing > > > the way how the type of the material-laws is picked by the spatial > > > parameter class, but that's just a excercise in copy-and-paste.) > > > > > > have fun > > > > > > Andreas > > > > > > > From: Andreas Lauser <[email protected]> > > > > To: Shawn Zhang <[email protected]> > > > > Sent: Thursday, March 8, 2012 11:13 AM > > > > Subject: Parallel dumux > > > > > > > > Hi Shawn, > > > > > > > > could you be so kind and retry parallel execution with the latest > > > > Dumux > > > > trunk version? I've found and fixed a bug when loading grids from DGF > > > > files. (which most test problems do.) > > > > > > > > Alternatively, you can also go to the Dumux root directory and apply > > > > the > > > > attached patch using > > > > > > > > patch -p1 < $PATH_TO_PATCH_FILE > > > > > > > > cheers > > > > > > > > Andreas -- Andreas Lauser Department of Hydromechanics and Modelling of Hydrosystems University of Stuttgart Pfaffenwaldring 61 D-70569 Stuttgart Phone: (+49) 711 685-64719 Fax: (+49) 711 685-60430 www.hydrosys.uni-stuttgart.de _______________________________________________ Dumux mailing list [email protected] https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
