Hi Shawn, Thanks for your results, I found them quite interesting. what kind of grid do you use? unstructured grids like ALU or UG require a lot more memory than the cartesian grids like YaspGrid.
for the memory, my detailed estimate for the mem besides the grid would be the following: - 27 * N * 8 bytes for the contents of the matrix (N=number of vertices, 27=number of neighbors in 3D, 8=sizeof(Scalar)) - 28 * N * 4 bytes for storing the sparsity pattern of the matrix - currently dumux copies the matrix before the linear solve so the total size for storing the matrices would be 27*N*24 + 2*N*4 bytes - additionally, there are approximately 7 vectors (but that's just my gut- feeling) which store solutions, i.e. 7*N*8 bytes - for a 200x200x200 grid N is 8M, so the total would be at approximately 5.7 GB. cheers Andreas On Montag, 2. April 2012, 13:33:18 you wrote: > In one of the earlier discussions with Andreas, we touched upon how much > memory is necessary for a 500x500x500 run. > Since we have a scaling problem with parallel solver, I abadoned that route > for now (waiting for later development). > With 1p box problems, here is a simple estimate, > > 100 cube, requires about 1.7GB memory. > > Memory scales pretty linearly with number of element, so, > > 200 cube, we need about 1.7x8 = 13.6GB memory (which is roughtly the largest > problem I can run before my workstation ran out of memory). 500 cube, we > need about 1.7x125 = 212.5GB memory > > For parallel run and more complex models, the memory requirement will be > larger -- to be tested. > I hope this will be helpful for some of you. > > Best, > -Shawn > > > From: Andreas Lauser <[email protected]> > To: Shawn Zhang <[email protected]> > Sent: Tuesday, March 20, 2012 11:37 AM > Subject: Re: Parallel dumux > > Hi Shawn, > > On Dienstag, 20. März 2012 you wrote: > > Hi Andreas, > > > > > > > > I just plan to use cartisian grid, however, I will need both 1p and 2p > > model to the minimum. Do you think your patch can work for both cases? If > > so, it will definitely be appreciated. > > well, for the 1p model most time is spend when calculating the finite volume > gemetries for the elements and the resulting linearization is symmetric if > the fluid is incompressible which allows to use conjugated gradients > instead of Bi- conjugate gradients as linear solver. > > for the 2p model it's a different story: here, evaluating the local jacobian > is quite a bit more expensive because that comes with a cost of > O((numPrimaryVars*numVerticesInElement)^2) and also the material laws are > more expensive. This means that you gain almost nothing if you hack your > way around recalculating the finite volume geometries (which can be done > assuming that all elements look the same). Also for 2p, the resulting > linearization is always unsymmetric, which means that a more expensive > linear solver has to be used in any case... > > > I am also facinated by your memory estimate. > > > > > > > > So in my failed case, I have 100x100x100 = 1M vertices. If these vertices > > are in 2D, with my 4 core machine, I should be able to handle 40M (10M per > > core)? > > yes, but there is some duplication between the cores. usually, the grid is > initially present on all cores and later distributed. I might be wrong here, > though... > > > How come this constraint is related to number of cores, not memory? I must > > have misunderstood something. > > My plan is to get a 500x500x500 grid, 2p model running on a 8-core, 12GB > > RAM machine. What is your rough estimate on memory requirement and > > computational time? > > actually, I don't know. You should probably find out yourself and inform me > about the results ;). 500x500x500 sounds quite too large for a 2p model, > though. (even if you get this into memory, you will probably run into > convergence problems for the linear solver and the time steps will also > probably be quite small.) > > cheers > Andreas > > > Again, any advices/inputs will be greatly appreciated. > > > > > > > > -Shawn > > > > > > > > > > From: Andreas Lauser <[email protected]> > > To: Shawn Zhang <[email protected]> > > Sent: Thursday, March 15, 2012 5:42 AM > > Subject: Re: Parallel dumux > > > > Hi Shawn, > > > > I can confirm points 1 and 2. The second issue is probably due to the fact > > that the overlap is limited the boundary of the sub-domains. We can't > > increase it though, since we have a bug with larger overlaps. I will have > > to look into this in the near to medium term future... > > > > The third issue is strange. I suppose there might be a memory leak > > somewhere in the linear algebra infrastructure. How many vertices do you > > assign to each Core? I think for the isothermal single-phase model you > > should be able to go up to roughly 10 megavertices per Core on a 2D grid > > on your laptop (on 3D grids the number is lower) before you should run out > > of memory. Another possibility is that the MPI library leaks some memory. > > Which one are you using? Also, do you intend to only use cartesian grids > > and incompressible fluids with the 1p model? If yes, I can prepare a small > > patch for you which speeds things up by about factor 5... > > > > On a different note, I would recomend you to run your simulations natively > > on linux if you care about performance. If you use "VirtualBox" as your > > hypervisor, it slows you down by appoximately factor 2. Also, try to use a > > recent compiler (i.e. GCC 4.6 or newer) since you get decent speedups just > > by using them. > > > > cheers > > > > Andreas > > > > On Monday, March 12, 2012 08:33:31 PM you wrote: > > > > Hi Andreas, > > > > After upgrade and patch, I got the following performance results on some > > test runs. In summary, > > > > 1. Parallel solver runs, and the results appears to be correct. > > 2. I observed very bad scaling. > > 3. when problem size is big, the parallel solver crashed. This may due to > > memory limitation. I am using a vbox Debian virtual machine on a Win7 > > laptop, with 4 CPU and 4GB allocated for the virtual machine. > > > > Any suggestions on the latter two issues? > > > > **************** > > Dumux performance: > > > > 1p, 3D, 40x40x40, > > # CPUs, Time > > 1, 74 sec > > 2, 91sec > > 4, 109sec > > > > 1p, 3D, 100x100x100 > > 1, 1567.9 > > 2, with the following error, > > Welcome aboard DuMuX airlines. Please fasten your seatbelts! Emergency > > exits are near the time integration. > > Initializing problem '1ptest' > > Writing result file for "1ptest" > > Writing result file for "1ptest" > > Assemble: r(x^k) = dS/dt + div F - q; M = grad rAssemble: r(x^k) = dS/dt > > + > > divSolve: M deltax^k = > > r------------------------------------------------------------------------- > > - > > mpirun noticed that process rank 0 with PID 30539 on node szdebian exited > > on signal 9 (Killed). > > -------------------------------------------------------------------------- > > > > > > > > From: Andreas Lauser <[email protected]> > > To: Shawn Zhang <[email protected]> > > Sent: Thursday, March 8, 2012 12:13 PM > > Subject: Re: Parallel dumux > > > > > By the way, this patch works with v2.1, right? > > > > yep. > > > > > I am still on v2.0.1, so I need to upgrade first. > > > > If you do not have deep changes like custom models or fluid systems > > implemended on top of 2.0.1, I would strongly recommend to update since > > there are a _lot_ of improvments in the new release. If I did not forget > > something, it is possible to just copy your old problems. (After changing > > the way how the type of the material-laws is picked by the spatial > > parameter class, but that's just a excercise in copy-and-paste.) > > > > have fun > > > > Andreas > > > > > From: Andreas Lauser <[email protected]> > > > To: Shawn Zhang <[email protected]> > > > Sent: Thursday, March 8, 2012 11:13 AM > > > Subject: Parallel dumux > > > > > > Hi Shawn, > > > > > > could you be so kind and retry parallel execution with the latest Dumux > > > trunk version? I've found and fixed a bug when loading grids from DGF > > > files. (which most test problems do.) > > > > > > Alternatively, you can also go to the Dumux root directory and apply the > > > attached patch using > > > > > > patch -p1 < $PATH_TO_PATCH_FILE > > > > > > cheers > > > > > > Andreas -- Andreas Lauser Department of Hydromechanics and Modelling of Hydrosystems University of Stuttgart Pfaffenwaldring 61 D-70569 Stuttgart Phone: (+49) 711 685-64719 Fax: (+49) 711 685-60430 www.hydrosys.uni-stuttgart.de _______________________________________________ Dumux mailing list [email protected] https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
