Thanks for your suggestions! If it's just 2X, I will not waste my time! Hom Nath
On Fri, Jan 22, 2016 at 5:06 PM, Matthew Knepley <[email protected]> wrote: > On Fri, Jan 22, 2016 at 3:47 PM, Hom Nath Gharti <[email protected]> > wrote: >> >> Hi Matt, >> >> SPECFEM currently has only an explicit time scheme and does not have >> full gravity implemented. I am adding implicit time scheme and full >> gravity so that it can be used for interesting quasistatic problems >> such as glacial rebound, post seismic relaxation etc. I am using Petsc >> as a linear solver which I would like to see GPU implemented. > > > Why? It really does not make sense for those operations. > > It is an unfortunate fact, but the usefulness of GPUs has been oversold. You > can certainly > get some mileage out of a SpMV on the GPU, but there the maximum win is > maybe 2x or > less for a nice CPU, and then you have to account for transfer time and > other latencies. > Unless you have a really compelling case, I would not waste your time. > > To come to this opinion, I used years of my own time looking at GPUs. > > Thanks, > > Matt > >> >> Thanks, >> Hom Nath >> >> On Fri, Jan 22, 2016 at 4:33 PM, Matthew Knepley <[email protected]> >> wrote: >> > On Fri, Jan 22, 2016 at 12:17 PM, Hom Nath Gharti <[email protected]> >> > wrote: >> >> >> >> Thanks Matt for great suggestion. One last question, do you know >> >> whether the GPU capability of current PETSC version is matured enough >> >> to try for my problem? >> > >> > >> > The only thing that would really make sense to do on the GPU is the SEM >> > integration, which >> > would not be part of PETSc. This is what SPECFEM has optimized. >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks again for your help. >> >> Hom Nath >> >> >> >> On Fri, Jan 22, 2016 at 1:07 PM, Matthew Knepley <[email protected]> >> >> wrote: >> >> > On Fri, Jan 22, 2016 at 11:47 AM, Hom Nath Gharti >> >> > <[email protected]> >> >> > wrote: >> >> >> >> >> >> Thanks a lot. >> >> >> >> >> >> With AMG it did not converge within the iteration limit of 3000. >> >> >> >> >> >> In solid: elastic wave equation with added gravity term \rho >> >> >> \nabla\phi >> >> >> In fluid: acoustic wave equation with added gravity term \rho >> >> >> \nabla\phi >> >> >> Both solid and fluid: Poisson's equation for gravity >> >> >> Outer space: Laplace's equation for gravity >> >> >> >> >> >> We combine so called mapped infinite element with spectral-element >> >> >> method (higher order FEM that uses nodal quadrature) and solve in >> >> >> frequency domain. >> >> > >> >> > >> >> > 1) The Poisson and Laplace equation should be using MG, however you >> >> > are >> >> > using SEM, so >> >> > you would need to use a low order PC for the high order problem, >> >> > also >> >> > called p-MG (Paul Fischer), see >> >> > >> >> > http://epubs.siam.org/doi/abs/10.1137/110834512 >> >> > >> >> > 2) The acoustic wave equation is Helmholtz to us, and that needs >> >> > special >> >> > MG >> >> > tweaks that >> >> > are still research material so I can understand using ASM. >> >> > >> >> > 3) Same thing for the elastic wave equations. Some people say they >> >> > have >> >> > this >> >> > solved using >> >> > hierarchical matrix methods, something like >> >> > >> >> > http://portal.nersc.gov/project/sparse/strumpack/ >> >> > >> >> > However, I think the jury is still out. >> >> > >> >> > If you can do 100 iterations of plain vanilla solvers, that seems >> >> > like a >> >> > win >> >> > right now. You might improve >> >> > the time using FS, but I am not sure about the iterations on the >> >> > smaller >> >> > problem. >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> Hom Nath >> >> >> >> >> >> On Fri, Jan 22, 2016 at 12:16 PM, Matthew Knepley >> >> >> <[email protected]> >> >> >> wrote: >> >> >> > On Fri, Jan 22, 2016 at 11:10 AM, Hom Nath Gharti >> >> >> > <[email protected]> >> >> >> > wrote: >> >> >> >> >> >> >> >> Thanks Matt. >> >> >> >> >> >> >> >> Attached detailed info on ksp of a much smaller test. This is a >> >> >> >> multiphysics problem. >> >> >> > >> >> >> > >> >> >> > You are using FGMRES/ASM(ILU0). From your description below, this >> >> >> > sounds >> >> >> > like >> >> >> > an elliptic system. I would at least try AMG (-pc_type gamg) to >> >> >> > see >> >> >> > how >> >> >> > it >> >> >> > does. Any >> >> >> > other advice would have to be based on seeing the equations. >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Matt >> >> >> > >> >> >> >> >> >> >> >> Hom Nath >> >> >> >> >> >> >> >> On Fri, Jan 22, 2016 at 12:01 PM, Matthew Knepley >> >> >> >> <[email protected]> >> >> >> >> wrote: >> >> >> >> > On Fri, Jan 22, 2016 at 10:52 AM, Hom Nath Gharti >> >> >> >> > <[email protected]> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> Dear all, >> >> >> >> >> >> >> >> >> >> I take this opportunity to ask for your important suggestion. >> >> >> >> >> >> >> >> >> >> I am solving an elastic-acoustic-gravity equation on the >> >> >> >> >> planet. >> >> >> >> >> I >> >> >> >> >> have displacement vector (ux,uy,uz) in solid region, >> >> >> >> >> displacement >> >> >> >> >> potential (\xi) and pressure (p) in fluid region, and >> >> >> >> >> gravitational >> >> >> >> >> potential (\phi) in all of space. All these variables are >> >> >> >> >> coupled. >> >> >> >> >> >> >> >> >> >> Currently, I am using MATMPIAIJ and form a single global >> >> >> >> >> matrix. >> >> >> >> >> Does >> >> >> >> >> using a MATMPIBIJ or MATNEST improve the >> >> >> >> >> convergence/efficiency >> >> >> >> >> in >> >> >> >> >> this case? For your information, total degrees of freedoms are >> >> >> >> >> about >> >> >> >> >> a >> >> >> >> >> billion. >> >> >> >> > >> >> >> >> > >> >> >> >> > 1) For any solver question, we need to see the output of >> >> >> >> > -ksp_view, >> >> >> >> > and >> >> >> >> > we >> >> >> >> > would also like >> >> >> >> > >> >> >> >> > -ksp_monitor_true_residual -ksp_converged_reason >> >> >> >> > >> >> >> >> > 2) MATNEST does not affect convergence, and MATMPIBAIJ only in >> >> >> >> > the >> >> >> >> > blocksize >> >> >> >> > which you >> >> >> >> > could set without that format >> >> >> >> > >> >> >> >> > 3) However, you might see benefit from using something like >> >> >> >> > PCFIELDSPLIT >> >> >> >> > if >> >> >> >> > you have multiphysics here >> >> >> >> > >> >> >> >> > Matt >> >> >> >> > >> >> >> >> >> >> >> >> >> >> Any suggestion would be greatly appreciated. >> >> >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> Hom Nath >> >> >> >> >> >> >> >> >> >> On Fri, Jan 22, 2016 at 10:32 AM, Matthew Knepley >> >> >> >> >> <[email protected]> >> >> >> >> >> wrote: >> >> >> >> >> > On Fri, Jan 22, 2016 at 9:27 AM, Mark Adams >> >> >> >> >> > <[email protected]> >> >> >> >> >> > wrote: >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> I said the Hypre setup cost is not scalable, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> I'd be a little careful here. Scaling for the matrix >> >> >> >> >> >> triple >> >> >> >> >> >> product >> >> >> >> >> >> is >> >> >> >> >> >> hard and hypre does put effort into scaling. I don't have >> >> >> >> >> >> any >> >> >> >> >> >> data >> >> >> >> >> >> however. >> >> >> >> >> >> Do you? >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > I used it for PyLith and saw this. I did not think any AMG >> >> >> >> >> > had >> >> >> >> >> > scalable >> >> >> >> >> > setup time. >> >> >> >> >> > >> >> >> >> >> > Matt >> >> >> >> >> > >> >> >> >> >> >>> >> >> >> >> >> >>> but it can be amortized over the iterations. You can >> >> >> >> >> >>> quantify >> >> >> >> >> >>> this >> >> >> >> >> >>> just by looking at the PCSetUp time as your increase the >> >> >> >> >> >>> number >> >> >> >> >> >>> of >> >> >> >> >> >>> processes. I don't think they have a good >> >> >> >> >> >>> model for the memory usage, and if they do, I do not know >> >> >> >> >> >>> what >> >> >> >> >> >>> it >> >> >> >> >> >>> is. >> >> >> >> >> >>> However, generally Hypre takes more >> >> >> >> >> >>> memory than the agglomeration MG like ML or GAMG. >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> agglomerations methods tend to have lower "grid >> >> >> >> >> >> complexity", >> >> >> >> >> >> that >> >> >> >> >> >> is >> >> >> >> >> >> smaller coarse grids, than classic AMG like in hypre. THis >> >> >> >> >> >> is >> >> >> >> >> >> more >> >> >> >> >> >> of a >> >> >> >> >> >> constant complexity and not a scaling issue though. You >> >> >> >> >> >> can >> >> >> >> >> >> address >> >> >> >> >> >> this >> >> >> >> >> >> with parameters to some extent. But for elasticity, you >> >> >> >> >> >> want >> >> >> >> >> >> to >> >> >> >> >> >> at >> >> >> >> >> >> least >> >> >> >> >> >> try, if not start with, GAMG or ML. >> >> >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >> >>> Thanks, >> >> >> >> >> >>> >> >> >> >> >> >>> Matt >> >> >> >> >> >>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> Giang >> >> >> >> >> >>>> >> >> >> >> >> >>>> On Mon, Jan 18, 2016 at 5:25 PM, Jed Brown >> >> >> >> >> >>>> <[email protected]> >> >> >> >> >> >>>> wrote: >> >> >> >> >> >>>>> >> >> >> >> >> >>>>> Hoang Giang Bui <[email protected]> writes: >> >> >> >> >> >>>>> >> >> >> >> >> >>>>> > Why P2/P2 is not for co-located discretization? >> >> >> >> >> >>>>> >> >> >> >> >> >>>>> Matt typed "P2/P2" when me meant "P2/P1". >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> -- >> >> >> >> >> >>> What most experimenters take for granted before they begin >> >> >> >> >> >>> their >> >> >> >> >> >>> experiments is infinitely more interesting than any >> >> >> >> >> >>> results >> >> >> >> >> >>> to >> >> >> >> >> >>> which >> >> >> >> >> >>> their >> >> >> >> >> >>> experiments lead. >> >> >> >> >> >>> -- Norbert Wiener >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > -- >> >> >> >> >> > What most experimenters take for granted before they begin >> >> >> >> >> > their >> >> >> >> >> > experiments >> >> >> >> >> > is infinitely more interesting than any results to which >> >> >> >> >> > their >> >> >> >> >> > experiments >> >> >> >> >> > lead. >> >> >> >> >> > -- Norbert Wiener >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > What most experimenters take for granted before they begin >> >> >> >> > their >> >> >> >> > experiments >> >> >> >> > is infinitely more interesting than any results to which their >> >> >> >> > experiments >> >> >> >> > lead. >> >> >> >> > -- Norbert Wiener >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > What most experimenters take for granted before they begin their >> >> >> > experiments >> >> >> > is infinitely more interesting than any results to which their >> >> >> > experiments >> >> >> > lead. >> >> >> > -- Norbert Wiener >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their >> >> > experiments >> >> > is infinitely more interesting than any results to which their >> >> > experiments >> >> > lead. >> >> > -- Norbert Wiener >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener
