Ok I think I get it now. I'll look into DM (and probably come back with more questions!)
Thanks On Wed, Oct 2, 2013 at 11:36 AM, Matthew Knepley <[email protected]> wrote: > On Wed, Oct 2, 2013 at 1:32 PM, Mohammad Mirzadeh <[email protected]> > wrote: >> >> Matt, >> >> I just discovered that my approach does not work properly with >> VecGhostUpdateBegin/End -- I get wrong data after ghost update >> process. Any idea why? When I change the IS, is PETSc aware that my >> ghost values are not just at the end of vector? > > > Then I was probably misunderstanding what you want to do. > > 1) L2GM is for translating local to global indices automatically, but > knows nothing about "ghosting" in the Vec > > 2) For ghosting, you have a local vector and global vector and a > VecScatter that maps between them > > 3) VecGhost is a special kind of 2) since we know that the local vector > fits in side the global vec > > 4) Since you break relationship 3), I would just use the DM (as I say > below). > > Matt > >> >> Mohammad >> >> On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <[email protected]> wrote: >> > On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <[email protected]> >> > wrote: >> >> >> >> Hi guys, >> >> >> >> I just did something by pure guessing which seems to work and I want >> >> to make sure its the right thing! >> >> >> >> I have a specific layout for my vectors that look like this >> >> >> >> -------------------------------------------------------------- >> >> | ghost values | local values | ghost values | >> >> -------------------------------------------------------------- >> >> 0, 1, 2, ... m, m+1, ... m+n, m+n+1 ... N >> >> >> >> which means all nodes with index [0,m) and [m+n, N) are to be treated >> >> as ghost and all intermediate ones as local. Since PETSc stores the >> >> ghost values at the end of ghosted vec, so far I have been manually >> >> mapping back and forth between petsc and my application numbering. >> >> After profiling my code, it turned out that about 15% of run-time was >> >> just doing this mapping which is absolutely ridiculous! >> >> >> >> Anyway, to fix this now I set up an ISLocalToGlobalMapping object >> >> using my own application original local2global relation shown above. >> >> Then I create the petsc vec like this: >> >> >> >> // Create PetscVec >> >> // We do not care about the actual global index of ghost nodes at this >> >> point >> >> std::vector<PetscInt> ghost_nodes(N - n -1, 0); >> >> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(), >> >> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr); >> >> >> >> >> >> // Apply mapping >> >> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr); >> >> >> >> After this point I do the usual VecGetArray on the vec and set the >> >> values, but this time without any extra mapping ... my very simple >> >> test seems to be ok. Is this the correct way of using >> >> ISLocalToGlobalMapping? >> > >> > >> > Yes, this is the intention. And at a higher level, PETSc now tends to >> > use a >> > DM to >> > organize this process. You tell the DM about your data layout (somewhat >> > like >> > giving >> > the L2G mapping) and then you can DMGetLocalVector(), >> > DMGetGlobalVector(), >> > and >> > DMLocalToGlobal(). >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> I guess I'm suspicious because VecCreateGhost is supposed to >> >> internally set the mapping and it is supposed to position ghost nodes >> >> at the end of the array which I don't want it to do... >> >> >> >> Thanks and sorry about the long email! >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener
