Matt, I just discovered that my approach does not work properly with VecGhostUpdateBegin/End -- I get wrong data after ghost update process. Any idea why? When I change the IS, is PETSc aware that my ghost values are not just at the end of vector?
Mohammad On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knep...@gmail.com> wrote: > On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirza...@gmail.com> > wrote: >> >> Hi guys, >> >> I just did something by pure guessing which seems to work and I want >> to make sure its the right thing! >> >> I have a specific layout for my vectors that look like this >> >> -------------------------------------------------------------- >> | ghost values | local values | ghost values | >> -------------------------------------------------------------- >> 0, 1, 2, ... m, m+1, ... m+n, m+n+1 ... N >> >> which means all nodes with index [0,m) and [m+n, N) are to be treated >> as ghost and all intermediate ones as local. Since PETSc stores the >> ghost values at the end of ghosted vec, so far I have been manually >> mapping back and forth between petsc and my application numbering. >> After profiling my code, it turned out that about 15% of run-time was >> just doing this mapping which is absolutely ridiculous! >> >> Anyway, to fix this now I set up an ISLocalToGlobalMapping object >> using my own application original local2global relation shown above. >> Then I create the petsc vec like this: >> >> // Create PetscVec >> // We do not care about the actual global index of ghost nodes at this >> point >> std::vector<PetscInt> ghost_nodes(N - n -1, 0); >> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(), >> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr); >> >> >> // Apply mapping >> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr); >> >> After this point I do the usual VecGetArray on the vec and set the >> values, but this time without any extra mapping ... my very simple >> test seems to be ok. Is this the correct way of using >> ISLocalToGlobalMapping? > > > Yes, this is the intention. And at a higher level, PETSc now tends to use a > DM to > organize this process. You tell the DM about your data layout (somewhat like > giving > the L2G mapping) and then you can DMGetLocalVector(), DMGetGlobalVector(), > and > DMLocalToGlobal(). > > Thanks, > > Matt > >> >> I guess I'm suspicious because VecCreateGhost is supposed to >> internally set the mapping and it is supposed to position ghost nodes >> at the end of the array which I don't want it to do... >> >> Thanks and sorry about the long email! > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener