[petsc-users] Ghost communication

[Milan Mitrovic]

Hello everyone,

I'm trying to use petsc with ppm and here is what I have so far:

First I create parallel vectors with ghost locations using an existing
array that already holds the data:

          CALL VecCreateGhostWithArray(PETSC_COMM_WORLD, &
               Particles%Npart,                          &
               PETSC_DECIDE,                             &
               Particles%Mpart-Particles%Npart,         &
               gi(Particles%Npart:Particles%Mpart),    &
               wp, petsc_vector, info)

where gi is the global indexing, wp is the existing data array, Npart
is the number of real, and Mpart the number of real+ghost particles.

Then, I create a matrix shell, and provide my own multiplication
routine (this part works fine I think).

Inside the multiplication routine I do this:

          CALL VecGhostUpdateBegin(from,INSERT_VALUES,SCATTER_FORWARD,info)
          CALL VecGhostUpdateEnd(from,INSERT_VALUES,SCATTER_FORWARD,info)

          ! get ghosted versions of vectors
          CALL VecGhostGetLocalForm(from,xl,info)
          CALL VecSet(to,0.0_MK,info)
          ! get arrays from petsc vectors
          CALL VecGetArray(xl,xx,xxi,info)
          CALL VecGetArray(to,yy,yyi,info)

Then I modify the data, and do the following:

          CALL VecRestoreArray(xl,xx,xxi,info)
          CALL VecRestoreArray(to,yy,yyi,info)
          ! release local form of vectors
          CALL VecGhostRestoreLocalForm(from,xl,info)

          CALL VecGhostUpdateBegin(to,ADD_VALUES,SCATTER_REVERSE,info)
          CALL VecGhostUpdateEnd(to,ADD_VALUES,SCATTER_REVERSE,info)

And then I call KSP solve:

  call KSPCreate(PETSC_COMM_WORLD, ksp, info)
  call KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN, info)
  call KSPSolve(ksp, b, x, info)

I noticed that the values of the ghost particles never change. The
communication never seems to take place...

Can you give me some hints on what to try? Or what I do wrong?

Thanks!

Milan Mitrovic