[petsc-dev] DMPlex label distribution

[Adrian Croucher]

hi






I'm running the 'next' branch.



If I run it using

mpiexec -np 2 testlabel > out

and then go

grep '(45)' out

I get:

[0]: 0 (45)
[1]: 409 (45)
[1]: 450 (45)










Am I misunderstanding something or is this unexpected behaviour?

Cheers, Adrian

--
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.crouc...@auckland.ac.nz
tel: +64 (0)9 923 84611


program testlabel

  ! tests DMLabel on DMPlex

  implicit none

#include <petsc/finclude/petsc.h90>

  PetscInt, parameter :: dim = 3, dof = 1, overlap = 1
  MPI_Comm :: comm
  PetscMPIInt :: rank
  DM :: dm, dist_dm
  PetscErrorCode :: ierr
  character(40) :: filename = "2Dgrd.exo"
  character(24) :: name = "test label"
  DMLabel :: label
  PetscInt :: start_global_cell, end_global_cell, c, label_value

  call PetscInitialize(PETSC_NULL_CHARACTER, ierr); CHKERRQ(ierr)
  comm = PETSC_COMM_WORLD
  call MPI_comm_rank(comm, rank, ierr)

  call DMPlexCreateFromFile(comm, filename, PETSC_TRUE, dm, ierr); CHKERRQ(ierr)

  call DMPlexSetAdjacencyUseCone(dm, PETSC_TRUE, ierr); CHKERRQ(ierr)
  call DMPlexSetAdjacencyUseClosure(dm, PETSC_FALSE, ierr); CHKERRQ(ierr)

  call DMCreateLabel(dm, name, ierr); CHKERRQ(ierr)
  call DMPlexGetHeightStratum(dm, 0, start_global_cell, end_global_cell, ierr); CHKERRQ(ierr)
  do c = start_global_cell, end_global_cell - 1
     label_value = c
     call DMSetLabelValue(dm, name, c, label_value, ierr); CHKERRQ(ierr)
  end do

  call DMPlexDistribute(dm, overlap, PETSC_NULL_OBJECT, dist_dm, ierr)
  CHKERRQ(ierr)
  if (dist_dm /= 0) then
     call DMDestroy(dm, ierr); CHKERRQ(ierr)
     dm = dist_dm
  end if

  call DMGetLabel(dm, name, label, ierr); CHKERRQ(ierr)
  call DMLabelView(label, PETSC_VIEWER_STDOUT_WORLD, ierr); CHKERRQ(ierr)

  call DMDestroy(dm, ierr); CHKERRQ(ierr)
  call PetscFinalize(ierr); CHKERRQ(ierr)

end program testlabel

I'm seeing some slightly puzzling behaviour when I set up a DMLabel on a DMPlex, then distribute the DMPlex using DMPlexDistribute(). Specifically, if I make the label values unique to start with (i.e. each mesh cell has a unique value), this is no longer always the case after distribution. In some cases two cells (not ghost cells) end up with the same label value, and one of the original values goes missing. The attached minimal example shows it. In this example the label values are just the cell indices before distribution, so they're unique. There are 900 mesh cells, so if running on 2 processors, each rank gets 450 of them. So there are 3 cells ending up with the label value 45. The first one is the one I expect to get this label value. The third one has cell index > 449, so I figure it's a ghost cell on rank 1 (actually the same cell as above on rank 0). So that's probably OK too. But I don't know where the second one has come from. Its index indicates it isn't a ghost cell. Based on how the other cells are labelled, I would expect this cell 409 on rank 1 to have label value 814. But it turns out there is no cell in the distributed mesh with that label value- even though there was before distribution (I checked).

2Dgrd.exo
Description: Binary data