Hi Matt,
I use the latest Fortran style in PETSc 3.8. Enclosed are the PETSc
configuration, code compiling log and the function that causes compiling
error. The compiling error happens after I include petscsf.h in the
following section. I didn't find petscsf.h in petsc/finclude/ folder so
I use the head file in the 'include' folder and this seems not allowed.
#ifdef PETSC_V3_8_X
#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscdmplex.h>
#include <petsc/finclude/petscdmlabel.h>
#include <petscsf.h>
use petscsys
use petscdmplex
use petscsf
#endif
Thanks,
Danyang
On 18-03-02 12:08 PM, Matthew Knepley wrote:
On Fri, Mar 2, 2018 at 3:00 PM, Danyang Su <danyang...@gmail.com
<mailto:danyang...@gmail.com>> wrote:
On 18-03-02 10:58 AM, Matthew Knepley wrote:
On Fri, Mar 2, 2018 at 1:41 PM, Danyang Su <danyang...@gmail.com
<mailto:danyang...@gmail.com>> wrote:
On 18-02-19 03:30 PM, Matthew Knepley wrote:
On Mon, Feb 19, 2018 at 3:11 PM, Danyang Su
<danyang...@gmail.com <mailto:danyang...@gmail.com>> wrote:
Hi Matt,
Would you please let me know how to check if a cell is
local owned? When overlap is 0 in DMPlexDistribute, all
the cells are local owned. How about overlap > 0? It
sounds like impossible to check by node because a cell
can be local owned even if none of the nodes in this
cell is local owned.
If a cell is in the PetscSF, then it is not locally owned.
The local nodes in the SF are sorted, so I use
PetscFindInt
(http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscFindInt.html
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscFindInt.html>).
Hi Matt,
Would you please give me a little more about how to mark the
ghost cells when overlap > 0? What do you mean a cell is in
the PetscSF? I use PetscSFView to export the graph (original
mesh file pile.vtk) and it exports all the cells, including
the ghost cells (PETScSFView.txt).
Yes, I will send you some sample code when I get time. The first
problem is that you are looking at a different PetscSF. This
looks like the
one returned by DMPlexDistribute(). This is mapping the serial
mesh to the parallel mesh. You want
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMGetPointSF.html
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMGetPointSF.html>
Then you can look at
https://bitbucket.org/petsc/petsc/src/1788fc36644e622df8cb1a0de85676ccc5af0239/src/dm/impls/plex/plexsubmesh.c?at=master&fileviewer=file-view-default#plexsubmesh.c-683
<https://bitbucket.org/petsc/petsc/src/1788fc36644e622df8cb1a0de85676ccc5af0239/src/dm/impls/plex/plexsubmesh.c?at=master&fileviewer=file-view-default#plexsubmesh.c-683>
I get the pointSF, get out the list of leaves, and find points in
it using PetscFindInt()
Hi Matt,
By using the local dm, I can get the PetscSF I want, as shown
below. Now I need to get the number of ghost cells or local cells
(here 4944) or number of leaves (here 825) for each processor. I
try to use PetscSFGetGraph to get number of leaves in Fortran.
After including "petscsf.h", I got compilation error saying "You
need a ISO C conforming compiler to use the glibc headers". Is
there any alternative way to do this? I do not need the
ghost-neighbor mapping, but just the number of local owned cells.
Also, make sure you are using the latest Fortran style for PETSc:
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html
[0] Number of roots=11449, leaves=825, remote ranks=1
[0] 4944 <- (1,0)
[0] 4945 <- (1,28)
[0] 4946 <- (1,56)
...
[1] Number of roots=11695, leaves=538, remote ranks=1
[1] 5056 <- (0,21)
[1] 5057 <- (0,43)
[1] 5058 <- (0,65)
[1] 5059 <- (0,87)
In file included from /usr/include/features.h:375:0,
from /usr/include/stdio.h:28,
from
/home/dsu/Soft/PETSc/petsc-3.8.3/include/petscsys.h:175,
from
/home/dsu/Soft/PETSc/petsc-3.8.3/include/petscsf.h:7,
from ../../solver/solver_ddmethod.F90:4837:
/usr/include/x86_64-linux-gnu/sys/cdefs.h:30:3: error: #error "You
need a ISO C conforming compile\
r to use the glibc headers"
# error "You need a ISO C conforming compiler to use the glibc
headers"
Can you send this to petsc-ma...@mcs.anl.gov
<mailto:petsc-ma...@mcs.anl.gov>? It looks like a build problem that
can be fixed.
Thanks,
Matt
Thanks,
Danyang
Thanks,
Matt
Thanks,
Danyang
Thanks,
Matt
Thanks,
Danyang
--
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
<http://www.caam.rice.edu/%7Emk51/>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to
which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
<http://www.caam.rice.edu/%7Emk51/>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>
PETSC configuration
--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-fblaslapack --download-mpich --download-hypre --download-superlu_dist --download-hdf5=yes --download-ctetgen --with-debugging=1
/home/dsu/Soft/PETSc/petsc-3.8.3/linux-gnu-dbg/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/home/dsu/Soft/lis/lis-1.7.36/lis-mpi/include -I/home/dsu/Soft/PETSc/petsc-3.8.3/include -I/home/dsu/Soft/PETSc/petsc-3.8.3/linux-gnu-dbg/include -DLINUX -DDEBUG -DDEBUG_X64 -DSTANDARD_FORTRAN -DUSG -DPETSC -DMPI -DPETSC_V3_8_X -DOUTPUT_DOUBLE -DPETSC_HAVE_MUMPS -DPETSC_HAVE_SUPERLU -DLIS -o ../../solver/solver_ddmethod.o ../../solver/solver_ddmethod.F90
In file included from /usr/include/features.h:375:0,
from /usr/include/stdio.h:28,
from /home/dsu/Soft/PETSc/petsc-3.8.3/include/petscsys.h:175,
from /home/dsu/Soft/PETSc/petsc-3.8.3/include/petscsf.h:7,
from ../../solver/solver_ddmethod.F90:4837:
/usr/include/x86_64-linux-gnu/sys/cdefs.h:30:3: error: #error "You need a ISO C conforming compiler to use the glibc headers"
# error "You need a ISO C conforming compiler to use the glibc headers"
^
In file included from /usr/include/features.h:399:0,
from /usr/include/stdio.h:28,
from /home/dsu/Soft/PETSc/petsc-3.8.3/include/petscsys.h:175,
from /home/dsu/Soft/PETSc/petsc-3.8.3/include/petscsf.h:7,
from ../../solver/solver_ddmethod.F90:4837:
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:0: fatal error: gnu/stubs-32.h: No such file or directory
# include <gnu/stubs-32.h>
^
compilation terminated.
subroutine solver_dd_create_dmplex
#ifdef USG
#ifdef PETSC_V3_8_X
#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscdmplex.h>
#include <petsc/finclude/petscdmlabel.h>
#include <petscsf.h>
use petscsys
use petscdmplex
use petscsf
#else
#ifndef PETSC_V3_7_X
#ifndef PETSC_V3_6_X
#ifndef PETSC_V3_5_X
#ifndef PETSC_V3_4_X
#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscdmplex.h>
#include <petsc/finclude/petscdmlabel.h>
#include <petscsf.h>
use petscsys
use petscdmplex
use petscsf
#endif
#endif
#endif
#endif
#endif
use geometry_definition
use gen, only : ilog, idbg, rank, str_rank, b_enable_output
use usg_mesh_data, only : num_nodes, num_cells, nodes, cells, &
num_nodes_per_cell, cell_projection
use file_unit, only : lun_get, lun_free
use petsc_mpi_common, only : petsc_mpi_finalize
implicit none
#ifdef PETSC_V3_7_X
#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscdmplex.h>
#include <petscsf.h>
#elif PETSC_V3_6_X
#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscdmplex.h>
#include <petscsf.h>
#elif PETSC_V3_5_X
#include <finclude/petscsys.h>
#include <finclude/petscdmplex.h>
#include <petscsf.h>
#elif PETSC_V3_4_X
#include <finclude/petscsys.h>
#include <finclude/petscdmplex.h>
#include <petscsf.h>
#endif
!c local variables
integer :: icell, inode, icnt, ifile, info_debug
integer, allocatable :: dmplex_cells(:)
real*8, allocatable :: dmplex_verts(:)
character(256) :: strtitle
PetscInt :: ipoint, istart, iend, ivalue, ndim
PetscInt :: nroots,nleaves
DM :: distributedMesh, cda
PetscSection :: cs
PetscSF :: sf
PetscErrorCode :: ierr
info_debug = 0
!c note: for 2D mesh in 3D, convert coordinates to 2D
!c before passing to PETSc DMPlex
if (cell_projection == projection_xyz) then
ndim = 3
else
ndim = 2
end if
dmda_flow%dim = ndim
if (rank == 0) then
allocate(dmplex_cells(num_cells*num_nodes_per_cell))
allocate(dmplex_verts(num_nodes*ndim))
do icell = 1, num_cells
do inode = 1, num_nodes_per_cell
icnt = (icell-1)*num_nodes_per_cell+inode
dmplex_cells(icnt) = cells(inode,icell)-1
end do
end do
if (cell_projection == projection_xyz) then
do inode = 1, num_nodes
icnt = (inode-1)*ndim
dmplex_verts(icnt + 1) = nodes(inode)%x
dmplex_verts(icnt + 2) = nodes(inode)%y
dmplex_verts(icnt + 3) = nodes(inode)%z
end do
else if (cell_projection == projection_xy) then
do inode = 1, num_nodes
icnt = (inode-1)*ndim
dmplex_verts(icnt + 1) = nodes(inode)%x
dmplex_verts(icnt + 2) = nodes(inode)%y
end do
else if (cell_projection == projection_yz) then
do inode = 1, num_nodes
icnt = (inode-1)*ndim
dmplex_verts(icnt + 1) = nodes(inode)%y
dmplex_verts(icnt + 2) = nodes(inode)%z
end do
else if (cell_projection == projection_xz) then
do inode = 1, num_nodes
icnt = (inode-1)*ndim
dmplex_verts(icnt + 1) = nodes(inode)%x
dmplex_verts(icnt + 2) = nodes(inode)%z
end do
end if
!c create DMPlex from cell list
call DMPlexCreateFromCellList(Petsc_Comm_World,ndim,num_cells, &
num_nodes,num_nodes_per_cell, &
Petsc_False,dmplex_cells,ndim, & !use Petsc_True to create intermediate mesh entities (faces, edges),
dmplex_verts,dmda_flow%da,ierr) !not work for prism for the current 3.8 version.
CHKERRQ(ierr)
!c create DMPlex from Gmsh file
!call DMPlexCreateGmshFromFile(Petsc_Comm_World,"stripf.msh", &
! Petsc_False, dmda_flow%da, ierr)
!CHKERRQ(ierr)
call DMGetCoordinateDM(dmda_flow%da,cda,ierr)
CHKERRQ(ierr)
call DMGetDefaultSection(cda,cs,ierr)
CHKERRQ(ierr)
call PetscSectionGetChart(cs,istart,iend,ierr)
CHKERRQ(ierr)
!c add local to global cell id mapping
do ipoint = 0, istart-1
#ifdef PETSC_V3_8_X
call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
#elif PETSC_V3_7_X
call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
#elif PETSC_V3_6_X
call DMPlexSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
#elif PETSC_V3_5_X
call DMPlexSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
#elif PETSC_V3_4_X
call DMPlexSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
#else
call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, &
ipoint+1,ierr)
#endif
CHKERRQ(ierr)
end do
!c add local to global vertex id mapping
do ipoint = istart, iend-1
#ifdef PETSC_V3_8_X
call DMSetLabelValue(dmda_flow%da,"vid_lg2g",ipoint, &
ipoint-istart+1,ierr)
#elif PETSC_V3_7_X
call DMSetLabelValue(dmda_flow%da,"vid_lg2g",ipoint, &
ipoint-istart+1,ierr)
#elif PETSC_V3_6_X
call DMPlexSetLabelValue(dmda_flow%da,"vid_lg2g",ipoint, &
ipoint-istart+1,ierr)
#elif PETSC_V3_5_X
call DMPlexSetLabelValue(dmda_flow%da,"vid_lg2g",ipoint, &
ipoint-istart+1,ierr)
#elif PETSC_V3_4_X
call DMPlexSetLabelValue(dmda_flow%da,"vid_lg2g",ipoint, &
ipoint-istart+1,ierr)
#else
call DMSetLabelValue(dmda_flow%da,"vid_lg2g",ipoint, &
ipoint-istart+1,ierr)
#endif
CHKERRQ(ierr)
end do
deallocate(dmplex_cells)
deallocate(dmplex_verts)
else
call DMPlexCreateFromCellList(Petsc_Comm_World,ndim,0,0, &
num_nodes_per_cell, &
Petsc_False,dmplex_cells,ndim, & !use Petsc_True to create intermediate mesh entities (faces, edges),
dmplex_verts,dmda_flow%da,ierr) !not work for prism for the current 3.8 version.
CHKERRQ(ierr)
end if
#ifdef DEBUG
if (rank == 0 .and. b_enable_output) then
write(*,'(a)') "Create DMPlex from cell list, done"
end if
#endif
!c distribute mesh over processes
call DMPlexDistribute(dmda_flow%da,stencil_width, &
#ifdef PETSC_V3_8_X
PETSC_NULL_SF, &
#elif PETSC_V3_7_X
PETSC_NULL_OBJECT, &
#elif PETSC_V3_6_X
PETSC_NULL_OBJECT, &
#elif PETSC_V3_5_X
PETSC_NULL_OBJECT, &
#elif PETSC_V3_4_X
PETSC_NULL_OBJECT, &
#else
PETSC_NULL_SF, &
#endif
distributedMesh,ierr)
CHKERRQ(ierr)
#ifdef DEBUG
if (rank == 0 .and. b_enable_output) then
write(*,'(a)') "Distribute DMPlex from cell list, done"
end if
#endif
!c destroy original global mesh after distribution
#ifdef PETSC_V3_8_X
if (distributedMesh /= PETSC_NULL_DM) then
#elif PETSC_V3_7_X
if (distributedMesh /= PETSC_NULL_OBJECT) then
#elif PETSC_V3_6_X
if (distributedMesh /= PETSC_NULL_OBJECT) then
#elif PETSC_V3_5_X
if (distributedMesh /= PETSC_NULL_OBJECT) then
#elif PETSC_V3_4_X
if (distributedMesh /= PETSC_NULL_OBJECT) then
#else
if (distributedMesh /= PETSC_NULL_DM) then
#endif
call DMDestroy(dmda_flow%da,ierr)
CHKERRQ(ierr)
!c set the global mesh as distributed mesh
dmda_flow%da = distributedMesh
end if
call DMGetPointSF(dmda_flow%da,sf,ierr)
CHKERRQ(ierr)
call PetscSFGetGraph(sf,nroots,nleaves,PETSC_NULL_INTEGER, &
PETSC_NULL_INTEGER,ierr)
CHKERRQ(ierr)
write(*,*) "rank",rank,"nroots",nroots,"nleaves",nleaves
!call PetscSFView(sf,PETSC_VIEWER_STDOUT_WORLD,ierr)
!CHKERRQ(ierr)
write(*,*) "End of PetscSFView"
#endif
end subroutine solver_dd_create_dmplex
