Re: [petsc-users] Understanding DMPlexDistribute overlap

Wed, 05 Apr 2017 04:04:23 -0700 

Hi Michael,

thanks for the prompt reply!

While I am happy I mostly got it right, this means I have some kind of problem 
I cannot solve on my own. :(

I have this very simple 2D mesh I am experimenting with: a rectangle with  64 
vertexes and 45 cells (attached in exodus format as cantilever.e); I am using 
in this very simple petsc4py program to read it, define a section and output a 
vector. The overlay value can be controlled by the -o command line switch. The 
program is executed as:
mpiexec -np 2 python overlay-test.py -o <overlap_value> -log_view.


Everything works smoothly for <overlap_value> = 0 or to 1, but for values >2 
the program fails with the error message captured in the attached file 
error.log. Changing the number of processors does not alter the behavior. Note 
also that the same holds if I use a mesh generated by DMPlexCreateBoxMesh.

I would really appreciate hints on how to solve this issue and I will of 
course provide any needed additional information.

Thank you very much,
FC

On mercoledì 5 aprile 2017 10:50:59 CEST you wrote:
> Hi Francesco,
> 
> Your description is almost correct: the overlap defines the topological
> depth of shared entities as counted in "neighboring cells", where a cell
> counts as a neighbor of an owned cell according to the defined adjacency
> style. So for overlap=0 only facets, edges and vertices may be shared
> along the partition boundary, whereas for overlap=1 you can expect one
> additional "layer" of cells around each partition (the partitioning is
> done based on cell connectivity). For second neighbors, however, you
> need overlap=2. And yes, there is conceptually no upper bound on the
> overlap.
> 
> Hope this helps,
> 
> Michael
> 
> On 05/04/17 10:27, Francesco Caimmi wrote:
> > Dear all,
> > 
> > I was playing with DMPlex objects and I was trying to exactly figure out
> > what the `overlap` parameter in DMPlexDistribute does.
> > 
> >  From the tutorial "Flexible, Scalable Mesh and Data Management
> > 
> > using PETSc DMPlex" (slide 10) and from the work by Knepley et al.
> > "Unstructured Overlapping Mesh Distribution in Parallel" I somehow got the
> > idea that it should control the "depth" of the mesh overlap.
> > That is, given the partition boundary, if overlay is set to 0 only the
> > entities adjacent (in the DMPlex topological sense and with the "style"
> > defined by the AdjacencyUse routines) to entities at the boundary are
> > shared, if overlay is 1 the first and the second neighbors (always in the
> > DMPlex topological sense) are shared and so on, up to the point were we
> > have a full duplicate of the mesh on each process (i.e. there is no upper
> > bound on `overlap`).
> > 
> > Is this correct or am I -totally- misunderstanding the meaning of the
> > parameter?
> > 
> > I am asking this because I see some behavior I cannot explain at varying
> > the value of the overlap, but before going into the details I would like
> > to be sure to understand  exactly what the overlap parameter is supposed
> > to do.
> > 
> > Many thanks,


-- 
Francesco Caimmi
 
Laboratorio di Ingegneria dei Polimeri
http://www.chem.polimi.it/polyenglab/

Politecnico di Milano - Dipartimento di Chimica,
Materiali e Ingegneria Chimica “Giulio Natta”

P.zza Leonardo da Vinci, 32
I-20133 Milano
Tel. +39.02.2399.4711
Fax +39.02.7063.8173

[email protected]
Skype: fmglcaimmi (please arrange meetings by e-mail)

CDF

len_string!len_lineQfour	time_stepnum_dim	num_nodes@num_elem-
num_el_blk
num_qa_rec
num_node_setsnum_nod_ns1num_nod_ns2num_nod_ns3$num_el_in_blk1-num_nod_per_el1num_att_in_blk1api_version@�\)version@�\)floating_point_word_size	file_sizetitleGcubit(C:/users/fcaimmi/My Documents/temp/test.e): 03/19/2017: 09:09:55
time_whole�
qa_records��
coor_namesDHeb_names$�	ns_status	�ns_prop1	nameID�ns_names	d�node_ns1
,
dist_fact_ns1
 <node_ns2\
dist_fact_ns2`node_ns3�h
dist_fact_ns3 �elem_map�
elem_num_map�
�node_num_map�	eb_status�eb_prop1nameID�attrib1
h�attrib_name1$
�connect1
	elem_typeSHELL4�coordx�coordy�CUBIT15.003/19/201709:09:55xy 0@?�?�?�?�?�@2 0?>=<;:98765431"	
?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�	

 !"#$%&'()*+,-	

 !"#$%&'()*+,-ACedDfEgFhGiHjIkJlKmLnMoNpOqPrBRsctuvwxyz{|}~�SbTa`_^]\[ZYXWVUQ?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�	
	


 !"#!$#
%$
&%'&(')(*)+*,+-,.-/. 0/"!12!#31#$43$%54%&65&'76'(87()98)*:9*+;:+,<;,-=<-.>=./?>/0@?@@@������@@@@@������@@������@@?�?�?�?���������������������������������������@������@@������@@������@?�?�����������������������@@@@@@?�?�����������?�?�?�������?�?�?�?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�������?�?�����������������������������������������������������������������
#!/usr/bin/python
"""
author: [email protected]

This code plays with DMPlex overlaps

"""


from __future__ import print_function
import argparse
import numpy as np
import sys
import petsc4py


def create_dm(fname=None):
    """
    Creates the dmplex object
    """
    if fname is None:
        #create a 2D box mesh
        dim = 2
        dm = PETSc.DMPlex().createBoxMesh(dim)
    else:
        #read from file
        dm = PETSc.DMPlex().createExodusFromFile(fname)
        dim = dm.getDimension()

    return dm

def create_section(dm):
    """
    Creates the default section

    """
    dim = dm.getDimension()
    #define a scalar field on vertices
    numFields = 1
    numComp = np.zeros(numFields, dtype=PETSc.IntType)
    numComp[0] = 1
    numDof = np.zeros(numFields*(dim+1), dtype=PETSc.IntType)
    numDof[0] = 1 #dofs on vertex only
    vector_section = dm.createSection(numComp, numDof)
    vector_section.setName('vector')
    dm.setDefaultSection(vector_section)
    return vector_section

def main(fname='cantilever.e', overlap=0):

    dm = create_dm(fname=fname)
    dm.view()
#    dm.setAdjacencyUseClosure(False)
#    dm.setAdjacencyUseCone(True)
    if PETSc.COMM_WORLD.size == 1:
        PETSc.Sys.Print('This example should be run on multiple processes')
        sys.exit(1)
    else:
        dm.distribute(overlap = overlap)

    section = create_section(dm)
    v = dm.createLocalVec()
    v.setName('rank')
    v.set(PETSc.COMM_WORLD.rank)
    viewer = PETSc.Viewer().createVTK('out.vtu')
    v.view(viewer)
    del viewer
    PETSc.Sys.Print('DM after distribution with overlap', overlap)
    dm.view()
#    PETSc.Sys.Print('Section after distribution with overlap', overlap)
#    section.view()
    PETSc.Sys.Print('Local vector length distribution with overlap', overlap)
    PETSc.Sys.syncPrint(v.getSize())
    PETSc.Sys.syncFlush()



if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=
                                     'Playing with dmplex distrubute overlay')
    parser.add_argument('-o', '--overlap', help="Overlap value", type=int,
                        default=0)
    parser.add_argument('-f', '--file', help="Overlap value",
                        default='cantilever.e')
    options, petsc_args = parser.parse_known_args(sys.argv[1:])
    petsc4py.init(petsc_args)
    from petsc4py import PETSc
    options = vars(options)

    main(fname=options['file'], overlap=options['overlap'])

Traceback (most recent call last):
  File "./overlay-test.py", line 90, in <module>
    main(fname=options['file'], overlap=options['overlap'])
  File "./overlay-test.py", line 59, in main
    dm.distribute(overlap = overlap)
  File "PETSc/DMPlex.pyx", line 476, in petsc4py.PETSc.DMPlex.distribute (src/petsc4py.PETSc.c:229733)
petsc4py.PETSc.Error: error code 63
[1] DMPlexDistribute() line 1635 in /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[1] DMPlexDistributeOverlap() line 1729 in /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[1] DMPlexCreateOverlap() line 615 in /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[1] DMPlexPartitionLabelPropagate() line 1633 in /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexpartition.c
[1] DMLabelGather() line 1092 in /home/fcaimmi/Packages/petsc/src/dm/label/dmlabel.c
[1] PetscSFSetGraph() line 347 in /home/fcaimmi/Packages/petsc/src/vec/is/sf/interface/sf.c
[1] PetscTableAdd() line 54 in /home/fcaimmi/Packages/petsc/include/petscctable.h
[1] Argument out of range
[1] key 369 is greater than largest key allowed 2
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: [1] PetscCommDuplicate line 136 /home/fcaimmi/Packages/petsc/src/sys/objects/tagm.c
[1]PETSC ERROR: [1] PetscViewerASCIIGetStdout line 38 /home/fcaimmi/Packages/petsc/src/sys/classes/viewer/impls/ascii/vcreatea.c
[1]PETSC ERROR: [1] PetscOptionsGetViewer line 147 /home/fcaimmi/Packages/petsc/src/sys/classes/viewer/interface/viewreg.c
[1]PETSC ERROR: [1] PetscLogViewFromOptions line 1804 /home/fcaimmi/Packages/petsc/src/sys/logging/plog.c
[1]PETSC ERROR: [1] PetscFinalize line 1004 /home/fcaimmi/Packages/petsc/src/sys/objects/pinit.c
[1]PETSC ERROR: [1] PetscTableAdd line 52 /home/fcaimmi/Packages/petsc/include/petscctable.h
[1]PETSC ERROR: [1] PetscSFSetGraph line 290 /home/fcaimmi/Packages/petsc/src/vec/is/sf/interface/sf.c
[1]PETSC ERROR: [1] DMLabelGather line 1063 /home/fcaimmi/Packages/petsc/src/dm/label/dmlabel.c
[1]PETSC ERROR: [1] DMPlexPartitionLabelPropagate line 1628 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexpartition.c
[1]PETSC ERROR: [1] DMPlexCreateOverlap line 561 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[1]PETSC ERROR: [1] DMPlexDistributeOverlap line 1715 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[1]PETSC ERROR: [1] DMPlexDistribute line 1555 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.7.5, unknown 
[1]PETSC ERROR: ./overlay-test.py on a my-linux-petsc named pc-fcaimmi by fcaimmi Wed Apr  5 12:47:25 2017
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-openmpi --download-hdf5 --download-f2cblaslapack --download-netcdf --with-valgrind=True --download-ptscotch --download-exodusii --download-superlu_dist --download-fftw --download-metis --download-parmetis --download-chaco --download-hypre --download-triangle --download-petsc4py
[1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] PetscCommDuplicate line 136 /home/fcaimmi/Packages/petsc/src/sys/objects/tagm.c
[0]PETSC ERROR: [0] PetscHeaderCreate_Private line 36 /home/fcaimmi/Packages/petsc/src/sys/objects/inherit.c
[0]PETSC ERROR: [0] PetscSectionCreate line 42 /home/fcaimmi/Packages/petsc/src/vec/is/utils/vsectionis.c
[0]PETSC ERROR: [0] DMLabelDistribute_Internal line 898 /home/fcaimmi/Packages/petsc/src/dm/label/dmlabel.c
[0]PETSC ERROR: [0] DMLabelGather line 1063 /home/fcaimmi/Packages/petsc/src/dm/label/dmlabel.c
[0]PETSC ERROR: [0] DMPlexPartitionLabelPropagate line 1628 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexpartition.c
[0]PETSC ERROR: [0] DMPlexCreateOverlap line 561 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[0]PETSC ERROR: [0] DMPlexDistributeOverlap line 1715 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[0]PETSC ERROR: [0] DMPlexDistribute line 1555 /home/fcaimmi/Packages/petsc/src/dm/impls/plex/plexdistribute.c
[pc-fcaimmi:07183] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[pc-fcaimmi:07183] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Reply via email to