Given the format I described in the previous post, I have the following mesh:
8 9 3 2 0 1 3 1 4 3 1 2 4 2 5 4 3 4 6 4 7 6 4 5 7 5 8 7 0.0 0.0 0.5 0.0 1.0 0.0 0.0 0.5 0.5 0.5 1.0 0.5 0.0 1.0 0.5 1.0 1.0 1.0 8 0 1 2 3 5 6 7 8 If I If i want to partition across 2 MPI processes, I would: a) have all processes open the file and read the first four entires. b) declare that rank 0 holds cells 0-3 and rank 1 holds cell 4-7. A special algorithm would be needed to handle cases where the number of cells is not divisible by the number of MPI processes c) rank 0 stores 0 1 3 1 4 3 1 2 4 2 5 4, rank 1 reads 3 4 6 4 7 6 4 5 7 5 8 7 d) traverse the respective arrays and read in the coordinates associated with the vertices. I am guessing this part is "tricky" because I cannot simply do a naive partitioning of the coordinates the way I could with the connectivity? e) Wont worry about exterior nodes for now f) With the above I invoke DMPlexCreateFromDAG(), (of course making sure that the numbering is of right format) g) When creating a SF for shared boundary, does this mean determining which nodes are ghosted and which ones are local? For a structured looking grid this would be easy since I could employ some sort of round-robin scheme. h) Does the DMPlexDistribute() also take care of the repartitioning and redistribution? Thanks, Justin On Sat, Jun 27, 2015 at 6:01 AM, Matthew Knepley <[email protected]> wrote: > On Sat, Jun 27, 2015 at 2:05 AM, Justin Chang <[email protected]> wrote: > >> Hi everyone, >> >> I see that parallel IO of various input mesh formats for DMPlex is not >> currently supported. However, is there a way to read a custom mesh file in >> parallel? >> >> For instance, I have a binary data file formatted like this: >> >> <No. of elements> >> <No. of vertices> >> <No. of nodes per element> >> <Spatial dimension> >> <Connectivity array ... >> ....> >> <Coordinates array ... >> ....> >> <No. of exterior nodes> >> <List of exterior nodes ... >> ....> >> >> Reading this will allow me to create a DMPlex with DMPlexCreateFromDAG(). >> Currently, only the root process reads this binary mesh file and creates >> the DMPlex whereas the other processes create an empty DMPlex. Then all >> processes invoke DMPlexDistribute(). This one-to-all distribution seems to >> be a bottleneck, and I want to know if it's possible to have each process >> read a local portion of the connectivity and coordinates arrays and let >> DMPlex/METIS/ParMETIS handle the load balancing and redistribution. >> Intuitively this would be easy to write, but again I want to know how to do >> this through leveraging the functions and routines within DMPlex. >> > > This is on our agenda for the fall, but I can describe the process: > > a) Do a naive partition of the cells (easy) > > b) Read the connectivity for "your" cells (easy) > > c) Read "your" coordinates (tricky) > > d) Read "your" exterior (tricky) > > e) Create local DAGs (easy) > > f) Create SF for shared boundary (hard) > > g) Repartition and redistribute (easy) > > You could start writing this for your format and we could help. I probably > will not get to the generic one until late in the year. > > Thanks, > > Matt > > Thanks, >> Justin >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >
