On Mon, 2010-08-09 at 16:24 +0200, Anders Logg wrote: > On Mon, Aug 09, 2010 at 03:21:59PM +0100, Garth N. Wells wrote: > > On Mon, 2010-08-09 at 16:14 +0200, Anders Logg wrote: > > > On Mon, Aug 09, 2010 at 02:34:45PM +0100, Garth N. Wells wrote: > > > > On Mon, 2010-08-09 at 15:20 +0200, Anders Logg wrote: > > > > > On Mon, Aug 09, 2010 at 02:12:49PM +0100, Garth N. Wells wrote: > > > > > > On Mon, 2010-08-09 at 15:05 +0200, Anders Logg wrote: > > > > > > > On Mon, Aug 09, 2010 at 01:54:04PM +0100, Garth N. Wells wrote: > > > > > > > > On Mon, 2010-08-09 at 14:46 +0200, Anders Logg wrote: > > > > > > > > > On Mon, Aug 09, 2010 at 01:09:47PM +0100, Garth N. Wells > > > > > > > > > wrote: > > > > > > > > > > On Mon, 2010-08-09 at 13:53 +0200, Anders Logg wrote: > > > > > > > > > > > On Mon, Aug 09, 2010 at 12:47:10PM +0100, Garth N. Wells > > > > > > > > > > > wrote: > > > > > > > > > > > > On Mon, 2010-08-09 at 13:37 +0200, Anders Logg wrote: > > > > > > > > > > > > > On Sat, Aug 07, 2010 at 01:24:44PM +0100, Garth N. > > > > > > > > > > > > > Wells wrote: > > > > > > > > > > > > > > On Fri, 2010-08-06 at 19:55 +0100, Garth N. Wells > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > On Fri, 2010-08-06 at 20:53 +0200, Anders Logg > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > On Fri, Aug 06, 2010 at 07:51:18PM +0100, Garth > > > > > > > > > > > > > > > > N. Wells wrote: > > > > > > > > > > > > > > > > > On Fri, 2010-08-06 at 20:36 +0200, Anders > > > > > > > > > > > > > > > > > Logg wrote: > > > > > > > > > > > > > > > > > > On Fri, Aug 06, 2010 at 04:55:44PM +0100, > > > > > > > > > > > > > > > > > > Garth N. Wells wrote: > > > > > > > > > > > > > > > > > > > On Fri, 2010-08-06 at 08:42 -0700, Johan > > > > > > > > > > > > > > > > > > > Hake wrote: > > > > > > > > > > > > > > > > > > > > On Friday August 6 2010 08:16:26 you > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > revno: 4896 > > > > > > > > > > > > > > > > > > > > > committer: Garth N. Wells > > > > > > > > > > > > > > > > > > > > > <gn...@cam.ac.uk> > > > > > > > > > > > > > > > > > > > > > branch nick: dolfin-all > > > > > > > > > > > > > > > > > > > > > timestamp: Fri 2010-08-06 16:13:29 > > > > > > > > > > > > > > > > > > > > > +0100 > > > > > > > > > > > > > > > > > > > > > message: > > > > > > > > > > > > > > > > > > > > > Add simple Stokes solver for > > > > > > > > > > > > > > > > > > > > > parallel testing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Other Stokes demos don't run in > > > > > > > > > > > > > > > > > > > > > parallel because MeshFunction io is > > > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > supported in parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Does anyone have an overview of what is > > > > > > > > > > > > > > > > > > > > needed for this to be fixed. I > > > > > > > > > > > > > > > > > > > > couldn't find a blueprint on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here it is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://blueprints.launchpad.net/dolfin/+spec/parallel-io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am interested in getting this fixed :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Me too! We need to look at all the io > > > > > > > > > > > > > > > > > > > since much of it is broken in > > > > > > > > > > > > > > > > > > > parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We need to settle on how to handle XML > > > > > > > > > > > > > > > > > > > data. I favour (and I know Niclas > > > > > > > > > > > > > > > > > > > Janson does too) the VTK approach in > > > > > > > > > > > > > > > > > > > which we have a 'master file' that > > > > > > > > > > > > > > > > > > > points to other XML files which contain > > > > > > > > > > > > > > > > > > > portions of the vector/mesh, > > > > > > > > > > > > > > > > > > > etc. Process zero can read the 'master > > > > > > > > > > > > > > > > > > > file' and then instruct the other > > > > > > > > > > > > > > > > > > > processes on which file(s) they should > > > > > > > > > > > > > > > > > > > read in. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This only works if the data is already > > > > > > > > > > > > > > > > > > partitioned. Most of our demos > > > > > > > > > > > > > > > > > > assume that we have the mesh in one single > > > > > > > > > > > > > > > > > > file which is then > > > > > > > > > > > > > > > > > > partitioned on the fly. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The approach does work for data which is not > > > > > > > > > > > > > > > > > partitioned. Just like with > > > > > > > > > > > > > > > > > VTK, one can read the 'master file' or the > > > > > > > > > > > > > > > > > individual files. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The initial plan was to support two > > > > > > > > > > > > > > > > > > different ways of reading data in parallel: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. One file and automatic partitioning > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > DOLFIN gets one file "mesh.xml", each > > > > > > > > > > > > > > > > > > process reads one part of it (just > > > > > > > > > > > > > > > > > > skipping other parts of the file), then the > > > > > > > > > > > > > > > > > > mesh is partitioned and > > > > > > > > > > > > > > > > > > redistributed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Several files and no partitioning > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > DOLFIN get multiple files and each process > > > > > > > > > > > > > > > > > > reads one part. In this > > > > > > > > > > > > > > > > > > case, the mesh and all associated data is > > > > > > > > > > > > > > > > > > already partitioned. This > > > > > > > > > > > > > > > > > > should be very easy to fix since everything > > > > > > > > > > > > > > > > > > that is needed is already > > > > > > > > > > > > > > > > > > in place; we just need to fix the logic. In > > > > > > > > > > > > > > > > > > particular, the data > > > > > > > > > > > > > > > > > > section of each local mesh contains all > > > > > > > > > > > > > > > > > > auxilliary parallel data. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This can be handled in two different ways. > > > > > > > > > > > > > > > > > > Either a user specifies the > > > > > > > > > > > > > > > > > > name of the file as "mesh*.xml", in which > > > > > > > > > > > > > > > > > > case DOLFIN appends say > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "_%d" % MPI::process_number() > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on each local process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The other way is to have a master file > > > > > > > > > > > > > > > > > > which lists all the other > > > > > > > > > > > > > > > > > > files. In this case, I don't see a need for > > > > > > > > > > > > > > > > > > process 0 to take any kind > > > > > > > > > > > > > > > > > > of responsibility for communicating file > > > > > > > > > > > > > > > > > > names. It would work fine for > > > > > > > > > > > > > > > > > > each process to read the master file and > > > > > > > > > > > > > > > > > > then check which file it > > > > > > > > > > > > > > > > > > should use. Each process could also check > > > > > > > > > > > > > > > > > > that the total number of > > > > > > > > > > > > > > > > > > processes matches the number of partitions > > > > > > > > > > > > > > > > > > in the file. We could let > > > > > > > > > > > > > > > > > > process 0 handle the parsing of the master > > > > > > > > > > > > > > > > > > file and then communicate > > > > > > > > > > > > > > > > > > the file names but maybe that is an extra > > > > > > > > > > > > > > > > > > complication. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This fails when the number of files differs > > > > > > > > > > > > > > > > > from the number of > > > > > > > > > > > > > > > > > processes. It's very important to support m > > > > > > > > > > > > > > > > > files on n processes. We've > > > > > > > > > > > > > > > > > discussed this at length before. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't remember. Can you remind me of what the > > > > > > > > > > > > > > > > reasons are? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I perform a simulation using m processes, and > > > > > > > > > > > > > > > write the result to m > > > > > > > > > > > > > > > files. Later I want to use the result later in > > > > > > > > > > > > > > > another computation using > > > > > > > > > > > > > > > n processors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've looked a little into parallel io, and looked > > > > > > > > > > > > > > at what Trilinos and > > > > > > > > > > > > > > PETSc do. Both support HDF5, and HDF5 has been > > > > > > > > > > > > > > developed to work > > > > > > > > > > > > > > parallel. HDF5 does not advocate the one-file per > > > > > > > > > > > > > > processes (too awkward > > > > > > > > > > > > > > and complicated they say), but advocates a one file > > > > > > > > > > > > > > approach. It has > > > > > > > > > > > > > > tools that allow different processes to write to > > > > > > > > > > > > > > different parts of the > > > > > > > > > > > > > > same file in parallel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > >From reading this, what I propose (for now) is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. We only ever write one XML file for a given > > > > > > > > > > > > > > object. This file can be > > > > > > > > > > > > > > read by different processes, with each reading in > > > > > > > > > > > > > > only a chunk. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. We should add an XML format for partitioning > > > > > > > > > > > > > > data (Trilinos calls > > > > > > > > > > > > > > this a 'map'). If a map file is present, it is used > > > > > > > > > > > > > > to define the > > > > > > > > > > > > > > partitions. It may make sense to have a map file > > > > > > > > > > > > > > for each process (but > > > > > > > > > > > > > > no need for a 'master file'). > > > > > > > > > > > > > > > > > > > > > > > > > > I suggest something slightly different. I'm ok with > > > > > > > > > > > > > the one file > > > > > > > > > > > > > approach, but it would be good to store that data in > > > > > > > > > > > > > a partitioned > > > > > > > > > > > > > way. Our current model for parallel computing is that > > > > > > > > > > > > > each process has > > > > > > > > > > > > > a Mesh and each process has the partitioning data it > > > > > > > > > > > > > needs stored in > > > > > > > > > > > > > the data section of the Mesh. So each process has > > > > > > > > > > > > > just a regular mesh > > > > > > > > > > > > > with some auxilliary data attached to it. That makes > > > > > > > > > > > > > it easy to read > > > > > > > > > > > > > and write using already existing code. (No need for a > > > > > > > > > > > > > special parallel > > > > > > > > > > > > > format.) > > > > > > > > > > > > > > > > > > > > > > > > > > But we could easily throw all that data into one big > > > > > > > > > > > > > file, something > > > > > > > > > > > > > like this: > > > > > > > > > > > > > > > > > > > > > > > > > > <distributed_mesh num_parts="16"> > > > > > > > > > > > > > <mesh ...> > > > > > > > > > > > > > ... > > > > > > > > > > > > > </mesh> > > > > > > > > > > > > > <mesh ...> > > > > > > > > > > > > > ... > > > > > > > > > > > > > </mesh> > > > > > > > > > > > > > ... > > > > > > > > > > > > > </distributed_mesh> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would like to separate mesh and partitioning data. A > > > > > > > > > > > > partitioning of a > > > > > > > > > > > > given mesh is not unique to a mesh, so it should be > > > > > > > > > > > > separated. A > > > > > > > > > > > > partition could still go in the same XML file though. > > > > > > > > > > > > > > > > > > > > > > It is separate from the mesh, but it is stored locally as > > > > > > > > > > > part of the > > > > > > > > > > > Mesh in MeshData. I think this has proved to be a very > > > > > > > > > > > good way > > > > > > > > > > > (efficient and simple) to store the data. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This breaks the concept of using a given Mesh on a > > > > > > > > > > different number of > > > > > > > > > > processes. For a given mesh, I may want to store and use > > > > > > > > > > different > > > > > > > > > > partitions. > > > > > > > > > > > > > > > > > > > > The same applied to Vectors. > > > > > > > > > > > > > > > > > > > > > Or do you suggest that we store it differently on file > > > > > > > > > > > than we do as > > > > > > > > > > > part of the DOLFIN data structures? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I suggest that we decouple the Mesh and the partition data > > > > > > > > > > for file > > > > > > > > > > output. It comes back to permitting a variable number of > > > > > > > > > > processes. > > > > > > > > > > > > > > > > > > I thought that the storage output from m processes would be > > > > > > > > > the mesh > > > > > > > > > partitioned into m pieces with auxilliary partitioning > > > > > > > > > information. This would then be read by n processes and > > > > > > > > > repartitioned. > > > > > > > > > > > > > > > > > > If you don't want the partitioning into a specific number of > > > > > > > > > partitions (like m partitions above), then what extra > > > > > > > > > partitioning > > > > > > > > > data would be required? It would just be a regular mesh file. > > > > > > > > > > > > > > > > > > > > > > > > > It would map entities to processes. It could also be used for > > > > > > > > Vectors. > > > > > > > > > > > > > > Then how is that not coupled to a specific number of processes? > > > > > > > > > > > > > > And why would it be better than what we have today, which stores > > > > > > > all > > > > > > > the parallel data we need (not just that mapping but everything > > > > > > > else > > > > > > > that can and must be generated from it) as part of each mesh? > > > > > > > > > > > > > > > > Partitioning data is always coupled to a specific number of > > > > > > > > > processes > > > > > > > > > as far as I can imagine. > > > > > > > > > > > > > > > > > > > > > > > > > Naturally, and so it should be. But the mesh output doesn't > > > > > > > > have to be, > > > > > > > > which why I suggest having a Mesh and partitioning data, e.g. > > > > > > > > > > > > > > > > // Let DOLFIN partition and distribute the mesh > > > > > > > > Mesh meshA("mesh.xml") > > > > > > > > > > > > > > Is this just one regular mesh file? > > > > > > > > > > > > > > > > > > > Yes, and ParMETIS would partition the mesh. > > > > > > > > > > > > > > // Partition the mesh according to my_partition.xml mesh. > > > > > > > > Throw an > > > > > > > > // error if there is a mis-match between my_partition and the > > > > > > > > number > > > > > > > > // of processes > > > > > > > > Mesh meshB("mesh.xml", "my_partition.xml") > > > > > > > > > > > > > > I don't understand this use-case. The first line above would > > > > > > > already > > > > > > > partition the mesh (using ParMETIS), and then you want to > > > > > > > repartition > > > > > > > it. Is it the case that my_partition.xml contains a better > > > > > > > partitioning than what ParMETIS can compute? > > > > > > > > > > > > > > > > > > > In the case "meshB("mesh.xml", "my_partition.xml")", ParMETIS would > > > > > > never be called. The mesh would be distributed according to > > > > > > "my_partition.xml". > > > > > > > > > > I missed that you created two different meshes. ok, that sounds like a > > > > > good thing to have. > > > > > > > > > > But how would the mesh.xml be stored on file? Just one big file? > > > > > > > > > > > > > Yes, just as we have now. > > > > > > ok. > > > > > > > > And how is the above example related to the storing of *partitioned* > > > > > meshes? > > > > > > > > > > > > > >From reading around, I don't think that we should store partitioned > > > > meshes in the sense that each partition is a separate file. We should > > > > store an entire mesh in one file, and store the partitioning data if > > > > desired. > > > > > > I don't understand why not. It's a good feature to have to be able to > > > restart a simulation (on the same number of processors) and completely > > > avoid the partitioning step. Moreover, everything we need is already > > > implemented since we can read/write meshes including mesh data. > > > > > > > It is a good feature. To do it, save the mesh and the partition, and > > then restart using > > > > Mesh mesh("mesh.xml", "partition.xml"); > > > > This will not involve any partitioning. > > It's true that it wouldn't involve any partitioning (as in calling > ParMETIS) but it would involve redistributing the mesh between all > processes and computing all the auxilliary data that is needed, like > the global vertex numbering and mappings to neighboring processes. >
No, because the process would only read the bits which it needs. The partition data would contains the other data. > By storing one file per process (which is almost already supported), > that would also be avoided. Each process would read a regular mesh > file and be ready to go. No communication and no computation involved. > > > > In summary, if I understand correctly that you want things stored in > > > one file, we would have the following: > > > > > > 1. Reading from one file and automatic partitioning/redistribution > > > > > > This is already supported today, and it already supports the use case > > > of first running on m processors and then n processors since in both > > > cases the mesh will be read from one single file and partitioned. > > > > > > > Yes. > > > > > 2. Reading from multiple files where the mesh has already been > > > partitioned and m = n. This is almost supported today. We just need to > > > decide on the logic for reading from multiple files. > > > > > > > I advocate not supporting multiple files. Keeping to one file-file model > > makes things simple. > > Multiple files would not be much of a complication since it is almost > already in place. Each process just reads its corresponding file > without any communication. > Having dug around in the XML files over the past few days, I wouldn't agree with this. It may seem outwardly simple, but it means files having different states and it gets messy. Garth > -- > Anders _______________________________________________ Mailing list: https://launchpad.net/~dolfin Post to : dolfin@lists.launchpad.net Unsubscribe : https://launchpad.net/~dolfin More help : https://help.launchpad.net/ListHelp
