On Wed, Feb 24, 2010 at 9:09 AM, Jed Brown <jed at 59a2.org> wrote: > On Wed, 24 Feb 2010 08:31:32 -0600, Dmitry Karpeev <karpeev at mcs.anl.gov> > wrote: > > Yes, I think SQL or some such approach would be a good solution. > > I don't even think the actual file format matters too much: we can just > > create collections of files that share keys. The database is needed only > > to manage file names. It could also store other data, of course, but > > that's just gravy. > > I think the database needs to hold a nontrivial amount of semantic > information. For example, suppose we have a DMComposite covering > multiple domains, with some domains having more than one DM on the same > mesh (as in mixed FEM). These DMs will share coordinate DMs and the > associated position vectors (which may be time-dependent). Other > metadata, such as precision, endianness, units, scaling factors, time, > and projections, would (in my opinion) also go in the database so that > everything can be wired up without opening these files, and they can be > slurped in with a single collective read.
Yes, we had to add all this to PyLith to get things to make sense. We should try and get a list together. I can start with what PyLith has. Matt > > Yes, labels are cumbersome, since they have to be create manually, etc. > > However, when we decide where on the filesystem to place a file, we are > > essentially selecting its labels: the directories on the path. At least > those > > are *some* of the labels we'd like to attach to the file and the > filesystem only > > allows "labels" encoded as directories. I agree that it would be nice to > allow > > more general queries, but based on what (permissions, timestamp? those > > sound like natural candidates)? > > I wasn't thinking of filesystem metadata at all, it's the user-visible > attributes and relationship among objects in the simulation that are > significant. We have to drop the files somewhere and give them a name, > but I'd be happy if they were just named by SHA1. The name has no > significance since you can't do anything with it without the semantic > information in the database. > > Jed > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100224/b3d57b12/attachment.html>
