Here are some questions that came up when writing dbingest, that
suggest some architecting or standards that would be desirable:
*** Should policy info for a particular application be supplied as
one big policy file or as a set of policy files?
dbingest needs basically two kinds of policy information:
- pipeline info (names of slices, data directories...)
- database info (server, username...)
I asked at the time and didn't hear anything, so I went with one
policy file. But I see that Robyn went with multiple files and it
certainly makes a lot of sense to do so.
I personally suggest the following:
- Add one level of hierarchy to policy files, so keywords have a namespace.
- Use one policy file per command-line executable.
The next question only makes sense if we decide that applications
should have multiple policy files, one per type of data:
*** Should policy files that contain widely useful information be
standardized in hopes of being able to use one file for many
different applications? On the good side, standardizing reduces data
duplication. On the bad side, if it is overdone it can make code more
brittle by adding needless coupling.
An example of where it is tempting but risky: dbingestinit.py and
dbingestworker.py share a lot of information about databases. But
they can't just share a database policy file because dbingestinit.py
wants a list of databases, whereas dbingestworker.py only can handle
one database.
I suggest we pick standard names for most policy file keywords, but
not try to standardize and share policy files themselves except
perhaps in a few special cases.
*** Could we pick naming rules for slices, data directories, etc?
For example: dbingest right now assumes that all data it must ingest
will appear in subdirectories (one level deep) of a main directory.
The subdirectories will be named based on slice name with a possible
prefix and suffix (specifiable in the policy file). I'd be happy to
get rid of the prefix and/or suffix if not needed, or at least pick
appropriate defaults so one would normally not have to set it.
Similarly, dbingest creates table names based on the slice name and
the current date. I'd love to have the rules codified.
*** What should a pipeline stage do with data it cannot handle?
If dbingest can't get the database to accept a data file, at present
it logs an error and leaves the "bad" file right where it was (and
does not try to ingest it again). Is that best? Should the file be
moved somewhere else? (I can't imagine it should be deleted.)
Similarly, dbingest deletes data files after they've been
successfully ingested. Is that best, or should it be kept around
temporarily as a backup? If so, should it be moved somewhere?
-- Russell
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
