Hi Russell,
*** Should policy info for a particular application be supplied as one
big policy file or as a set of policy files?
dbingest needs basically two kinds of policy information:
- pipeline info (names of slices, data directories...)
- database info (server, username...)
I asked at the time and didn't hear anything, so I went with one policy
file. But I see that Robyn went with multiple files and it certainly
makes a lot of sense to do so.
I'd go with separate files for this case. These are two fairly
disjoint sets.
*** Should policy files that contain widely useful information be
standardized in hopes of being able to use one file for many different
applications? On the good side, standardizing reduces data duplication.
On the bad side, if it is overdone it can make code more brittle by
adding needless coupling.
An example of where it is tempting but risky: dbingestinit.py and
dbingestworker.py share a lot of information about databases. But they
can't just share a database policy file because dbingestinit.py wants a
list of databases, whereas dbingestworker.py only can handle one database.
We had similar dilemma when configuring xrootd monitoring. We went
with a config file shared among different applications, and allowed
each application to extract the bits and pieces it needs. When
duplicating, it is easy to have inconsistent information and guarding
against it is pretty tricky. In a common file-approach, when you code
the applications correctly such that they extract precisely what they
need and ignore everything else, you end up with a more robust system.
I suggest we pick standard names for most policy file keywords, but not
try to standardize and share policy files themselves except perhaps in a
few special cases.
sounds like we agree
*** What should a pipeline stage do with data it cannot handle?
If dbingest can't get the database to accept a data file, at present it
logs an error and leaves the "bad" file right where it was (and does not
try to ingest it again). Is that best? Should the file be moved
somewhere else? (I can't imagine it should be deleted.)
Similarly, dbingest deletes data files after they've been successfully
ingested. Is that best, or should it be kept around temporarily as a
backup? If so, should it be moved somewhere?
For DC1 I'd move files to two directories, e.g. "done" and "bad".
Moving files (within the same file system) is cheap, and space should
not be an issue. You never know what sort of issues might come up,
and having these files around might help with debugging.
Thanks,
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
