Here are some notes and questions about the catalog ingest service
(henceforth CIS):
It is written in python. So far I am coding a class that, when
instantiated with a suitable policy (that includes a list of slices)
in a process will:
- create one connection to a MySQL database
- create a set of tables in that database, one per slice
- monitor the appropriate directories (one per slice) for catalog data files
(these files are written by Jacek's data converters).
- as new files are found, load them into the appropriate database table
**** Assumptions/proposals ****
** One instance can only talk to one database. Thus at least two
instances will be required (since we have two databases). More than
two are fine; it is entirely up to whoever instantiates these.
** The CIS needs some way to know it is done. I propose that the data
ingest converter write an empty file with a special name (e.g.
"done") when the last data is processed. The advantage is that the
data converter is part of the pipeline and will know when there is no
more data to be processed, whereas the CIS runs asynchronously and so
a message saying "all data processed by the pipeline" is meaningless.
** Each slice has a unique name (or integer ID, in which case the
name is the string representation of the integer ID). I hope and
assume that directory for data from a given slice is composed of a
constant prefix + name + constant suffix. Ditto for the corresponding
database table name (but it may be a different prefix and/or suffix).
It would be nice if we could just use a prefix or a suffix. One
subtlety, if using an integer ID, is whether to use leading zeros. If
we do that, then the field width must be available (e.g. via a
policy).
** Once one table exists, the easiest way to create another is to ask
the database server to copy it. So...this brings up the question of
initialization. Should the CIS create the first table, e.g. from a
file containing schema, or should a template table be created as part
of setting up the databases initially? The latter is simpler for the
CIS and slightly more efficient (since otherwise each instance of a
CIS needs to first create one table using SQL before copying the
rest), but it's really no big deal. In any case, I have coded the CIS
to work either way -- if a template table is specified, it is used,
else a scheme file must be specified.
**** Questions ****
** How many policies should the CIS get? Issues include handling
shared info while avoiding duplication of info among different
policies, and avoiding keyword collisions.
My gut feeling is that multiple policies make sense. I'm thinking
that the CIS could use three:
- slice info (includes info for converting slice name to a directory
for files and to a database table name); this same policy is also
sent to the data converters.
- CIS-specific info: the list of slices to process
- database info for connecting to the database (db name, host, user,
password); I think this policy would be useful for any process that
needs to connect to that database, but could also see it just being
part of the CIS policy
** The CIS needs some way to send a message when it finishes, so that
queries on the ingested data can begin. How should that be handled?
** Who instantiates the CIS? Is supplying a python class sufficient
on my part, or should I also be writing some higher level code for
this purpose?
** What should the CIS do with the data files after ingesting them?
Presumably delete them if the ingest succeeds, but what if it fails?
Should it save them in that case?
- How does the CIS report errors?
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
