I talked to Jacek on Friday and we tentatively agreed -- subject to
appropriate approval -- to try to eliminate the separate Data Ingest
Converter pipeline stage by having the code that creates the catalogs
output data in the desired format.
In the current model, application code that writes catalogs must
output the catalog data in some standardized binary format (to be
determined). Then a separate Data Ingest Converter pipeline stage
converts that binary data to whatever format is required for database
ingest. This isolates the cataloging code from the details of the
databases and provides a natural interface between the application
developer and the data ingest programmer.
However, having the catalog stage output the data in the final
desired form has these advantages:
- It is more efficient. For the real LSST, this efficiency may be
worth gaining.
- The code that formats the data for database ingest is easiest to
write using the objects (or data structures) available to the code
that creates the catalogs.
- It avoids the need to pick a "neutral" format (one that will change
if we ever decide we need to add columns to the database). I admit
that such changes will be very unlikely once we get far enough along,
but I expect a few between now and production (since our schema is
not finalized).
For DC1 we will use comma-separated ASCII values. This is a format
that is understood by most, if not all databases (with minor
variation in details--for example how to represent NULL). It also has
the advantage that details of the database column types are
irrelevant because the database performs data conversion as required
during ingest (something that is impossible with binary data). Thus
it isolates the data from minor changes to the schema. Fortunately,
it is also the most efficient means of ingesting data into MySQL, so
there is no efficiency tradeoff for DC1. (We may also wish to use the
same format for the final system. That will depend radically on the
database system chosen and whether comma-separated ASCII values can
be ingested quickly and efficiently enough and what other options
there are.)
For DC1, I suggest that some combination of Robyn, Jacek and I write
the formatting code. I think it will be quite easy (once we are sure
of the schema).
-- Russell
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
