Hello all, I'm okay with this for DC1, and willing to discuss further in the reference design, but the baseline should remain that the pipelines deal with the data in a standard format appropriate to the pipeline processing, and the DIC does a conversion as part of ingest if necessary.
Jeff > From: Russell E Owen <[EMAIL PROTECTED]> > Reply-To: LSST Data Management <[email protected]> > Date: Mon, 5 Jun 2006 10:01:48 -0700 > To: [email protected] > Subject: [LSST-data] Data Ingest Converters; should we get rid of them? > > I talked to Jacek on Friday and we tentatively agreed -- subject to > appropriate approval -- to try to eliminate the separate Data Ingest > Converter pipeline stage by having the code that creates the catalogs > output data in the desired format. > > In the current model, application code that writes catalogs must > output the catalog data in some standardized binary format (to be > determined). Then a separate Data Ingest Converter pipeline stage > converts that binary data to whatever format is required for database > ingest. This isolates the cataloging code from the details of the > databases and provides a natural interface between the application > developer and the data ingest programmer. > > However, having the catalog stage output the data in the final > desired form has these advantages: > - It is more efficient. For the real LSST, this efficiency may be > worth gaining. > - The code that formats the data for database ingest is easiest to > write using the objects (or data structures) available to the code > that creates the catalogs. > - It avoids the need to pick a "neutral" format (one that will change > if we ever decide we need to add columns to the database). I admit > that such changes will be very unlikely once we get far enough along, > but I expect a few between now and production (since our schema is > not finalized). > > For DC1 we will use comma-separated ASCII values. This is a format > that is understood by most, if not all databases (with minor > variation in details--for example how to represent NULL). It also has > the advantage that details of the database column types are > irrelevant because the database performs data conversion as required > during ingest (something that is impossible with binary data). Thus > it isolates the data from minor changes to the schema. Fortunately, > it is also the most efficient means of ingesting data into MySQL, so > there is no efficiency tradeoff for DC1. (We may also wish to use the > same format for the final system. That will depend radically on the > database system chosen and whether comma-separated ASCII values can > be ingested quickly and efficiently enough and what other options > there are.) > > For DC1, I suggest that some combination of Robyn, Jacek and I write > the formatting code. I think it will be quite easy (once we are sure > of the schema). > > -- Russell > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
