there is "science data" that is never updated (level 0 uncalibrated, level 1 calibrated/derived). Pixels are level 0 and object catalogs are level 1.
The alerts and summaries and etc are derived data products (level 2). They can be re-derived on the fly from level 0-1 data if needed. But, once "published" you should keep data so that other scientists can reproduce the results (that is one of the "rules" of science). Level 0 data are never updated. Derived data can be updated (aka- discarded (update is equivalent to delete-insert). ) Summary data is often updated (but this is often best done in versions.) The SDSS went to a version scheme (one version per year) to offer consistently processed data. (it is hard to deal with data where the meanings of variables changes over time). LSST does not have that luxury, and will have to confront this problem head on (since it is real time). My guess is that LSST will publish editions where the old data is reprocessed in the new way. The old editions will be frozen and the newly arriving data will be added to the new edition. Summary data can be versioned on a daily basis. These editions give polynomial growth in storage space (above the simple linear growth you get as time passes) -- so called "data inflation". This is all doable in a world of infinite sized disks (i.e 10TB disk drives). To repeat my song -- we do not have a space problem, we have a access-per-second problem. There is NO problem keeping all versions. In your timeframe disks are 100x their current size. Jim Gray Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 8222 fax: 425 706 7329 [EMAIL PROTECTED] http://research.Microsoft.com/~gray -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tim Axelrod Sent: Friday, August 18, 2006 5:43 AM To: Jacek Becla Cc: LSST Data Management; [EMAIL PROTECTED] Subject: Re: [LSST-data] Re: cpu for queries Jacek, Ouch! I had forgotten that you wanted to do this. I'm afraid this makes more than just the alertid proposal unworkable. The whole concept of having summary information for the object, such as a wavelet decomposition of its lightcurve or even the mean magnitude, that is updated as we go along is gone too. Living with this would be very difficult! What alternatives do we have? Tim Jacek Becla wrote: > Kirk, Ani > > There is one small complication with keeping the most recent alertid > for each Object: it requires updating existing rows. > It would be nice to have the design of largest tables such that each > row is written one, and kept read-only after that. > Such approach guarantees reproducibility, and make it more manageable > (easier to replicate and distribute). > Changes/updates would be done through versioning, but I'm not sure it > is worth using versioning in this case. > > Jacek > > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
