Hi All, I strongly support Tim on this issue. When there is an alert, a scientist should be able to access _ALL_ of the data on the object/position of the alert. We have discussed this extensively, and I thought our conclusion was that it would be a 'buyer beware' data issuance (note, I tried not to use the word release), but we would make the intra-release data available.
Kem > Jacek, > > I am not questioning the need for a data release to be a static entity - > I have in fact argued for it, and that assumption has been explicit in > all of our storage sizing. The issue is rather what happens *in > between* releases. If we cannot have an Object table which gives us > the state of our knowledge at a given moment, I think a number of > science usecases will be compromised. In particular, an observatory > dedicated to getting information on transients out in 30 seconds should > not then offer up only an Object table which is six months old! > > So, I think we need to be able to update, or a functional equivalent. > > Tim > > Jacek Becla wrote: >> Jim, >> >> Thanks! I am glad I am not alone in pushing the idea >> of read-only data =) >> >> What you said is exactly how I understood this problem: >> - released/published data is never updated >> - we may have some freedom in updating unreleased data >> - reprocessing all data is a way to 'refresh' the data, >> and will make old releases "fade away" >> - we have to use versioning to track changes for released data >> >> Tim: >> >> It sounds like we should dedicate at least one telecon to >> these issues and better understand what the implications >> of making published data immutable are. Do you want to wait >> until September or we should try doing it sooner? >> Would you be able to prepare a list of things that will >> be "unworkable"? (Based on BaBar's experience I strongly >> believe published data should be read only, and it looks >> like Jim does too) >> >> thanks, >> Jacek >> >> >> >> >> Jim Gray wrote: >>> there is "science data" that is never updated (level 0 uncalibrated, >>> level 1 calibrated/derived). Pixels are level 0 and object catalogs >>> are >>> level 1. >>> >>> The alerts and summaries and etc are derived data products (level 2). >>> They can be re-derived on the fly from level 0-1 data if needed. But, >>> once "published" you should keep data so that other scientists can >>> reproduce the results (that is one of the "rules" of science). >>> Level 0 data are never updated. Derived data can be updated (aka- >>> discarded (update is equivalent to >>> delete-insert). ) >>> Summary data is often updated (but this is often best done in >>> versions.) >>> >>> The SDSS went to a version scheme (one version per year) to offer >>> consistently processed data. (it is hard to deal with data where the >>> meanings of variables changes over time). >>> LSST does not have that luxury, and will have to confront this problem >>> head on (since it is real time). >>> My guess is that LSST will publish editions where the old data is >>> reprocessed in the new way. The old editions will be frozen and the >>> newly arriving data will be >>> added to the new edition. Summary data can be versioned on a daily >>> basis. >>> >>> These editions give polynomial growth in storage space (above the >>> simple >>> linear growth you get as time passes) -- so called "data inflation". >>> This is all doable in a world of infinite sized disks (i.e 10TB disk >>> drives). >>> To repeat my song -- we do not have a space problem, we have a >>> access-per-second problem. There is NO problem keeping all versions. >>> In your timeframe disks are 100x their current size. >>> >>> >>> Jim Gray >>> Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 >>> 8222 fax: 425 706 7329 [EMAIL PROTECTED] >>> http://research.Microsoft.com/~gray >>> >>> >>> -----Original Message----- >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Tim Axelrod >>> Sent: Friday, August 18, 2006 5:43 AM >>> To: Jacek Becla >>> Cc: LSST Data Management; [EMAIL PROTECTED] >>> Subject: Re: [LSST-data] Re: cpu for queries >>> >>> Jacek, >>> >>> Ouch! I had forgotten that you wanted to do this. I'm afraid this >>> makes more than just the alertid proposal unworkable. The whole >>> concept >>> of having summary information for the object, such as a wavelet >>> decomposition of its lightcurve or even the mean magnitude, that is >>> updated as we go along is gone too. Living with this would be very >>> difficult! What alternatives do we have? >>> >>> Tim >>> >>> Jacek Becla wrote: >>>> Kirk, Ani >>>> >>>> There is one small complication with keeping the most recent alertid >>>> for each Object: it requires updating existing rows. >>>> It would be nice to have the design of largest tables such that each >>>> row is written one, and kept read-only after that. >>>> Such approach guarantees reproducibility, and make it more >>>> manageable (easier to replicate and distribute). >>>> Changes/updates would be done through versioning, but I'm not sure >>>> it is worth using versioning in this case. >>>> >>>> Jacek >>>> >>>> >>> >>> _______________________________________________ >>> LSST-data mailing list >>> [email protected] >>> http://www.lsstmail.org/mailman/listinfo/lsst-data >> > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
