Hi Deborah, Actually, the approach you cited is the baseline. We store raw images and source and object catalogs, plus provenance needed to re-generate the catalogs from the raw images if we need to. There are a large number of intermediate image processing products that we do not store, but re-create on they fly as needed.
Jeff > From: Deborah Levine <[EMAIL PROTECTED]> > Reply-To: LSST Data Management <[email protected]> > Date: Wed, 12 Jul 2006 09:42:28 -0700 > To: LSST Data Management <[email protected]> > Subject: RE: [LSST-data] disk space for old releases > > Hi, > > I'm just the peanut gallery here, but with the volumes of data LSST is > going to produce -- does it make more sense to store all the old processed > data -- or does it make more sense to be able to reprocess on-the-fly with > any previous version of software if needed? In any event, you want to be > able to "get at" any processing version of any piece of sky, but what's the > best way to enable that? > > Deborah Levine > > At 11:55 AM -0400 7/12/06, Ani Thakar wrote: >> one approach would be to only provide asynchronous (batch) access to the >> older data. a reasonable turnaround would be a day. >> >> ani >> >> On Tue, 11 Jul 2006, Jim Gray wrote: >> >>> Kirk >>> >>> GREAT!! >>> >>> Re the cost of the old copies >>> >>> Every petabyte needs some care an feeding. >>> >>> I assume that MOST of the access will go to the new data. >>> >>> You can limit access to the old stuff by providing limited IO/s and GB/s >>> to the data >>> (e.g. put it on 20TB disks and let people make their >>> own copies if they want more IOps and GB/s than those disks provide). >>> This kind of quota system will encourage all but the really needy to get >>> their own copy or go to the modern stuff. >>> >>> >>> >>> Jim Gray >>> Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 >>> 8222 fax: 425 706 7329 [EMAIL PROTECTED] >>> http://research.Microsoft.com/~gray >>> >>> >>> -----Original Message----- >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Kirk Borne >>> Sent: Tuesday, July 11, 2006 5:01 PM >>> To: Jim Gray >>> Cc: [email protected]; [EMAIL PROTECTED] >>> Subject: RE: [LSST-data] disk space for old releases >>> >>> Thanks Jim. That's great info. I hope that I did not give the >>> impression that I was concerned about the disk *space*. >>> I believe that your vigilant reminders on this theme have sunk in. >>> >>> So, the real question: are there any disk access (QoS) issues related >>> to keeping all of the old releases? >>> >>> - Kirk >>> >>> >>> >>>> Date: Tue, 11 Jul 2006 16:50:41 -0700 >>>> From: Jim Gray <[EMAIL PROTECTED]> >>>> Subject: RE: [LSST-data] disk space for old releases >>>> To: Kirk Borne <[EMAIL PROTECTED]>, >>> LSST Data Management <[email protected]>, >>> [EMAIL PROTECTED] >>>> Cc: Jim Gray <[EMAIL PROTECTED]>, Vik Singh <[EMAIL PROTECTED]> >>>> >>> I think LSST should plan to preserve ANY scientific data it publishes. >>> The cost of collecting the data is orders of magnitude higher than the >>> cost of preserving it. >>> >>> Vik Singh and I are in the process of analyzing how the SkyServer SDSS >>> data products have been used over the last 5 years. >>> (Ani Thakar and Alex Szalay are helping us do this analysis). >>> The 5th official product is just now public -- there was a 6th "early >>> data release" >>> As the graphs below show, there is continuing interest in each of the >>> releases. >>> These are the SQL queries per month. >>> There are about 20x more web hits per month. >>> A more comprehensive document is in preparation, but I think these >>> graphs show that the data products are interesting 5 years into the >>> future. >>> In addition, there are multiple copies of this data stored around the >>> world (China, Japan, US, Germany,...). >>> >>> To harp on my constant theme, it is not disk space that you need to >>> worry about, it is disk accesses per second and disk megabytes per >>> second. >>> >>> >>> <<Picture (Enhanced Metafile)>> >>> >>> >>> Jim Gray >>> Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 >>> 8222 fax: 425 706 7329 [EMAIL PROTECTED] >>> http://research.Microsoft.com/~gray >>> >>> >>> -----Original Message----- >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Kirk Borne >>> Sent: Tuesday, July 11, 2006 4:25 PM >>> To: [EMAIL PROTECTED] >>> Cc: [email protected] >>> Subject: Re: [LSST-data] disk space for old releases >>> >>> Thanks Jacek for the info. >>> >>> I agree with those use cases. As I indicated, astronomers often want to >>> complete their analyses with the same versioning of the calibration >>> pipeline and algorithms. Similarly, they may wish to go back to that >>> version (earlier release) in order to verify or reproduce some >>> previously published results. Thus, people will want the older >>> versions, but (as you say) we have to determine what are our legitimate >>> QoS (Quality of Service) obligations in this regard. >>> >>> - Kirk >>> >>> >>>> Date: Tue, 11 Jul 2006 16:17:17 -0700 >>>> From: Jacek Becla <[EMAIL PROTECTED]> >>>> To: Kirk Borne <[EMAIL PROTECTED]>, >>> LSST Data Management <[email protected]> >>>> Subject: Re: [LSST-data] disk space for old releases >>>> >>>> Kirk >>>> >>>> We will carry forward between releases all the Sources (detections), >>>> but we are not going to carry between releases all versions of all >>>> objects (objects = star or galaxy in deep/coadded catalog). So in >>>> practice, if you base your publication on Object table from DR 2, and >>>> we have on disk DR 3 and 4, in order to reproduce your results you >>>> will need to stage data from DR2. >>>> >>>> Another use case: students/astronomers who want to stay with a given >>>> release and do their analysis on a relatively small data sample for an >>> >>>> extended period of time. >>>> >>>> The answer might be that whoever wants to get data from old releases >>>> need to find a space at his local institute and stage it there, I >>>> don't know. I do know we need to take care of this issue. >>>> Hmm, let's talk about it tomorrow. >>>> >>>> Jacek >>>> >>>> >>>> >>>> >>>> Kirk Borne wrote: >>>>> I guess I am not clear on what would be contained in the older >>> releases. >>>>> >>>>> In some (most?) projects, the newer releases supercede (and >>>>> include) the contents of the older releases. I suppose the older >>>>> releases do include unique calibrated data products that were >>>>> calibrated under some prior version of the data >>>>> processing/calibration algoriths. In that case, I can see some >>>>> utility and value in having the older releases available. >>>>> However, having these releases instantly accessible on spinning >>>>> disks (versus archival backup media) is another issue. >>>>> >>>>> - Kirk >>>>> >>>>> >>>>> >>>>>> Date: Tue, 11 Jul 2006 15:56:10 -0700 >>>>>> From: Jacek Becla <[EMAIL PROTECTED]> >>>>>> Subject: [LSST-data] disk space for old releases >>>>>> To: LSST Data Management <[email protected]> >>>>>> >>>>>> Keywords: DataAccWG >>>>>> >>>>>> Hello, >>>>>> >>>>>> In our current disk storage estimates we are assuming we will need >>>>>> disk space for 2 most recent releases and unreleased catalog. >>>>>> But what about the disk space for older releases that some people >>>>>> might want to stage in from tape? That is not included in the >>> estimates. >>>>>> How much should we reserve for that? Equivalent of size of the most >>>>>> recent release? >>>>>> >>>>>> BTW, if we end up being limited by disk IO and not space, we will >>>>>> get that "space for free" anyway, BUT... it is important to get an >>>>>> idea how much disk space we will need because disk io pushes us >>>>>> towards smaller disks (which are faster, have better seek time), so >>>>>> we can end up in a situation where required number of small disks >>>>>> does not give us enough disk space (I went through the numbers with >>>>>> Don and that can happen). I'm in the process of building a model for >>> >>>>>> that. >>>>>> >>>>>> >>>>>> Jacek >>> >>> _______________________________________________ >>> LSST-data mailing list >>> [email protected] >>> http://www.lsstmail.org/mailman/listinfo/lsst-data >>> >>> _______________________________________________ >>> LSST-data mailing list >>> [email protected] >>> http://www.lsstmail.org/mailman/listinfo/lsst-data >>> >> >> -- >> Aniruddha R. Thakar, Research Scientist >> Center for Astrophysical Sciences, JHU, Bloomberg 375 >> 3701 San Martin Drive, Baltimore MD 21218-2695 >> 410-516-4850, Fax: 410-516-5096 >> [EMAIL PROTECTED], http://www.sdss.jhu.edu/~thakar >> ----------------------------------------------------------------------- >> Black holes are where God divided by zero. [Steven Wright] >> _______________________________________________ >> LSST-data mailing list >> [email protected] >> http://www.lsstmail.org/mailman/listinfo/lsst-data > > > -- > --------------------------------------------------------- > Deborah Levine, Ph.D. (626)395-8567 (Office) > Operations Scientist (626)590-7500 (Mobile) > (626)568-0673 (FAX) > Spitzer Science Center [EMAIL PROTECTED] > Mail Code 314-6, Pasadena, CA 91125 text messaging : [EMAIL PROTECTED] > > "Make everything as simple as possible, but not simpler." > Albert Einstein > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
