one approach would be to only provide asynchronous (batch) access to the
older data. a reasonable turnaround would be a day.
ani
On Tue, 11 Jul 2006, Jim Gray wrote:
> Kirk
>
> GREAT!!
>
> Re the cost of the old copies
>
> Every petabyte needs some care an feeding.
>
> I assume that MOST of the access will go to the new data.
>
> You can limit access to the old stuff by providing limited IO/s and GB/s
> to the data
> (e.g. put it on 20TB disks and let people make their
> own copies if they want more IOps and GB/s than those disks provide).
> This kind of quota system will encourage all but the really needy to get
> their own copy or go to the modern stuff.
>
>
>
> Jim Gray
> Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778
> 8222 fax: 425 706 7329 [EMAIL PROTECTED]
> http://research.Microsoft.com/~gray
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Kirk Borne
> Sent: Tuesday, July 11, 2006 5:01 PM
> To: Jim Gray
> Cc: [email protected]; [EMAIL PROTECTED]
> Subject: RE: [LSST-data] disk space for old releases
>
> Thanks Jim. That's great info. I hope that I did not give the
> impression that I was concerned about the disk *space*.
> I believe that your vigilant reminders on this theme have sunk in.
>
> So, the real question: are there any disk access (QoS) issues related
> to keeping all of the old releases?
>
> - Kirk
>
>
>
> > Date: Tue, 11 Jul 2006 16:50:41 -0700
> > From: Jim Gray <[EMAIL PROTECTED]>
> > Subject: RE: [LSST-data] disk space for old releases
> > To: Kirk Borne <[EMAIL PROTECTED]>,
> LSST Data Management <[email protected]>,
> [EMAIL PROTECTED]
> > Cc: Jim Gray <[EMAIL PROTECTED]>, Vik Singh <[EMAIL PROTECTED]>
> >
> I think LSST should plan to preserve ANY scientific data it publishes.
> The cost of collecting the data is orders of magnitude higher than the
> cost of preserving it.
>
> Vik Singh and I are in the process of analyzing how the SkyServer SDSS
> data products have been used over the last 5 years.
> (Ani Thakar and Alex Szalay are helping us do this analysis).
> The 5th official product is just now public -- there was a 6th "early
> data release"
> As the graphs below show, there is continuing interest in each of the
> releases.
> These are the SQL queries per month.
> There are about 20x more web hits per month.
> A more comprehensive document is in preparation, but I think these
> graphs show that the data products are interesting 5 years into the
> future.
> In addition, there are multiple copies of this data stored around the
> world (China, Japan, US, Germany,...).
>
> To harp on my constant theme, it is not disk space that you need to
> worry about, it is disk accesses per second and disk megabytes per
> second.
>
>
> <<Picture (Enhanced Metafile)>>
>
>
> Jim Gray
> Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778
> 8222 fax: 425 706 7329 [EMAIL PROTECTED]
> http://research.Microsoft.com/~gray
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Kirk Borne
> Sent: Tuesday, July 11, 2006 4:25 PM
> To: [EMAIL PROTECTED]
> Cc: [email protected]
> Subject: Re: [LSST-data] disk space for old releases
>
> Thanks Jacek for the info.
>
> I agree with those use cases. As I indicated, astronomers often want to
> complete their analyses with the same versioning of the calibration
> pipeline and algorithms. Similarly, they may wish to go back to that
> version (earlier release) in order to verify or reproduce some
> previously published results. Thus, people will want the older
> versions, but (as you say) we have to determine what are our legitimate
> QoS (Quality of Service) obligations in this regard.
>
> - Kirk
>
>
> > Date: Tue, 11 Jul 2006 16:17:17 -0700
> > From: Jacek Becla <[EMAIL PROTECTED]>
> > To: Kirk Borne <[EMAIL PROTECTED]>,
> LSST Data Management <[email protected]>
> > Subject: Re: [LSST-data] disk space for old releases
> >
> > Kirk
> >
> > We will carry forward between releases all the Sources (detections),
> > but we are not going to carry between releases all versions of all
> > objects (objects = star or galaxy in deep/coadded catalog). So in
> > practice, if you base your publication on Object table from DR 2, and
> > we have on disk DR 3 and 4, in order to reproduce your results you
> > will need to stage data from DR2.
> >
> > Another use case: students/astronomers who want to stay with a given
> > release and do their analysis on a relatively small data sample for an
>
> > extended period of time.
> >
> > The answer might be that whoever wants to get data from old releases
> > need to find a space at his local institute and stage it there, I
> > don't know. I do know we need to take care of this issue.
> > Hmm, let's talk about it tomorrow.
> >
> > Jacek
> >
> >
> >
> >
> > Kirk Borne wrote:
> > > I guess I am not clear on what would be contained in the older
> releases.
> > >
> > > In some (most?) projects, the newer releases supercede (and
> > > include) the contents of the older releases. I suppose the older
> > > releases do include unique calibrated data products that were
> > > calibrated under some prior version of the data
> > > processing/calibration algoriths. In that case, I can see some
> > > utility and value in having the older releases available.
> > > However, having these releases instantly accessible on spinning
> > > disks (versus archival backup media) is another issue.
> > >
> > > - Kirk
> > >
> > >
> > >
> > >>Date: Tue, 11 Jul 2006 15:56:10 -0700
> > >>From: Jacek Becla <[EMAIL PROTECTED]>
> > >>Subject: [LSST-data] disk space for old releases
> > >>To: LSST Data Management <[email protected]>
> > >>
> > >>Keywords: DataAccWG
> > >>
> > >>Hello,
> > >>
> > >>In our current disk storage estimates we are assuming we will need
> > >>disk space for 2 most recent releases and unreleased catalog.
> > >>But what about the disk space for older releases that some people
> > >>might want to stage in from tape? That is not included in the
> estimates.
> > >>How much should we reserve for that? Equivalent of size of the most
> > >>recent release?
> > >>
> > >>BTW, if we end up being limited by disk IO and not space, we will
> > >>get that "space for free" anyway, BUT... it is important to get an
> > >>idea how much disk space we will need because disk io pushes us
> > >>towards smaller disks (which are faster, have better seek time), so
> > >>we can end up in a situation where required number of small disks
> > >>does not give us enough disk space (I went through the numbers with
> > >>Don and that can happen). I'm in the process of building a model for
>
> > >>that.
> > >>
> > >>
> > >>Jacek
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
>
--
Aniruddha R. Thakar, Research Scientist
Center for Astrophysical Sciences, JHU, Bloomberg 375
3701 San Martin Drive, Baltimore MD 21218-2695
410-516-4850, Fax: 410-516-5096
[EMAIL PROTECTED], http://www.sdss.jhu.edu/~thakar
-----------------------------------------------------------------------
Black holes are where God divided by zero. [Steven Wright]
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
