Hi,
I'm just the peanut gallery here, but with the volumes of data LSST is
going to produce -- does it make more sense to store all the old processed
data -- or does it make more sense to be able to reprocess on-the-fly with
any previous version of software if needed? In any event, you want to be
able to "get at" any processing version of any piece of sky, but what's the
best way to enable that?
Deborah Levine
At 11:55 AM -0400 7/12/06, Ani Thakar wrote:
one approach would be to only provide asynchronous (batch) access to the
older data. a reasonable turnaround would be a day.
ani
On Tue, 11 Jul 2006, Jim Gray wrote:
Kirk
GREAT!!
Re the cost of the old copies
Every petabyte needs some care an feeding.
I assume that MOST of the access will go to the new data.
You can limit access to the old stuff by providing limited IO/s and GB/s
to the data
(e.g. put it on 20TB disks and let people make their
own copies if they want more IOps and GB/s than those disks provide).
This kind of quota system will encourage all but the really needy to get
their own copy or go to the modern stuff.
Jim Gray
Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778
8222 fax: 425 706 7329 [EMAIL PROTECTED]
http://research.Microsoft.com/~gray
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Kirk Borne
Sent: Tuesday, July 11, 2006 5:01 PM
To: Jim Gray
Cc: [email protected]; [EMAIL PROTECTED]
Subject: RE: [LSST-data] disk space for old releases
Thanks Jim. That's great info. I hope that I did not give the
impression that I was concerned about the disk *space*.
I believe that your vigilant reminders on this theme have sunk in.
So, the real question: are there any disk access (QoS) issues related
to keeping all of the old releases?
- Kirk
> Date: Tue, 11 Jul 2006 16:50:41 -0700
> From: Jim Gray <[EMAIL PROTECTED]>
> Subject: RE: [LSST-data] disk space for old releases
> To: Kirk Borne <[EMAIL PROTECTED]>,
LSST Data Management <[email protected]>,
[EMAIL PROTECTED]
> Cc: Jim Gray <[EMAIL PROTECTED]>, Vik Singh <[EMAIL PROTECTED]>
>
I think LSST should plan to preserve ANY scientific data it publishes.
The cost of collecting the data is orders of magnitude higher than the
cost of preserving it.
Vik Singh and I are in the process of analyzing how the SkyServer SDSS
data products have been used over the last 5 years.
(Ani Thakar and Alex Szalay are helping us do this analysis).
The 5th official product is just now public -- there was a 6th "early
data release"
As the graphs below show, there is continuing interest in each of the
releases.
These are the SQL queries per month.
There are about 20x more web hits per month.
A more comprehensive document is in preparation, but I think these
graphs show that the data products are interesting 5 years into the
future.
In addition, there are multiple copies of this data stored around the
world (China, Japan, US, Germany,...).
To harp on my constant theme, it is not disk space that you need to
worry about, it is disk accesses per second and disk megabytes per
second.
<<Picture (Enhanced Metafile)>>
Jim Gray
Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778
8222 fax: 425 706 7329 [EMAIL PROTECTED]
http://research.Microsoft.com/~gray
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Kirk Borne
Sent: Tuesday, July 11, 2006 4:25 PM
To: [EMAIL PROTECTED]
Cc: [email protected]
Subject: Re: [LSST-data] disk space for old releases
Thanks Jacek for the info.
I agree with those use cases. As I indicated, astronomers often want to
> complete their analyses with the same versioning of the calibration
pipeline and algorithms. Similarly, they may wish to go back to that
version (earlier release) in order to verify or reproduce some
previously published results. Thus, people will want the older
versions, but (as you say) we have to determine what are our legitimate
QoS (Quality of Service) obligations in this regard.
- Kirk
> Date: Tue, 11 Jul 2006 16:17:17 -0700
> From: Jacek Becla <[EMAIL PROTECTED]>
> To: Kirk Borne <[EMAIL PROTECTED]>,
LSST Data Management <[email protected]>
> Subject: Re: [LSST-data] disk space for old releases
>
> Kirk
>
> We will carry forward between releases all the Sources (detections),
> but we are not going to carry between releases all versions of all
> objects (objects = star or galaxy in deep/coadded catalog). So in
> practice, if you base your publication on Object table from DR 2, and
> we have on disk DR 3 and 4, in order to reproduce your results you
> will need to stage data from DR2.
>
> Another use case: students/astronomers who want to stay with a given
> release and do their analysis on a relatively small data sample for an
> extended period of time.
>
> The answer might be that whoever wants to get data from old releases
> need to find a space at his local institute and stage it there, I
> don't know. I do know we need to take care of this issue.
> Hmm, let's talk about it tomorrow.
>
> Jacek
>
>
>
>
> Kirk Borne wrote:
> > I guess I am not clear on what would be contained in the older
releases.
> >
> > In some (most?) projects, the newer releases supercede (and
> > include) the contents of the older releases. I suppose the older
> > releases do include unique calibrated data products that were
> > calibrated under some prior version of the data
> > processing/calibration algoriths. In that case, I can see some
> > utility and value in having the older releases available.
> > However, having these releases instantly accessible on spinning
> > disks (versus archival backup media) is another issue.
> >
> > - Kirk
> >
> >
> >
> >>Date: Tue, 11 Jul 2006 15:56:10 -0700
> >>From: Jacek Becla <[EMAIL PROTECTED]>
> >>Subject: [LSST-data] disk space for old releases
> >>To: LSST Data Management <[email protected]>
> >>
> >>Keywords: DataAccWG
> >>
> >>Hello,
> >>
> >>In our current disk storage estimates we are assuming we will need
> >>disk space for 2 most recent releases and unreleased catalog.
> >>But what about the disk space for older releases that some people
> >>might want to stage in from tape? That is not included in the
estimates.
> >>How much should we reserve for that? Equivalent of size of the most
> >>recent release?
> >>
> >>BTW, if we end up being limited by disk IO and not space, we will
> >>get that "space for free" anyway, BUT... it is important to get an
> >>idea how much disk space we will need because disk io pushes us
> >>towards smaller disks (which are faster, have better seek time), so
> >>we can end up in a situation where required number of small disks
> >>does not give us enough disk space (I went through the numbers with
> >>Don and that can happen). I'm in the process of building a model for
> >>that.
> >>
> >>
> >>Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
--
Aniruddha R. Thakar, Research Scientist
Center for Astrophysical Sciences, JHU, Bloomberg 375
3701 San Martin Drive, Baltimore MD 21218-2695
410-516-4850, Fax: 410-516-5096
[EMAIL PROTECTED], http://www.sdss.jhu.edu/~thakar
-----------------------------------------------------------------------
Black holes are where God divided by zero. [Steven Wright]
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
--
---------------------------------------------------------
Deborah Levine, Ph.D. (626)395-8567 (Office)
Operations Scientist (626)590-7500 (Mobile)
(626)568-0673 (FAX)
Spitzer Science Center [EMAIL PROTECTED]
Mail Code 314-6, Pasadena, CA 91125 text messaging : [EMAIL PROTECTED]
"Make everything as simple as possible, but not simpler."
Albert Einstein
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
