Tim,
In between releases we will have on disk "unreleased catalog"
(in addition to two most recently released catalogs). I think
it should be OK to keep updating data in this unreleased catalog.
At some point we used to call it "latest-greatest, use at your
own risk". We should make it very clear that results based
on unreleased catalog will not be reproducible. So going back
to the alertId issue: would it be enough to have 5-months old
"latest alertId" in the released Object catalog, and the real
"latest alertId" in unreleased catalog only?
thanks,
Jacek
Tim Axelrod wrote:
Jacek,
I am not questioning the need for a data release to be a static entity -
I have in fact argued for it, and that assumption has been explicit in
all of our storage sizing. The issue is rather what happens *in
between* releases. If we cannot have an Object table which gives us
the state of our knowledge at a given moment, I think a number of
science usecases will be compromised. In particular, an observatory
dedicated to getting information on transients out in 30 seconds should
not then offer up only an Object table which is six months old!
So, I think we need to be able to update, or a functional equivalent.
Tim
Jacek Becla wrote:
Jim,
Thanks! I am glad I am not alone in pushing the idea
of read-only data =)
What you said is exactly how I understood this problem:
- released/published data is never updated
- we may have some freedom in updating unreleased data
- reprocessing all data is a way to 'refresh' the data,
and will make old releases "fade away"
- we have to use versioning to track changes for released data
Tim:
It sounds like we should dedicate at least one telecon to
these issues and better understand what the implications
of making published data immutable are. Do you want to wait
until September or we should try doing it sooner?
Would you be able to prepare a list of things that will
be "unworkable"? (Based on BaBar's experience I strongly
believe published data should be read only, and it looks
like Jim does too)
thanks,
Jacek
Jim Gray wrote:
there is "science data" that is never updated (level 0 uncalibrated,
level 1 calibrated/derived). Pixels are level 0 and object catalogs are
level 1.
The alerts and summaries and etc are derived data products (level 2).
They can be re-derived on the fly from level 0-1 data if needed. But,
once "published" you should keep data so that other scientists can
reproduce the results (that is one of the "rules" of science).
Level 0 data are never updated. Derived data can be updated (aka-
discarded (update is equivalent to
delete-insert). )
Summary data is often updated (but this is often best done in versions.)
The SDSS went to a version scheme (one version per year) to offer
consistently processed data. (it is hard to deal with data where the
meanings of variables changes over time).
LSST does not have that luxury, and will have to confront this problem
head on (since it is real time).
My guess is that LSST will publish editions where the old data is
reprocessed in the new way. The old editions will be frozen and the
newly arriving data will be
added to the new edition. Summary data can be versioned on a daily
basis.
These editions give polynomial growth in storage space (above the simple
linear growth you get as time passes) -- so called "data inflation".
This is all doable in a world of infinite sized disks (i.e 10TB disk
drives).
To repeat my song -- we do not have a space problem, we have a
access-per-second problem. There is NO problem keeping all versions.
In your timeframe disks are 100x their current size.
Jim Gray
Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778
8222 fax: 425 706 7329 [EMAIL PROTECTED]
http://research.Microsoft.com/~gray
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Tim Axelrod
Sent: Friday, August 18, 2006 5:43 AM
To: Jacek Becla
Cc: LSST Data Management; [EMAIL PROTECTED]
Subject: Re: [LSST-data] Re: cpu for queries
Jacek,
Ouch! I had forgotten that you wanted to do this. I'm afraid this
makes more than just the alertid proposal unworkable. The whole concept
of having summary information for the object, such as a wavelet
decomposition of its lightcurve or even the mean magnitude, that is
updated as we go along is gone too. Living with this would be very
difficult! What alternatives do we have?
Tim
Jacek Becla wrote:
Kirk, Ani
There is one small complication with keeping the most recent alertid
for each Object: it requires updating existing rows.
It would be nice to have the design of largest tables such that each
row is written one, and kept read-only after that.
Such approach guarantees reproducibility, and make it more
manageable (easier to replicate and distribute).
Changes/updates would be done through versioning, but I'm not sure
it is worth using versioning in this case.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
