Hi Jacek,
Sorry I missed the Database telecon. I do want to comment on your
notes regarding the data release sizes. It's not quite correct to
equate the 'time db' to the detection table. Both the time dependent
and the deep detection database are logically part of the object
database. It's best to think of this database having a part that gives
time-integrated information about objects, and a part that gives the
time-dependent information. The bulk of the time-integrated
information is the output of the deep detection pipeline, and mostly
concerns galaxies. But *all* objects have entries in this table,
including stars and solar system objects. If an object is bright
enough to have time-dependent information, this information will be
included by pointing off to the relevant entries in the detection
table. The detection table will have some amount of junk in it,
however, such as detections of noise spikes or unrecognized cosmic
rays. If the association pipeline is doing its job, these junk
entries will be orphans that no entry in the object database links to.
This is where your remark is relevant that the size estimates depend on
the detection threshold. The detection threshold should *not*
significantly affect the size of the object database (within reasonable
bounds, anyway). It *will* affect the size of the detection database,
however! My size estimate was for no junk, so it is optimistic. If
we have significant junk, our size growth will transition to the linear
regime earlier than my plot showed.
We may well wish to periodically clean the detection database of orphans
to limit its size. If we do, it should not be done too quickly.
Today's orphan may be tomorrow's missing point on the orbit of some
solar system object.
Tim
Estimating amount of data per release:
- Tim's numbers only deal with part of the data
- what part?
- time db --> detection table
- deep db --> object table
- data related to simulation
- usually bigger than real data size in HEP
- not very significant fraction in astronomy
- detections and object tables are the largest fraction of database
size, remaining things include near neighbor data, provenance
- near neighbor data: depends on search radius, everybody uses
different, if we apply 90/10 rule and pick radius used by 90%
of users, and store results, then it is ~10% of data size
- currently in baseline requirement: object detection will
be 0.4-4.5 TB/night depending where you point in the sky
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a Database telecon tomorrow at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- database and image archive storage estimates
- DC1 db-related code implementation: progress/update
Relevant url:
https://www.lsstcorp.org/docushare/dsweb/Get/Document-1779/lsst_storage_estimates.xls
(note that we are currently actively working on this spreadsheet,
so it may change between now and tomorrow)
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
begin:vcard
fn:Tim Axelrod
n:Axelrod;Tim
org:;Large Synoptic Survey Telescope
email;internet:[EMAIL PROTECTED]
title:Data Management Project Scientist
tel;work:520-322-8735
version:2.1
end:vcard
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
