Keywords: DataAccWG
Attendees
Russell Owen
Ray Plante
Ani Thakar
Maria Nieto-Santisteban
Sergei Nikolaev
Kem Cook
Arun Jagatheesan
Jacek Becla
database and image archive storage estimates
============================================
We need to decide "how deep into the noise we want to dig"
- depending on that, the numbers provided by Tim might change
"latest-greatest" release
- better name? --> "unreleased"
- yes, we definitely need disk space for unreleased data
and provide quick access
- the spreadsheet currently assumes size of latest-greatest
data = size of last release, will fix it
how to do science if only 2 releases on disk?
- working on theses ~3 years --> 6 releases
- if big demand from community for older releases,
community might be able to help and provide disk space
- storage should be cheap enough to allow for making
local copies of an older release or a part of release
might be cheaper to rebuild index than to bring it from tape
- perhaps if we hand data to "someone else", it is easier
if no extra steps like rebuilding indexes are required
- we will have mechanisms in place to automatically "patch"
data while bringing it from tape, so not a real issue
estimated deep storage size needed for all releases of
catalog ~30PB
- estimate for images:~60 PB
- db: 1/3 of total deep storage
- mostly due to the fact that each release bigger
- that is roughly consistent with sdss, where image data
x3-4 bigger than database size
calibrated images will be cached for certain amount of time on disk
disk space calculation for image data in separate document now
in docushare
--> need to decide how to merge the db-spreadsheet (which
also contains deep storage info for images now) with
the Ray's document
- might turn info from Ray's document into spreadsheet
- would be nice to have all numbers for disk and deep storage
sizing in one place
- Ray/Jacek discuss offline
calibration images can easily be many times the size of raw images
- are we planning to store them?
- do we want to store complete bias images?
what about darks, flats?
--> need to understand that
image sizes: flat 18TB/night?
- yes, that is a fair assumption
1 PB in 2012 for archive?
- no, assume 0 in 2012, 2012 is for putting the system together
allotment of storage starts in 2013 based on current baseline
Estimating amount of data per release:
- Tim's numbers only deal with part of the data
- what part?
- time db --> detection table
- deep db --> object table
- data related to simulation
- usually bigger than real data size in HEP
- not very significant fraction in astronomy
- detections and object tables are the largest fraction of database
size, remaining things include near neighbor data, provenance
- near neighbor data: depends on search radius, everybody uses
different, if we apply 90/10 rule and pick radius used by 90%
of users, and store results, then it is ~10% of data size
- currently in baseline requirement: object detection will
be 0.4-4.5 TB/night depending where you point in the sky
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a Database telecon tomorrow at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- database and image archive storage estimates
- DC1 db-related code implementation: progress/update
Relevant url:
https://www.lsstcorp.org/docushare/dsweb/Get/Document-1779/lsst_storage_estimates.xls
(note that we are currently actively working on this spreadsheet,
so it may change between now and tomorrow)
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
