Keywords: DataAccWG
Attendees:
Tim Axelrod
Ray Plante
Kem Cook
Ani Thakar
Maria Nieto-Santisteban
Kirk Borne
Jacek Becla
database size estimates
=======================
time db growth too rapid? (shouldn't the number of objects
be constant after certain time of rapid growth?)
- each new release should contain _all_ detections, not only
new detections
- one of the reasons why: we will want to run Detection Pipeline
in each release with latest algorithms
- that is not too bad, the rapid growth would mean we will need
more deep storage space, it will not increase size of disk
space needed too much
- conclusion: Jacek's spreadsheet is fine
How does the bottom-up estimate compares to top down now?
- top down: 300 TB in first year
- bottom up: 68 TB in first year
- pretty realistic
- the top down was conservative (meaning it tried to
reserve too much for catalog)
Should we keep all 3 _unreleased_ catalogs on disk
(deep, temp, moving objects)?
- yes for temporal
- yes for moving objects
- no for deep (deep db is based on stack images which will be
produced infrequently: e.g. once per every release)
BTW, is the separation into 3 catalogs: deep, time and
moving objects ok?
- yes, reasonable
Should we keep these catalogs separate? If so, then we would
have to duplicate some data
- catalog is highly processed information, it a product to do
science at certain level, it is scientifically self contained,
it is a set of tables, but it is not necessarily a self
contained database
- having e.g. links from moving object catalog to detections
is a quite reasonable approach
- there might be a reason to replicate just a catalog,
without detections
- detections that the catalog points to must be immutable
the size of database might change +/- factor of two depending on
the final observing strategy: how much we delve in galaxies
- will not know what that strategy is until we are on the sky
- the number we have is reasonable for the construction proposal
Conclusion:
- once we add database overheads on top, we will use the spreadsheet
as baseline
[but see Zeljko's email, we should address it....]
hypothetical end user access workload
=====================================
"200 low volume queries"
- will assume each query goes over 10 million objects
- will assume each query retrieves 1 Gbyte
- should each query see 100Mbps?
- not sure, try approach: how many simultaneous queries seems
reasonable from disk point of view
200 low volume queries talks about querying catalog _or_ image archive
1) What do we mean by "query image archive"?
- browsing catalog and getting postage stamps (small iconic
representation of image), so all queries should start from
querying catalog
Should we keep postage stamps in database? For each object?
For each field? Or maybe generate them on the fly?
- Sloan:
- the Frame table is ~8% of total data (including indexes)
- many users only use image cut out, e.g. for finding charts,
navigations
- main complains from users: don't have images for
individual filters
- Sloan's experience shows we should be able to do it on the fly
- but maybe we should assume that automated tools doing
pattern recognition will process these images?
- that kind of use is infrequent
- conclusion: store in database small jpegs for entire sky or
per field, we should have a service that cuts out pieces of image
on the fly from these stored jpegs
- this is very small amount of extra data in db:
~2000 fields x ~200K per jpeg, all this once per release
- should we store individual jpegs for individual colors?
- not sure
- combining colors on the fly is not so simple
- who will be responsible for maintaining the knowledge about
the postage-stamps related decisions made here?
- Tim will write down what we decided
- Kem will incorporate this into DataProdWG
Now we really need some real SQL queries to proceed
with the disk io estimates
- 2, 3, at most 10 classes of queries should be sufficient,
then just use different coordinates
- will need to identify which queries to use for "low volume" type,
which for "high-volume", and which query is "super-high-volume"
Thanks,
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a DataAccWG telecon tomorrow at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
The agenda:
- database size estimates
- hypothetical end user access workload
Here is the latest version of the db size estimates
Kem and I put together:
http://ctiowo.ctio.noao.edu:8204/wgroups/dataaccesswg/technicalDiscussions/lsst_storage_estimates_v07.xls
I'll put it in docushare once I add database overheads
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
