Keywords: DataAccWG
Attendees:
Russell Owen
Ani Thakar
Maria Nieto-Santisteban
David Fleming
Tim Axelrod
Jacek Becla
database/catalog overheads
==========================
Current plan: x2.0 for overheads:
- 150% for index, stats etc
- 50% for db temp space
- in practice will need 900 TB of disk space to manage
300TB of data in db
- each release will contain "latest" version of objects, can't
effort to keep all previous versions in each release
- Planning to keep 2 releases on disk (1 year worth of data)
- driven by cost constraints
- will keep everything in deep storage
- it is essential to have reliable ways to access
previous releases
- older releases might be available at other sites
What is the size growth between releases?
- non-linear, larger at the beginning, will slow down
towards the end of the survey
- Tim will come up with some numbers / how to model the growth
(by Monday)
Main difference between lsst and sdss: recalibration of all
data will happen much more frequently in lsst
- based on sdss experience recalibration is very expensive,
takes months to complete, chose to document errors rather
then recalibrate often
Growth factor of pre-cached archive data at the Base Facility?
- size will stay the same after initial ramp up: need only
summary data of objects and most recent light curve, so it is
fixed-length "sliding window"
Database software implementation for DC1
========================================
Catalog Ingest Service
- Russell working on implementation in python, almost done
- no major issues/show stoppers
- what to do if files are failing on ingest?
- log the failure, continue ingest, do not delete the files,
files will be examined later
- discuss offline partial ingest (Jacek/Russell)
- would be good if we could have the data that will be
used for ingest in DC1
Data Ingest Converter
- original plan: in C++ for efficiency reasons
--> do it in python for DC1, then will better understand
number of rows to deal with, and performance needed
- Russell/Jacek will talk offline about details
What data will be used for ingest?
- DC1 will use real image data from DLS
(David Wittman is working on it)
- catalog data out of sextractor, pipeline will produce
"some kind of catalog object"
- Sergei schema should do fine for DC1
--> locate the most current copy (Jacek)
Partitioning/slicing pre-cached data
- Maria looking into this
- questions:
- area of sky covered by DC1?
- density of objects?
- size of data?
- expected queries?
- size: < 1TB
- number of objects/field uniform for DLS data
- non-continuous: part in northern sky, part in southern sky,
complicated pattern
--> look at DLS webpage
- don't fine-tune partitioning for DLS data for DC1
- expected queries
- Tim will provide, maybe Kem might help
- in real system will need a signal from OCS with
coordinates of observed area sent few minutes before
the observation
- to load to memory related parts of archived data
- will not have that signal in dc1
AOB
===
we have a reverse-engineered database schema used
by the PanSTARRS PS-MOPS
- not relevant to DC1
- just publish for now
- check with Jeff where in Docushare to put it
Checksum files (see Alex's suggestion)
--> raise this topic at DC1 or middleware telecon
Next meeting (Database telecon) this Wednesday
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a DataAccWG telecon tomorrow at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- database overheads
- database-related DC1 activities
- Ingest Service implementation
- precached archive data partitioning
- schema
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
