Keywords: DataAccWG
Attendees:
Tim Axelrod
David Fleming
Kem Cook
Nicole Silvestri
Robyn Allsman
Maria Nieto-Santisteban
Ani Thakar
Jacek Becla
Ingest at the base camp
=======================
[issue raised by Tim]
- do we need database ingest at the Base Camp at all?
- have epic, can run detection & assoc pipeline without
ingesting data to database
- will redo the ingest at the main archive anyway
[Kem]
- need to look carefully at science, eg. supernova case
- need to merge current image data with archive data and
with data taken some time ago during the current night,
could keep that data in flat files but is it convenient?
base camp db architecture
=========================
- see the emails "Base Camp Database requirements" yesterday
- assumptions:
- catalog size 10TB
- upper bound, could be 5TB
- data needed per image: 5GB
- how estimated?
- Expected number of objects/field: 10^7 objects
(high end, including all objects, galaxies, coadds)
- guessed 100 bytes/object
- That is 1 GB per color, x 5 colors
- data reuse: 100% for second image from any pair,
negligible for next images
- accepted latency to join: 10 sec for the first image,
1 sec for the second
--> we need to look at the timeline for the whole 60 sec latency,
Don started doing it some time ago.
- the latency database is allowed to introduce is
directly related to how much hardware will be needed
- need queries, would need them ~end of next week, Tim will
try to write them
- many small tables vs index whole 10TB table?
- Maria/Ani thinking about slightly different architecture,
details unclear, they will try to write short summary
and then we can look at the cost/complexity and compare
disk space requirements
=======================
- last week agreed on 50% for indexes
- based on recent statistics prepared for us by Ani,
in SDSS:
- main contributor to the overhead is index (~30%)
- other tables that are needed to do certain queries
don't add much,
- temp space - depends what we allow users to do,
~10% seems reasonable assumption
- need temp space to build indexes during ingest, disk
_space_ (don't confuse with disk IO) on ingest nodes
should not be an issue, we are ingesting relatively small
amounts of data
--> don't need to do any new radical requests to what was
requested
implementing db software for DC1
================================
- need to start planning who is doing what and when
- need to implement Data Ingest Service (scripts, probably python)
and Data Ingest Converter (C++)
missing Data Access use cases
=============================
- Querying use case
- still in very primitive stage
- added some contents to use case description, and new usecases
e.g. Catching Hostile Queries, Estimating Query cost
- need to discuss how much effort we want to do regarding
estimating query cost, caching query results, etc
- will discuss this topic in the near future
- Admin use cases
- a new use case in uml "Administer and Maintain Data"
in "Sys Admin and Operations" package
- for things like backup, replication, dealing with hot spots...
- again, need to discuss details in the near future
logistics
=========
- should we continue with status reports?
- Andromeda work now all visible through mailing list
- yes, status are useful
- have to supply input to monthly reports anyway, so
periodic summaries for DataAccWG don't add up much
extra work, as that can be reused for monthly reports
--> will continue
- DataAccWG telecon focused mostly on database related issues,
should we bring up more file system issues here?
- yes, periodically have a telecon focused on file systems
- organize one next Friday, Jacek likely unavailable,
David will discuss file systems, we should bring people
from US Davis to give us update on google file system tests
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
Reminder: we have a DataAccWG telecon tomorrow at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Fairly packed agenda this time:
1) disk space requirements
- what is still missing in our estimates
(SDSS: 300 million of 2kB records --> 600GB.
That is the size of PhotoObj, but the whole
db DR4 is 1.8TB due to indexes, spectroscopic
db, usnob, First, 2Mass, Stetson. Then there
is freespace, tempspace)
2) base camp db architecture
- I/O needed for joining Ingest Tables with
pre-cached archived data
- joining remote tables
3) implementing db software for DC1
(Data Ingest Service, Data Ingest Converter,
who is doing what and when?)
4) missing Data Access use cases
- Querying use case (still in very primitive stage)
- Admin use cases (e.g. backup, dealing with hot spots)
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
