Keywords: DataAccWG
Attendees: Ray, Jeff, Marcus, Ani, Nicole, Sergei, Ramon, Robyn, Jacek
Database requirements for DC1
=============================
[discussed before 11:00 at the Jeff's telecon]
- io rates not very challenging: 1TB/night = ~34MB/sec
- do-able even today with 1-2 decent servers
- archived data at base camp: ~100 TB max in real production if
we assume we want to bring all data for given section of the sky
--> for DC1: use 2 servers for ingest and 2 servers for archive data
(1 and 1 would suffice, but want to test multi-server environment)
- modest hardware would suffice
- replicate archive data across these 2 servers
- planned infrastructure outlined by Ray seems ok
- would like to see a diagram/picture
- something already available in docushare under
system engineering section, infrastructure design
- need to update, add more wording
- mysql version
- recently: relatively frequent updates of mysql releases
- a "big" release (5) out, still doing bug fixes
- might need to use "-max" because we may need federated engine
- work on details with Ramon
- Ray will set up account for Jacek on teragrid
database overheads/disk requirements
====================================
- planned storage requirements: pure data with no overhead
- need to fix it, database introduces overheads (internal, indexes)
- internal overhead for database: highly dependent on storage engine,
- in MySQL, MyISAM table (the one we are planning to use)
has zero overhead :=)
- InnoDB engine has x3 overhead, improved somewhat in ver 5.1
(5.1 still in beta), have not tried 5.1 at slac yet
--> action item for JHU: check SQL Server overheads
- look at most common lsst types (float, decimal, int)
- checking Source table is a good start
- could use packing, provided by some database engines
- useful if small number of rows read, bad for frequent
full table scans (MySQL can unpack individual rows, ideal
for index search + retrieving few data rows)
- tried packing 6GB table with 50 million sources
- compressed to 3.3GB (44%),
- full table scan went up from 2 min to 7 min
- might be useful to pack less frequently used data,
and use gained disk space for storing more indexes
- how much space should we reserve for indexes?
- some mysql gurus suggest to reserve initially 60-100% for indexes
- their experience is that typical larger applications
start with index size = data size (over-index), and
gradually reduce index size to ~60%
- index overhead in mysql ~standard comparing to other databases
- in lsst we will need larger pointers because of peta-scale,
so need extra 1-2 bytes per row per index
- SDSS experience: index size ~15-20% of data size, but...
- ~1/6 of data is duplicated (hot data),
- some software limitations used to prevent indexing more
than 16 column per table
- fixed in yukon, will be adding more indexes soon
- LSST needs more indexes because of spatio-temporal aspects
- duplicating data blindly not a convincing option
- might duplicate some parts of the data, plus build indexes
- checking with Jim Gray what his opinion is regarding how much
we should reserve for indexes, also looking on the web what
others suggest
- Jeff needs an estimate soon (next week)
- "Jacek's guess": 50% for indexes. Will use that for now
for the baseline and update as necessary
- action item for Jacek: inform TechAssessWG
Where did the "1TB/night" number come from?
===========================================
- from discussions related to AAS poster and SPIE paper:
- 300TB of catalog data per year is the "final" number
approved by management during AAS and SPIE poster/paper reviews
- this is worst-case:
- 300TB/year < 1TB/night
- 300TB includes all catalog data, at the base we will
ingest less
database servers at Archive Center
==================================
Baseline assumes 32 database servers and 1 petabyte of disk storage
at the Archive Center
- this is for all operations: ingest, merging, and serving community
- each server 8 core
- driven by "cost cap", not the number of users/queries that
we expect to see and support
- some concerns (Jacek) that 32 servers might not be enough
to serve reasonable number of user queries
- worry about changing (increasing) requirements, if we are
really going to store "only" 300TB/year, we might be ok
- suggesting ~40 servers (a guess)
backup
======
- we can't afford a full backup of all data
- we have a cost we can't exceed
- likely will have 2 copies of data at two different sites
which will serve as a backup
- fault-tolerant fail over is not expected from us in case
of major problems
mirroring ingest
================
it might be wise to load data at the base into two independent
sets of servers
- improved reliability at a very small cost
- in case of problem with disk or ingest host or db server,
we can immediately fail over to the other server
- production ingest will deal with ~34MB/sec, so
very small number of ingest nodes will be needed
concurrent write/reads
======================
needed at the base: loading data (writer) and association pipeline
querying (reader)
- current plan: allow both in the same table, loading/reindexing
should take < 3 sec, leaving enough time for AP to query data
- MySQL allows to write and read concurrently (with MyISAM - the
non-transactional engine and table-level locking)
- caveat: data can only be appended at the end, no
deletions/updates allowed
- we will only be appending, so fine
- this is mysql-specific, so will not 'pollute' uml with that,
but it is a nice feature, many large mysql customers use it,
worth using if we choose mysql
auto_increment
==============
- talked in the past about getting rid of auto_increment
--> use it for DC1, think about fixing that after DC1
SPIE database paper
===================
submitted few days ago, reviewed by all the authors, final version
in docushare:
https://www.lsstcorp.org/docushare/dsweb/Get/Document-1638/6270-29.pdf
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We have a Database telecon tomorrow (Friday) at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- database requirements for DC1
- database/index overheads and storage requirements
- AOB
Note that the first item is likely to be discussed
before 11:00, at the Jeff's Data Management telecon.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
