Some comments on the minutes. Correction: 34 MBps is 60% the bandwidth of 1 SATA disk It does not need 2 servers or even 2 disks. there is no disk IO bandwidth problem on write, READING 1PB in a day will pose some IO bandwidth problems :) .
But... There are substantial IO per second problems if the IO is random rather than sequential. Most of the pipeline is sequential. But incremental database loading and indexing is random (if we do it in the obvious way). Most of our intellectual energy should go into strategies for doing these incremental operations. (or coming up with a strategy that avoids incremental operations by adopting a massively parallel partitioned design) In our timeframe (2012, we will likely have 5TB disks so that is about a disk pair every 5 days.) Or... 1 pair of 10k$ 120TB 24-disk 3U bricks provisioned every quarter. My bias is to keep most of the archive at the base camp in this way Sorry to keep harping on this -- but simpler is better. RE DB Size inflation: The issue is IOPS no size, but... My load studies of MS SQLserver indicated the DB has a 10 byte per record inflation over raw binary files. (this is page overhead and record overhead. There is also internal fragmentation in pages that can add a few %). Indices typically inflate the database by 30% to 50% depending on how richly it is indexed (indices provide "tag" tables that are "vertical slices" of a table), Incidentally, now that MS SQL has fixed the 16-column limit I expect (1) the number of indices to decline. The PhotoTag table to be replaced by an index. So, I expect the total space to go down (but lets say 30% anyway). Each index costs an IO on each insert (to a first approximation. Since we have an IO problem, we have an index problem. It will require creativity to make this tradeoff. - Jim Gray Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 8222 fax: 425 706 7329 [EMAIL PROTECTED] http://research.Microsoft.com/~gray -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jacek Becla Sent: Friday, April 28, 2006 1:57 PM To: LSST Data Management Subject: [LSST-data] Notes f correction: 34rom Database telecon Fri Apr 28 Keywords: DataAccWG Attendees: Ray, Jeff, Marcus, Ani, Nicole, Sergei, Ramon, Robyn, Jacek Database requirements for DC1 ============================= [discussed before 11:00 at the Jeff's telecon] - io rates not very challenging: 1TB/night = ~34MB/sec - do-able even today with 1-2 decent servers - archived data at base camp: ~100 TB max in real production if we assume we want to bring all data for given section of the sky --> for DC1: use 2 servers for ingest and 2 servers for archive data (1 and 1 would suffice, but want to test multi-server environment) - modest hardware would suffice - replicate archive data across these 2 servers - planned infrastructure outlined by Ray seems ok - would like to see a diagram/picture - something already available in docushare under system engineering section, infrastructure design - need to update, add more wording - mysql version - recently: relatively frequent updates of mysql releases - a "big" release (5) out, still doing bug fixes - might need to use "-max" because we may need federated engine - work on details with Ramon - Ray will set up account for Jacek on teragrid database overheads/disk requirements ==================================== - planned storage requirements: pure data with no overhead - need to fix it, database introduces overheads (internal, indexes) - internal overhead for database: highly dependent on storage engine, - in MySQL, MyISAM table (the one we are planning to use) has zero overhead :=) - InnoDB engine has x3 overhead, improved somewhat in ver 5.1 (5.1 still in beta), have not tried 5.1 at slac yet --> action item for JHU: check SQL Server overheads - look at most common lsst types (float, decimal, int) - checking Source table is a good start - could use packing, provided by some database engines - useful if small number of rows read, bad for frequent full table scans (MySQL can unpack individual rows, ideal for index search + retrieving few data rows) - tried packing 6GB table with 50 million sources - compressed to 3.3GB (44%), - full table scan went up from 2 min to 7 min - might be useful to pack less frequently used data, and use gained disk space for storing more indexes - how much space should we reserve for indexes? - some mysql gurus suggest to reserve initially 60-100% for indexes - their experience is that typical larger applications start with index size = data size (over-index), and gradually reduce index size to ~60% - index overhead in mysql ~standard comparing to other databases - in lsst we will need larger pointers because of peta-scale, so need extra 1-2 bytes per row per index - SDSS experience: index size ~15-20% of data size, but... - ~1/6 of data is duplicated (hot data), - some software limitations used to prevent indexing more than 16 column per table - fixed in yukon, will be adding more indexes soon - LSST needs more indexes because of spatio-temporal aspects - duplicating data blindly not a convincing option - might duplicate some parts of the data, plus build indexes - checking with Jim Gray what his opinion is regarding how much we should reserve for indexes, also looking on the web what others suggest - Jeff needs an estimate soon (next week) - "Jacek's guess": 50% for indexes. Will use that for now for the baseline and update as necessary - action item for Jacek: inform TechAssessWG Where did the "1TB/night" number come from? =========================================== - from discussions related to AAS poster and SPIE paper: - 300TB of catalog data per year is the "final" number approved by management during AAS and SPIE poster/paper reviews - this is worst-case: - 300TB/year < 1TB/night - 300TB includes all catalog data, at the base we will ingest less database servers at Archive Center ================================== Baseline assumes 32 database servers and 1 petabyte of disk storage at the Archive Center - this is for all operations: ingest, merging, and serving community - each server 8 core - driven by "cost cap", not the number of users/queries that we expect to see and support - some concerns (Jacek) that 32 servers might not be enough to serve reasonable number of user queries - worry about changing (increasing) requirements, if we are really going to store "only" 300TB/year, we might be ok - suggesting ~40 servers (a guess) backup ====== - we can't afford a full backup of all data - we have a cost we can't exceed - likely will have 2 copies of data at two different sites which will serve as a backup - fault-tolerant fail over is not expected from us in case of major problems mirroring ingest ================ it might be wise to load data at the base into two independent sets of servers - improved reliability at a very small cost - in case of problem with disk or ingest host or db server, we can immediately fail over to the other server - production ingest will deal with ~34MB/sec, so very small number of ingest nodes will be needed concurrent write/reads ====================== needed at the base: loading data (writer) and association pipeline querying (reader) - current plan: allow both in the same table, loading/reindexing should take < 3 sec, leaving enough time for AP to query data - MySQL allows to write and read concurrently (with MyISAM - the non-transactional engine and table-level locking) - caveat: data can only be appended at the end, no deletions/updates allowed - we will only be appending, so fine - this is mysql-specific, so will not 'pollute' uml with that, but it is a nice feature, many large mysql customers use it, worth using if we choose mysql auto_increment ============== - talked in the past about getting rid of auto_increment --> use it for DC1, think about fixing that after DC1 SPIE database paper =================== submitted few days ago, reviewed by all the authors, final version in docushare: https://www.lsstcorp.org/docushare/dsweb/Get/Document-1638/6270-29.pdf Jacek Jacek Becla wrote: > Keywords: DataAccWG > > Hi, > > We have a Database telecon tomorrow (Friday) at 11:00 PST. > > Phone number: 866 330 1200 > passcode: 300 2363 > > Agenda: > - database requirements for DC1 > - database/index overheads and storage requirements > - AOB > > Note that the first item is likely to be discussed before 11:00, at > the Jeff's Data Management telecon. > > Jacek > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
