Hello all, We absolutely must freeze the disk spreadsheet effective immediately in order to finish our cost estimates. My sense is that we are close enough and so I am locking the database file in the archive as of now, it will remain frozen until our 8/24 baseline at least.
Thanks to all who contributed to this. On to CPUs! Jeff > From: Jacek Becla <[EMAIL PROTECTED]> > Organization: Stanford Linear Accelerator Center > Reply-To: LSST Data Management <[email protected]> > Date: Wed, 19 Jul 2006 16:26:32 -0700 > To: LSST Data Management <[email protected]> > Subject: [LSST-data] new updates to database storage and disk io estimates > > Keywords: DataAccWG > > Hello, > > I made rather significant change in the estimates: > we were forgetting that Deep Detection will run > _for each band_ (which means many more Sources...). > Building this into the model resulted in: > - number of Sources increased ~300% > - db size increased ~130% to 378 TB after the first year > - disk io: number of disks increased ~x2 to 3,922 > - size of RAM needed per processing center shot up > from 6 to 19 TB, without this memory we are talking > about 1 billion (yes, Billion) disks :=) > > The updated docs are in Docushare, Collection-413. > > BTW, all numbers were discussed with Tim and Kem > and approved by Tim. > > > Jacek > > > Jacek Becla wrote: >> Keywords: DataAccWG >> >> Hello, >> >> I refreshed the db storage and disk io estimate documents >> in Docushare, I believe these versions are very close >> to the final now. >> >> The changes include: >> >> storage (size up by 1%): >> - added 1% var galaxies >> >> disk io (number of disks up by 14%, 1,834 now): >> - added reverse estimates (how many queries we can >> handle for given number of disks) >> - fixed issue related to clustering galaxies >> (now that we have much fewer galaxies than stars >> the model should reflect worse hit ratio for galaxies, >> and it was not) >> >> >> Jacek >> >> >> >> Jacek Becla wrote: >>> Keywords: DataAccWG >>> >>> Hello, >>> >>> >>> I published in docushare the latest version of database >>> storage estimates and disk io estimates (Collection-413) >>> It includes latest numbers/assumptions provided by Tim, >>> and the changes discussed/approved at the database >>> telecon (Jul 12). >>> >>> In summary, we now expect >>> - to have 164 TB of data after the first year (down from 600) >>> - to need 1.6 K disks for the assumed query load (down from 4 K) >>> >>> Note that: >>> >>> 1) there is still a small debate over the galaxy numbers >>> for DR20 ("should be more like 18 billion"), and the star >>> numbers ("should be the 10 billion from the r-band counts"), >>> so further updates are likely. >>> >>> 2) We are still working on details regarding which table >>> should be used by which query (DIASource vs Source). This >>> can change disk io (reduce it - now I am using Source) >>> >>> 3) I introduced small data duplication by extracting 6 rows >>> from the Source table into a separate table in order to reduce >>> disk io for the near-neighbor query by 85% (which reduced >>> overall disk requirement by half). That should be ok given >>> we asked for 150% of data size for this sort of >>> "persistent overheads" (and are now "using" 8 TB out of 53 TB >>> in DR1, or 15%) >>> >>> >>> One worry that I have is that the estimates are now for >>> DR1 and in DR1 the Source table is "tiny" (if one can call >>> 40-billion-row table "tiny") comparing to what we will >>> have to deal with later (for example in release 5 >>> it is 10x bigger). The used DR number is selected based on >>> number of objects in Object Catalog: we need 10 billion and >>> DR1 has over 22 already. >>> >>> >>> Jacek >>> _______________________________________________ >>> LSST-data mailing list >>> [email protected] >>> http://www.lsstmail.org/mailman/listinfo/lsst-data >> >> _______________________________________________ >> LSST-data mailing list >> [email protected] >> http://www.lsstmail.org/mailman/listinfo/lsst-data > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
