Thank you Jeff. I am sinking my teeth into this. I will likely call upon my Data Products WG colleagues for inputs along the way.
- Kirk ----- Original Message ----- From: Jeffrey P Kantor <[EMAIL PROTECTED]> Date: Wednesday, June 7, 2006 11:57 am Subject: Re: [LSST-data] Growth of database size > Hello all, > > We have 2 major sources of requirements for disk I/o: > - Pipelines (read/write) > - End user access/queries (mostly read) > > We understand the first pretty well, but getting a handle on the > second has > been a much bigger problem. So far, we have the following completely > independent models of this access: > > - SDSS history and Sergei's extrapolation of it to queries > - Kem's 7-page list of science examples > - Tim's list of representative Level 3 data products in the Functional > Requirements Spec > - The UML model Community Science use cases > > I have asked Kirk Borne to take the lead in examining these items > and come > up with the "baseline" list of use cases and queries that we will > use to do > sizing for this. It is due at the end of June. I have asked > Chris Smith to > assist Kirk in this endeavor. I would ask the rest of the Data > ProductsWorking Group to offer assistance as well. > > This is a critical hole in our cost estimating that makes me > increasinglynervous. I hope that we will have a good model of > this by the end of the > month. > > Jeff > > > > From: Jacek Becla <[EMAIL PROTECTED]> > > Organization: Stanford Linear Accelerator Center > > Reply-To: LSST Data Management <[email protected]> > > Date: Tue, 06 Jun 2006 15:42:06 -0700 > > To: Jim Gray <[EMAIL PROTECTED]> > > Cc: LSST Data Management <[email protected]> > > Subject: Re: [LSST-data] Growth of database size > > > > Hi Jim, > > > > > > I completely agree with you that the biggest problem will be > disk I/O > > and that we should not underestimate it. The only reason why I > have not > > done the estimates yet is that the input data that I need for such > > exercise is pretty much unknown. To estimate disk IO that a database > > will generate/need, we need to know things like: > > - estimated number of queries > > - types of common queries > > - estimated size of data that these queries will need > > - indexes available and type of indexes > > - how data will be clustered > > - how data will be partitioned > > - access patterns > > - database engine used > > - and others... > > > > As far as I know we do not have the answers or good estimates > for any of > > the issues listed above, so I would not trust the disk IO > numbers that > > we could come up with today. If you have any suggestions how to > > realistically estimate disk IO for LSST today, I'd love to talk > to you! > > > > > > Thanks, > > Jacek > > > > p.s. We could do the disk IO estimate for Data Ingest, but as > > we both know the IO there is not very challenging (assuming we > > partition indexes correctly) > > > > > > > > > > > > Jim Gray wrote: > >> LSST sizing should focus more on disk IO/s than on disk bytes. > >> > >> The database size is definitely an issue -- it tells you how > much disk > >> capacity and how much network bandwidth you need and if you > know the > >> instruction density (instructions per object it implies the cpu > demand).>> > >> So, DB size is VERY good to know. > >> > >> But... In the 2012 world of 10GB/s network links, 10TB disks, > and 100 > >> core processors, these are not the scarce resources (we hope). > >> Indeed, you can afford a 10x cpu cost converting to and from > ASCI/csv>> rather than using binary for data ingest just to have a > simpler>> interface in the pipeline. > >> > >> As far as I can tell the scarce computing resource will be disk IO. > >> > >> Now we are expecting about 12 TB/night. > >> In rough numbers, one disk worth of data per night. > >> Each of the 10TB disks will deliver about 250 IO/s for small random > >> requests, So if we triplex the disks we get about 3x200 = 750 > IO/s to > >> do the processing. > >> The disk bandwidth goes as the square root of the aerial > density so we > >> can expect about 4x more bandwidth or 250MBps. > >> > >> 1e13 nightly LSST bytes written at 2.5e8 bytes/sec is 4e4 > seconds or 10 > >> hours > >> -- so the disks can be written in 10 hours but there is not a > lot of > >> slack to read them. > >> These disks will deliver about 250 TINY random Ios/s. > >> If you do LARGE 1MB reads and writes then the transfer time is > >> significant and the number drops to 125 IO/s > >> > >> page size (B) seek time (ms) transfer time (ms) random > >> transfers/sec > >> 1,000 4.00 0.00 249.75 > >> 10,000 4.00 0.04 247.52 > >> 100,000 4.00 0.40 227.27 > >> 1,000,000 4.00 4.00 125.00 > >> 10,000,000 4.00 40.00 22.73 > >> 100,000,000 4.00 400.00 2.48 > >> > >> And significantly, (again for random Ios) > >> page size (B) Bandwidth (MB/s) > >> 1,000 0 > >> 10,000 2 > >> 100,000 23 > >> 1,000,000 125 > >> 10,000,000 227 > >> 100,000,000 248 > >> > >> So, you should count on the DBMS doing 1MB/s and giving you 125 > >> MBps/disk and using MASSIVE main memory (this is a page size > 100x bigger > >> than today's sizes). > >> Now you are back to needing lots more disks/night or designing > the disk > >> arrays to use all the arms all the time. > >> > >> It is ESSENTIAL that the LSST pay attention to the disk IO/s issue. > >> It will be a gating technology (capacity will not be). > >> The simple way to think of this is just to imagine that each > disk has > >> infinite capacity but delivers only 150 IO/s. > >> The LSST IO/S requirements will imply WAY more disk capacity > than will > >> be needed to just store the data and indices. > >> > >> So these discussions of "database size" are great, > >> But they should all include the IO bandwidth (MB/s) and IO per > second>> requirements. > >> > >> I attach the spreadsheet if you want to try different > parameters on this > >> simple model > >> > >> > >> Jim Gray > >> Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: > 415 778 > >> 8222 fax: 425 706 7329 [EMAIL PROTECTED] > >> http://research.Microsoft.com/~gray > >> > >> > >> -----Original Message----- > >> From: [EMAIL PROTECTED] > >> [mailto:[EMAIL PROTECTED] On Behalf Of Kem Cook > >> Sent: Monday, June 05, 2006 11:55 PM > >> To: [EMAIL PROTECTED]; LSST Data Management > >> Subject: Re: [LSST-data] Growth of database size > >> > >> Hi All, > >> > >> I agree with Tim's esimates, but there are details which > haven't been > >> fleshed out. There are parameters which don't really add > volume to the > >> data, but they are there. The time dependent database needs > motion>> information: parallax, proper motion or orbital > parameters. The time > >> dependent objects will also contain added information in terms > of the > >> likelihood of blendedness, multiplicity and variability parameters. > >> These data are added on a per object basis and as such, do not > >> significantly increase the volume of data, but should not be > forgotten.>> Presumably, these parameters will be present from the > first detection of > >> a time dependent object and will not increase in volume with time. > >> > >> Kem > >> > >> > >>> Hi Jacek, > >>> > >>> I have created a simple model for how the size of the object > database>>> will grow between data releases (DR). Here are my > assumptions:>>> > >>> 1. Data releases occur every 6 months > >>> > >>> 2. We meet our SRD requirements of 100 visits per field per year > >>> > >>> 3. The database is split into two parts. The first, > dominated by > >>> galaxies, contains the static information for every object > detected at > >> > >> > >>> that point in the survey, mostly generated by combining the > >> > >> information > >> > >>> in image stacks. I'll call this the 'deep database' The > second,>>> dominated by stars, contains the time dependent > information for > >> > >> objects > >> > >>> bright enough to be usefully detected in individual exposures. > I'll > >>> call this the 'time dependent database'. > >>> > >>> 4. An object record in the deep database is about 100 bytes: > 6 band > >>> magnitude + errors; data quality flags; shape information. > >>> > >>> 5. An object record in the time dependent database is about > 10 bytes: > >>> 1 band magnitude + error + data quality flags. > >>> > >>> 6. For the first DR, the limiting magnitude for the time > dependent>>> database is 24.5 (where it remains), while the > limiting magnitude for > >>> the deep database is already at about 26.1 from stacking 20 R band > >>> images. So at DR1, there are already about 20 times more > objects in > >>> the deep database than in the time dependent. > >>> > >>> Consider first the growth of the deep database. The limiting > flux to > >>> fixed signal-to-noise will decrease as 1/sqrt(n_exp), where > n_exp is > >> > >> the > >> > >>> number of exposures effectively stacked and used for > detection. I > >>> assume that measurement occurs in all bands, but detection > occurs only > >>> in the R band. The SRD calls for 40 R band exposures per > field per > >>> year, or 20 additional for every DR. The limiting magnitude > >> > >> increases > >> > >>> as 1.25*log (20DR), and we go progressively fainter in the galaxy > >>> brightness distribution. I've taken the galaxy data here > from the > >>> Subaru Deep Field, which gives the slope of the cumulative > brightness>>> distribution to be d(logN)/d(mag) = 0.45 in the > region of interest. > >>> The size of the deep database then grows as 100 * > (20DR)**(0.45 * > >>> 1.25) > >>> > >>> The time dependent database grows strictly linearly with the > number of > >> > >> > >>> observations in all bands, which is 50 per DR, so it goes as > 10 * (50 > >> > >> DR). > >> > >>> Taking account of the factor of 20 difference in number of > objects at > >>> DR1, two attached plots show the growth of the deep database > size,>>> and the growth of both together. The roughly square root > growth of > >>> the deep data dominates the first half of the survey, but is then > >>> overtaken in the second half by the linear growth of the time > >> > >> dependent database. > >> > >>> In spite of my many assumptions, which are unlikely to be > right in > >>> detail, I think the overall behavior is about right. > >>> > >>> Let me know if you see an error or need more information. > >>> > >>> Cheers, > >>> Tim > >>> > >>> > >>> _______________________________________________ > >>> LSST-data mailing list > >>> [email protected] > >>> http://www.lsstmail.org/mailman/listinfo/lsst-data > >>> > >> > >> > >> _______________________________________________ > >> LSST-data mailing list > >> [email protected] > >> http://www.lsstmail.org/mailman/listinfo/lsst-data > >> > >> > >> ---------------------------------------------------------------- > -------- > >> > >> _______________________________________________ > >> LSST-data mailing list > >> [email protected] > >> http://www.lsstmail.org/mailman/listinfo/lsst-data > > > > _______________________________________________ > > LSST-data mailing list > > [email protected] > > http://www.lsstmail.org/mailman/listinfo/lsst-data > > > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
