Jacek: can you tell me ... when you sent around the "top 20" LSST queries document, you labeled the queries as Low Volume or High Volume... were these labels the ones that Kem and I used, or were these labels re-assigned to the queries after you learned just how big or how small the queries actually were? I think we should label them according to your estimated sizes, not according to Kem and my guesses about their sizes. This is particularly relevant for those cases where (I know, for my own use cases) I misunderstood the gravity of joining to some very large tables or joining to all sky partitions.... this made some of my low-volume queries into high-volume. And so these queries should be labeled appropriately as high-volume.
So, I am just wondering if your list included those updated assignments (low versus high)? p.s. I also noted that one of the positional queries still had the multiplicative cos(DECL) factor in the R.A. condition, when it should be divided by that factor. This was in the high-volume query "Find variable stars close to hot stars". Thanks. - Kirk > Date: Mon, 03 Jul 2006 19:46:12 -0700 > From: Jacek Becla <[EMAIL PROTECTED]> > Subject: Re: [LSST-data] database disk io estimates > To: LSST Data Management <[email protected]> > > Keywords: DataAccWG > > > Hello, > > > 1) > I found one mistake in the original estimate for non-split Object table. > Fixing the problem brought down the number of disk heads: > - indexes not in memory 166K --> 140K > - indexes in memory: 44K --> 16K > > 2) > I redid the disk io spreadsheet assuming non-split Object table > and measurements for Galaxies per frame. > - indexes not in memory: 208K > - indexes in memory: 18K > > > That is "only" less than 50% worse comparing to > the non-split object table. (especially if we put index in memory). > > 3) > I can bring down the numbers even further, e.g. down to 6K disks > by choosing a larger data block (512K). > > > All the numbers below assume disabled "select time series data > for a given cone" query (this query can still drive the numbers > unbelievably high, even with all indexes in memory) > > I need few more days to better validate everything, but I know > some of you really want the numbers ASAP, so there they are, > a little better version than the one before. > > Happy 4th of July! > > Jacek > > > > Jacek Becla wrote: > > Hi Jeff > > > > I promised to give you the disk io estimates by the end of Friday. > > I think I am slowly getting there, there is one query (classified as > > "low volume"!) that is causing troubles: "select time series data > > for a given cone". It highly depends on index selectivity, and > > if we choose a reasonable selectivity (50% of one partition is > > very reasonable, but even few % is causing problems), we end up > > fetching many millions of data rows which drives the number > > or required disks up many times (how many times - depends > > what selectivity we pick). > > > > If I pick very, very low selectivity for that query (effectively > > disable it) the spreadsheet suggests we need 166 K disk heads > > assuming we want to run all 300 low volume queries and 10 high > > volume queries, no super high volume query. (Reminded, the > > spreadsheet is for DR2). > > > > We can drive the numbers down by choosing to run less > > queries concurrently (in the current model all low volume > > queries will be done after 10 sec and all high volume > > after 10 min). > > > > It should also be possible to drive the io down by tuning > > the queries. > > > > I assumed disk random reads, block size 64K, realistically > > for a system with hundreds of simultaneous queries we should > > not count much on nice sequential reads. > > > > > > General comment: > > - none of the queries use Source table, they use DIASource > > table instead. DIASource is 5% of the Source table, so > > if someone chose the Source table, the numbers would get > > much worse. I don't feel comfortable with it: why did we > > reserved 100s of TB for Source table in the Database > > Size Estimations if we don't have any queries that would > > use that? > > - the numbers will get (much) worse if I redo the spreadsheet > > using the latest sizes (non-split object table...) > > > > > > Here is the most recent version of the spreadsheet and the doc: > > > > http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.xls > > http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.doc > > > > It will take me few more days to clean up the documentation > > and QA the spreadsheet better. > > > > I guess we knew the numbers would not be pretty.... > > > > To put a positive spin on all this, I think there is a solution: > > if we purchase 2.5 TB of RAM (very reasonable in 2012), we can > > pin all indexes used for these queries in memory, which will drive > > the number of required disk heads down to 44 K. BTW, this still > > does not solve the problem with time-series query which wants > > to touch petabytes of data rows... If this idea sounds attractive > > (think "PetaCache" =), I will do a more complete estimate > > of RAM needed. I did a first sketch in the spreadsheet, > > and you can turn it on and off to see the effect on various > > queries. > > > > Jacek > > > > > > _______________________________________________ > > LSST-data mailing list > > [email protected] > > http://www.lsstmail.org/mailman/listinfo/lsst-data > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data ----------------------------------------------------------------------------- Dr. Kirk D. Borne NASA Goddard Space Flight Center, SSDOO Program Manager, QSS Group Inc. and George Mason University, Associate Research Professor, College of Science <mailto:[EMAIL PROTECTED]> Tel. +1-301-286-0696 Fax: 301-286-1771 Staff page: http://rings.gsfc.nasa.gov/~borne/ US Virtual Observatory: http://www.us-vo.org/ Large Synoptic Survey Telescope: http://www.lssto.org/ _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
