Hi Jeff I promised to give you the disk io estimates by the end of Friday. I think I am slowly getting there, there is one query (classified as "low volume"!) that is causing troubles: "select time series data for a given cone". It highly depends on index selectivity, and if we choose a reasonable selectivity (50% of one partition is very reasonable, but even few % is causing problems), we end up fetching many millions of data rows which drives the number or required disks up many times (how many times - depends what selectivity we pick).
If I pick very, very low selectivity for that query (effectively disable it) the spreadsheet suggests we need 166 K disk heads assuming we want to run all 300 low volume queries and 10 high volume queries, no super high volume query. (Reminded, the spreadsheet is for DR2). We can drive the numbers down by choosing to run less queries concurrently (in the current model all low volume queries will be done after 10 sec and all high volume after 10 min). It should also be possible to drive the io down by tuning the queries. I assumed disk random reads, block size 64K, realistically for a system with hundreds of simultaneous queries we should not count much on nice sequential reads. General comment: - none of the queries use Source table, they use DIASource table instead. DIASource is 5% of the Source table, so if someone chose the Source table, the numbers would get much worse. I don't feel comfortable with it: why did we reserved 100s of TB for Source table in the Database Size Estimations if we don't have any queries that would use that? - the numbers will get (much) worse if I redo the spreadsheet using the latest sizes (non-split object table...) Here is the most recent version of the spreadsheet and the doc: http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.xls http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.doc It will take me few more days to clean up the documentation and QA the spreadsheet better. I guess we knew the numbers would not be pretty.... To put a positive spin on all this, I think there is a solution: if we purchase 2.5 TB of RAM (very reasonable in 2012), we can pin all indexes used for these queries in memory, which will drive the number of required disk heads down to 44 K. BTW, this still does not solve the problem with time-series query which wants to touch petabytes of data rows... If this idea sounds attractive (think "PetaCache" =), I will do a more complete estimate of RAM needed. I did a first sketch in the spreadsheet, and you can turn it on and off to see the effect on various queries. Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
