Keywords: DataAccWG Attendees: Ani, Kirk, Kem, Sergei, Russell, Ray, Jacek
The final number in the spreadsheet is for "independent heads", not "spindles" select time series data for given cone: - would help to keep mjd in DIASource table - not only for this query, but in general it would be useful to have that in DIASource - BTW, rename mjd to tai Need to put schema in subversion - after the io and sizing estimates done The top down estimate assumes 1 TB disk size, ~1000 disks for DR1+DR2 How many heads modern disk drives have? - we thought several: ~5-10, but were not certain Have wrong assumption about row size for Source table, should be 1/5 of what we have, because each source is in one color only. We do need to add 1 column indicating which color (filter) was used The select time series data for given cone: - wrong assumption for the DIASource x CCDImage join: - there should be 1 CCDImage for each DIASource, spreadsheet assumes 10K. - this helps a lot! :=) Jacek will send updated spreadsheets with db sizes and disk io estimates by the end of today Jacek Jacek Becla wrote:
Keywords: DataAccWG Hi all, We will have a Database telecon this Friday at 11:00 AM PDT. I'd like to go over the latest disk io numbers. Here is the latest version: http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.doc http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.xls Shortly speaking the numbers looks very reasonable for everything except two temporal queries, and both have the same problem (see below) Phone number: 866 330 1200 Pass code: 300 2363 Jacek ====================== Select time series data for given cone: SELECT o.objectId, o.ra, o.dec, i.MJD, d.* FROM Object o, CCDImage i, DIASource d WHERE o.ra BETWEEN ra1 + cos(decl1)*(0.5) AND ra1 - cos(decl1)*(0.5) AND o.decl between decl1 +0.5 and decl1 - 0.5 AND o.objectId = d.objectId AND d.ccdImageId = i.ccdImageId ORDER BY o.objectId, i.MJD ASC Object table: 10 million rows (one partition) DIASource table: 45 billion rows CCDImage table: 135 million rows At the moment, if I assume a modest 1% selectivity for ra,dec that gives 10K rows from Object table. Join with DIASource (5 DIASources per star) gives 50K rows, then join with CCDImage (assuming 10K DIASources per CCDImage) gives 50*10^12 rows. That is over a petabyte, which now needs to be sorted, then for each row we need to fetch full DIASource row.... _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
