Hi all, Here's the summary of DB performance/scaling so far, based on tests done in Livermore:
Expected data flow for LSST will be 100,000 DIA sources per 15 sec (estimate based on 50 objects/sq. arcmin source density and assumes 0.05 fraction of variable sources in FOV). The throughput we got so far with SM schema was 100,000-300,000 rows per minute (rows are ~100 byte long). This is the ingest speed into an unindexed table, writing to a local RAID5 partition (6x 400GB SATA 150 disks + 1 hot spare). The above figure is a factor 2-4X lower than the expected rate (400K sources/min). There are some additional factors to consider: 1. The MySQL engine we used was NOT optimized in any way. Tuning the internal parameters (e.g. cache buffer size) can give additional speedup on the ingest. 2. We used a single-CPU, single RAID model for ingest. Using federated DB (multi-CPU) and partitioned tables (spread across multiple disks) should definitely help. 3. The ingest was done with bulk upload (fastest), but MySQL only allows bulk upload from *disk* file. Effectively, one writes data to disk twice: first into an intermediate file, and then into DB file. Using RAMdisk here would eliminate one of these writes, and provide speedup. 4. Ingesting into indexed table, based on our tests, is a big no-no. It gets progressively worse as the table grows. 5. The ingest rate stayed fairly constant with table size, which means we did not "saturate" the disk in our tests, i.e. seek time and write performance were nominal. (The RAID5 partition held 300GB of raw data for ingest + 200GB DB size = 500GB, while the full capacity of our RAID was 1.8TB). As for the clustering, this was done on pooled SM data (34M rows), by finding "Neighbors" for each source (neighbor search involved querying two indexed fields, RA and zone (Dec), each indexed with a simple B-tree). This took 2 hours of wall clock time, using an external C++ code interacting with the DB. For a single LSST image (100K DIA sources), this suggests 20 sec clustering time (assuming linear scaling). Sergei. P.S. This setup is easily portable to a TeraGrid node (a few Perl scripts + C++ code + raw data), but I think for DC1 we should experiment with those additional things I listed above (federating, partitioning, etc). This suggests hardware that is capable of supporting these functions (e.g. we need a few machines to support federating, multiple local disks to support partitioning, etc). > All, > > Attached is an incomplete draft of my thinking on the requirements for > DC1, mainly input for discussion at the telecon later this morning. > > Tim > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
