Keywords: DataAccWG
Hello all, I run some ingest tests with mysql to get a feeling what hardware will be needed and how complex the multi-stream setup needs to be. A short summary: Used hardware: a dual CPU, 1.8GHz, rather old-ish machine with 2 large disk arrays and a local drive (each array capable of delivering ~50 MB/sec sequential read using 1MB blocks). OS: Redhat Enterprise 4. Input data used: SuperMacho Object data, ten ascii files, 5 million objects per file. Row size in database: 124 bytes. Input data read from disk. Used tables had no indices. 1 stream, 10 files - loading all 10 files took 807 sec (wall-clock time) - speed: 62K objects/sec - effective disk IO: 7.3 MB/sec - CPU: ~82% busy 2 streams, 5 files per stream: - whole test took 425 (wall-clock time) - speed: 118K objects/sec - effective IO: 13.9 MB/sec - CPU: 100% busy 5 streams, 2 files per stream: - whole test took 577 sec (wall-clock time) - speed: 87 objects/sec - CPU: 100% busy Earlier Tim's said: "I'd say we need to design for 10000 objects as a typical case, and about 50000 objects as a less frequent" [these numbers are per image, not reduced for DC1] So in the worst case, it should take < 1 sec to ingest all objects from given image. In practice data will arrive in ~200 streams (1/ccd) so we will deal with 50, max 250 objects per stream. Given how short a load of 250 object will be, I don't think we should worry about synchronizing loading between streams or the fact that we might have 200 simultaneous insert requests. So how long will it take to re-index that? 10000 objects / 200 = 50 objects per ccd. 50*4*60*10 = 120K per ccd stream per night (that is the worst case: at the end of the night). Adding 3 indices for a table with 120K objects took 2.35 sec. (120,000*124bytes = 14MB, so it can't take long...). So if it takes < 3 sec to load and reindex, association pipeline has 12 sec to query this data before data from the next image arrives. And even if it takes longer than 12 sec to query, we can still handle this by 'buffering' data in the ascii input file (which is now under Data Access Framework control, after we decided DAF is responsible for converting data to ascii). So I suggest we abandon the idea of 2 tables for each stream (one for loading and one for querying), as there is no need for it! The catalog ingest at the base really seems quite doable, even today. (should we move on to Data Challenge 2 ;=). More details about the tests are available in Docushare: https://www.lsstcorp.org/docushare/dsweb/Get/Document-1386/DBIngestTests.doc Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
