Jacek, I thought the intention of this test was to measure how to handle two different partitioning approaches (FOV Regions vs whole Declination stripe) rather than measure pure I/O. By increasing the number of rows you are increasing the amount of data (I/O) but not the number of objects per partition (3.3 deg stripe). Well .. you might make the partition thicker but then what Region size (FOV) would you use? Inserting twice the same objects, as you propose somewhere, would affect the spatial distribution in a non desirable way.
What I would propose is: 1) For the stripe partitioning - Use USNO positions to simulate 3 partitions. Each partition would cover 3.4 degrees declination height and each row would be 2 KB of fake values. I propose 3.4 degrees (~0.1 deg slack) to minimize the number of partitions we need to deal with in the unlikely case a FOV matches the partition's declination boundaries perfectly. - Since we are interested in worst cases, I would use three partitions covering the Galactic center (RA, DEC = 266.4, -28.93) - Each of these partitions should be clustered indexed by ZoneId and RA (zoneHeight = 16 arcsec) Once this database has been built: Define a few random FOVs and meassure what does it take to a) Read all objects included in a FOV+ b) Select randomly a X K to be updated + Yk new objects which need to be inserted (X and Y corresponding percentage of variable and new objects for whatever is the density of objects we have instead of the 10 million, 100k, 1k we have been talking about) 2) For the Region partitioning The same as above but now, each stripe is divided in Regions as described in http://www.lsstmail.org/mailman/private/lsst-data/attachments/20061109/1419264f/Partitioning4LSST-0001.doc Well, with the extra simplification that we will deal only with objects as per Jacek's suggestion. Later on we will have to analyze the impact of reading/inserting DIA sources from/into the database. These tests will not let us know what happens when we have to read 10 million objects, but they will give us a good starting point about how the two different partitioning approaches compare. Once we have done these tests, we can play with densities and make more realistic simulations as I did for ADASS. Cheers Maria On Fri, 1 Dec 2006, Jacek Becla wrote: > Serge > > > > Jacek - given the space restrictions on the test-system that sounds like > > the best we can do right now. But I admit I don't understand how > > increasing row size to 200bytes is "the same as increasing row size to 2 > > K from the disk I/O point of view". > > OK, I made a shortcut here. I meant "increasing row size to 200 bytes > and reading 10 million rows is the same as increasing row size to 2KB > and reading 1 million rows" (no matter what the row size, we are > disk-space limited so changing row size automatically means changing > number of rows we can handle) > > > > Also out of curiosity - how much > > RAM/sustained bandwidth to disk does this test system have? > > RAM: 16 GB > > Sustained bandwidth using 256K blocks: > - ~150 MB/sec sequential write > - ~145 MB/sec sequential read > > It has 2 disk arrays (Sun T3), 500 GB each. > > > We have two such servers. > > Jacek > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > -- ------------------------------------------------ Maria A. Nieto-Santisteban ([EMAIL PROTECTED]) Johns Hopkins University 3400 N. Charles St. Physics & Astronomy Department Baltimore, MD 21218 (USA) Tel: 1 410 516-7679 Fax: 1 410 516-5096 _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
