Keywords: DataAccWG Hi Jacek, > > Good luck with the distribution, especially if you want to > > run a meaningful x-match at some point. > > This test is not for x-match, so in other words I don't > want to run a meaningful x-match using this data.
As I said before, the more realistic distribution of objects you have in the sky, the more realistic distribution of objects you will have in the disk and in memory. Indexes will be different and therefore behave differently depending on the data distribution. Is really only me who see a problem here? > >> Let's do the test with 2 different row sizes: 200 bytes, and 2K. > >> Small row size stresses the large number of rows in database. > >> Larger row size stresses the system in a different way as Maria > >> pointed out (smaller number of objects per page). > >> > >> Let me know if you have any comments/questions. > > > > What is your plan to put 20 GB (10 million objects * 2KB) of data in > > memory? > > For 2KB-row size, I suggested to read 1 million rows (see mail sent > on 12/01). That is 2 GB, and the server has 16 GB. If the text below is what you are referring to as "read mail sent on 12/01", you mention disk-space limitations not memory limitations. You well intend to do every thing in memory but a 2 GB test when reality is 20 GB will prove nothing. We may think that by 2013 we will have lots of memory ... but what if not? I'm very aware of the fact that we will not have "realistic" simulations any time soon. But the little we do got to be sensible, meassurable, and scalable so we can drive reasonsable conclusions. I'm worry that if we do unrealistic tests it will add high risk in any design decissions we take. This applies equally for unrealistic distributions, unrealistic number of objects and unrealistic row sizes. Maria Date: Fri, 01 Dec 2006 15:21:47 -0800 From: Jacek Becla <[EMAIL PROTECTED]> Reply-To: LSST Data Management <[email protected]> To: LSST Data Management <[email protected]> Subject: Re: [LSST-data] DB tests using USNO data [ The following text is in the "ISO-8859-15" character set. ] [ Your display is set for the "ISO-8859-1" character set. ] [ Some characters may be displayed incorrectly. ] Serge > Jacek - given the space restrictions on the test-system that sounds like > the best we can do right now. But I admit I don't understand how > increasing row size to 200bytes is "the same as increasing row size to 2 > K from the disk I/O point of view". OK, I made a shortcut here. I meant "increasing row size to 200 bytes and reading 10 million rows is the same as increasing row size to 2KB and reading 1 million rows" (no matter what the row size, we are disk-space limited so changing row size automatically means changing number of rows we can handle) > Also out of curiosity - how much > RAM/sustained bandwidth to disk does this test system have? RAM: 16 GB Sustained bandwidth using 256K blocks: - ~150 MB/sec sequential write - ~145 MB/sec sequential read It has 2 disk arrays (Sun T3), 500 GB each. We have two such servers. Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
