Keywords: DataAccWG > > Hi Maria and Serge, > > Thanks for the input! Let me try to summarize it... > > The main purpose of running this test is to compare performance > of two partitioning approaches for Object table. The test should > include: > 1) locating 4-10 million objects located in the area of > observed FOV out of 1-3 billion objects > 2) loading them to memory and > 3) organizing them in memory (indexing). > > I suggest we test writing of the updated objects back to disk > and x-match performance in the subsequent tests. Given there is > a problem with USNO-B data as Alex pointed out, we should probably > discuss this issue with others and decide whether we want to use > USNO data for x-match tests at all. > > > Let's alter position of each individual object by a different > amount to avoid placing objects on top of each other.
Good luck with the distribution, especially if you want to run a meaningful x-match at some point. > Let's do the test with 2 different row sizes: 200 bytes, and 2K. > Small row size stresses the large number of rows in database. > Larger row size stresses the system in a different way as Maria > pointed out (smaller number of objects per page). > > Let me know if you have any comments/questions. What is your plan to put 20 GB (10 million objects * 2KB) of data in memory? Thanks Maria > Thanks, > Jacek > > Maria A. Nieto-Santisteban wrote: > > Serge, > > > >> Anyway, I think maybe we should test all the way through x-match. Maybe > >> not the x-match itself, but at least up to the part where we create the > >> in-memory zone-ra index just prior to x-match. I may very well be wrong, > >> but it seems to me that the layout of things on disk might influence how > >> fast we can do this in memory. > > > > How data is on disk will definitely help how fast you can put the data in > > memory. Yes, how data is sorted (indexed) in memory affects performance. > > > >> Yes of course you are right, but - in the 200b row case we will be > >> reading/updating/inserting 10x more objects (using some kind of random > >> object generation or 10 perturbed copies scheme), so the number of pages > >> read/written from disk should be about the same. If I understood Jacek > >> correctly, he basically wants to max out the test platform - i.e. use as > >> many 2kb rows in 3 strips as we have the space to store, or use realistic > >> row count (which means we have to shrink to 200b per row to fit all-data > >> on disk). > > > > > > didn't I say that doing the simulations for a whole strip would be more > > complicated? ;-) > > > > Cheers > > > > Maria > > > > _______________________________________________ > > LSST-data mailing list > > [email protected] > > http://www.lsstmail.org/mailman/listinfo/lsst-data > > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > -- ------------------------------------------------ Maria A. Nieto-Santisteban ([EMAIL PROTECTED]) Johns Hopkins University 3400 N. Charles St. Physics & Astronomy Department Baltimore, MD 21218 (USA) Tel: 1 410 516-7679 Fax: 1 410 516-5096 _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
