Keywords: DataAccWG
Hi Maria and Serge,
Thanks for the input! Let me try to summarize it...
The main purpose of running this test is to compare performance
of two partitioning approaches for Object table. The test should
include:
1) locating 4-10 million objects located in the area of
observed FOV out of 1-3 billion objects
2) loading them to memory and
3) organizing them in memory (indexing).
I suggest we test writing of the updated objects back to disk
and x-match performance in the subsequent tests. Given there is
a problem with USNO-B data as Alex pointed out, we should probably
discuss this issue with others and decide whether we want to use
USNO data for x-match tests at all.
Let's alter position of each individual object by a different
amount to avoid placing objects on top of each other.
Let's do the test with 2 different row sizes: 200 bytes, and 2K.
Small row size stresses the large number of rows in database.
Larger row size stresses the system in a different way as Maria
pointed out (smaller number of objects per page).
Let me know if you have any comments/questions.
Thanks,
Jacek
Maria A. Nieto-Santisteban wrote:
Serge,
Anyway, I think maybe we should test all the way through x-match. Maybe
not the x-match itself, but at least up to the part where we create the
in-memory zone-ra index just prior to x-match. I may very well be wrong,
but it seems to me that the layout of things on disk might influence how
fast we can do this in memory.
How data is on disk will definitely help how fast you can put the data in
memory. Yes, how data is sorted (indexed) in memory affects performance.
Yes of course you are right, but - in the 200b row case we will be
reading/updating/inserting 10x more objects (using some kind of random
object generation or 10 perturbed copies scheme), so the number of pages
read/written from disk should be about the same. If I understood Jacek
correctly, he basically wants to max out the test platform - i.e. use as
many 2kb rows in 3 strips as we have the space to store, or use realistic
row count (which means we have to shrink to 200b per row to fit all-data
on disk).
didn't I say that doing the simulations for a whole strip would be more
complicated? ;-)
Cheers
Maria
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
