On Mon, 04 Dec 2006 11:26:21 -0800, Maria A. Nieto-Santisteban
<[EMAIL PROTECTED]> wrote:
Jacek,
I thought the intention of this test was to measure how to handle two
different partitioning approaches (FOV Regions vs whole Declination
stripe) rather than measure pure I/O. By increasing the number of rows
you
are increasing the amount of data (I/O) but not the number of objects per
partition (3.3 deg stripe).Well .. you might make the partition thicker
but then what Region size (FOV) would you use? Inserting twice the same
objects, as you propose somewhere, would affect the spatial distribution
in a non desirable way.
Maria - could you expand on your thoughts here because I'm not seeing the
problem - of course you wouldn't want to insert multiple copies of objects
exactly "on top" of eachother, especially if you are going to be testing
all the way through x-match. I was thinking you could just insert N copies
of the USNO table, jittering the positions of each copy relative to the
others by some small amount (say 10-20 arcsec). Even better would be to
perturb each individual object's position by a different amount. Why is
having a more realistic distribution than that important for this test?
What I would propose is:
1) For the stripe partitioning
- Use USNO positions to simulate 3 partitions. Each partition would cover
3.4 degrees declination height and each row would be 2 KB of fake values.
I propose 3.4 degrees (~0.1 deg slack) to minimize the number of
partitions we need to deal with in the unlikely case a FOV matches
the partition's declination boundaries perfectly.
- Since we are interested in worst cases, I would use three partitions
covering the Galactic center (RA, DEC = 266.4, -28.93)
- Each of these partitions should be clustered indexed by ZoneId and RA
(zoneHeight = 16 arcsec)
Once this database has been built:
Define a few random FOVs and meassure what does it take to
a) Read all objects included in a FOV+
b) Select randomly a X K to be updated + Yk new objects which need to be
inserted
(X and Y corresponding percentage of variable and new objects for
whatever
is the density of objects we have instead of the 10 million, 100k,
1k we have been talking about)
2) For the Region partitioning
The same as above but now, each stripe is divided in Regions as described
in
http://www.lsstmail.org/mailman/private/lsst-data/attachments/20061109/1419264f/Partitioning4LSST-0001.doc
I can't access this document. Could you e-mail me a copy please?
Well, with the extra simplification that we will deal only with objects
as per Jacek's suggestion. Later on we will have to analyze
the impact of reading/inserting DIA sources from/into the database.
These tests will not let us know what happens when we have to read 10
million objects, but they will give us a good starting point about how
the
two different partitioning approaches compare. Once we have done these
tests, we can play with densities and make more realistic simulations as
I did for ADASS.
Cheers
Maria
To be honest I'm just don't see a strong argument for 2kb rows vs. 200
byte rows or vice versa. As far as I can tell either approach gives us a
basis for comparison. What about doing both?
Serge
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
