Please be aware of the fact, that the USNO-B data has a problem. Maria has detected when you plot the counts of objects as a function of declination, there is a substantial 6 degree sawtooth pattern on the counts.
This is due to the fact that at the corners of the USNOB plates the object matching is less then perfect, and we get objects in duplicates rather than having them merged. So this might create some artifacts in our results as well. I would also suggest that we pick on of our test areas such that it contains the North pole of the coordinate system. This will bring out possible problems if the underlying spatial index does not handle spherical coordinates well. On the equator everything is easy. --Alex -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Maria A. Nieto-Santisteban Sent: Monday, December 04, 2006 9:03 PM To: LSST Data Management Subject: Re: [LSST-data] DB tests using USNO data Serge, > > > > I thought the intention of this test was to measure how to handle > > two different partitioning approaches (FOV Regions vs whole > > Declination > > stripe) rather than measure pure I/O. By increasing the number of > > rows you are increasing the amount of data (I/O) but not the number > > of objects per partition (3.3 deg stripe).Well .. you might make the > > partition thicker but then what Region size (FOV) would you use? > > Inserting twice the same objects, as you propose somewhere, would > > affect the spatial distribution in a non desirable way. > > Maria - could you expand on your thoughts here because I'm not seeing > the problem - of course you wouldn't want to insert multiple copies of > objects exactly "on top" of eachother, Jacek wrote "... Do we want to add dummy values to make each row 2K in size (expected lsst size), or do we want to keep the size as is (49 bytes row), but replicate data many times into the same table to get realistic number of objects per FOV (4 million, or 10 million)." I agree. I was pointing out ... you just cannot replicate data many times. > especially if you are going to be testing all the way through x-match. That is also true. However, as I understand it, this test is not for x-matching, but to see what it takes to read the necessary data (well, only objects) from the disk database and put it into the memory database. Once the association has happened in memory you put the data back into the disk database. What happens in memory is a black box. > I was thinking you could just insert N copies of the USNO table, > jittering the positions of each copy relative to the others by some > small amount (say 10-20 arcsec). Even better would be to perturb each > individual object's position by a different amount. Why is having a > more realistic distribution than that important for this test? The more realistic distribution you have of the objects in the sky, the more realistic distribution you have of the objects on the disk. Doing the 10, 20 arcsecs and even the individual perturbation can give you funny distributions. A better aproach would be (I think) to break the sky in regions, make an histrogram to count number of sources, then generate randomly the right proportion of point for each region. Probably have into account the fact that USNOB probably see as many stars as LSST in the Galactic plane, but LSST will go deeper out of the galactic plane, which means the density will be higher. > > 2) For the Region partitioning > > The same as above but now, each stripe is divided in Regions as > > described in > > http://www.lsstmail.org/mailman/private/lsst-data/attachments/200611 > > 09/1419264f/Partitioning4LSST-0001.doc > > I can't access this document. Could you e-mail me a copy please? It is my partitioning for LSST document that you have read before. In any case you should be able to read it if you log in the LSST mail archive. > > Well, with the extra simplification that we will deal only with > > objects as per Jacek's suggestion. Later on we will have to analyze > > the impact of reading/inserting DIA sources from/into the database. > > > > These tests will not let us know what happens when we have to read > > 10 million objects, but they will give us a good starting point > > about how the two different partitioning approaches compare. Once we > > have done these tests, we can play with densities and make more > > realistic simulations as I did for ADASS. > > > > Cheers > > > > Maria > > To be honest I'm just don't see a strong argument for 2kb rows vs. 200 > byte rows or vice versa. As far as I can tell either approach gives us > a basis for comparison. What about doing both? The row size makes a difference. The samller is the object the more can fit in a page and the lower the I/O. A 64k page can host about 30 ~2Kb objects vs 300 200b objects. The smaller the number of objects per page the higher the cost. Maria _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
