Serge, > > > > I thought the intention of this test was to measure how to handle two > > different partitioning approaches (FOV Regions vs whole Declination > > stripe) rather than measure pure I/O. By increasing the number of rows > > you are increasing the amount of data (I/O) but not the number of > > objects per partition (3.3 deg stripe).Well .. you might make the > > partition thicker but then what Region size (FOV) would you use? > > Inserting twice the same objects, as you propose somewhere, would > > affect the spatial distribution in a non desirable way. > > Maria - could you expand on your thoughts here because I'm not seeing the > problem - of course you wouldn't want to insert multiple copies of objects > exactly "on top" of eachother,
Jacek wrote "... Do we want to add dummy values to make each row 2K in size (expected lsst size), or do we want to keep the size as is (49 bytes row), but replicate data many times into the same table to get realistic number of objects per FOV (4 million, or 10 million)." I agree. I was pointing out ... you just cannot replicate data many times. > especially if you are going to be testing all the way through x-match. That is also true. However, as I understand it, this test is not for x-matching, but to see what it takes to read the necessary data (well, only objects) from the disk database and put it into the memory database. Once the association has happened in memory you put the data back into the disk database. What happens in memory is a black box. > I was thinking you could just insert N copies of the USNO table, > jittering the positions of each copy relative to the others by some > small amount (say 10-20 arcsec). Even better would be to perturb each > individual object's position by a different amount. Why is having a more > realistic distribution than that important for this test? The more realistic distribution you have of the objects in the sky, the more realistic distribution you have of the objects on the disk. Doing the 10, 20 arcsecs and even the individual perturbation can give you funny distributions. A better aproach would be (I think) to break the sky in regions, make an histrogram to count number of sources, then generate randomly the right proportion of point for each region. Probably have into account the fact that USNOB probably see as many stars as LSST in the Galactic plane, but LSST will go deeper out of the galactic plane, which means the density will be higher. > > 2) For the Region partitioning > > The same as above but now, each stripe is divided in Regions as described > > in > > http://www.lsstmail.org/mailman/private/lsst-data/attachments/20061109/1419264f/Partitioning4LSST-0001.doc > > I can't access this document. Could you e-mail me a copy please? It is my partitioning for LSST document that you have read before. In any case you should be able to read it if you log in the LSST mail archive. > > Well, with the extra simplification that we will deal only with objects > > as per Jacek's suggestion. Later on we will have to analyze > > the impact of reading/inserting DIA sources from/into the database. > > > > These tests will not let us know what happens when we have to read 10 > > million objects, but they will give us a good starting point about how > > the > > two different partitioning approaches compare. Once we have done these > > tests, we can play with densities and make more realistic simulations as > > I did for ADASS. > > > > Cheers > > > > Maria > > To be honest I'm just don't see a strong argument for 2kb rows vs. 200 > byte rows or vice versa. As far as I can tell either approach gives us a > basis for comparison. What about doing both? The row size makes a difference. The samller is the object the more can fit in a page and the lower the I/O. A 64k page can host about 30 ~2Kb objects vs 300 200b objects. The smaller the number of objects per page the higher the cost. Maria _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
