Keywords: DataAccWG Hi
> > As I said before, the more realistic distribution of objects you have in > > the sky, the more realistic distribution of objects you will have in the > > disk and in memory. Indexes will be different and therefore behave > > differently depending on the data distribution. > > > > Is really only me who see a problem here? > > It is all true what you said. Notice however, that if we pick > a distribution (e.g. galactic center as you suggested), it will > tell us how to partition data for the region(s) with similar > distribution, but it will not necessarily be the best partitioning > for other regions, and perhaps on average we could be better off > choosing different partitioning. My point is that finding this > balance is really tricky, and we can't expect to solve all these > problems in a single test. I think you are confussing partitioning approaches (horizontal stripes vs regions) with distribution of data in partitions (yes we could put in the same disk partition data belonging to a wider strip but the strip contains the same amount of data). I know that is tricky, very tricky, and it will require a lot of tests. That's why the little you do got to be sensible. We have been talking about the Galactic center because that gives us the worst case. Let's note that it is not my suggestion only. > Would you be happier if we did a serious of tests using several regions > with different distributions? As I pointed out in http://www.lsstmail.org/mailman/private/lsst-data/2006-December/002123.html "... A better aproach would be (I think) to break the sky in regions, make an histrogram to count number of sources, then generate randomly the right proportion of point for each region. Probably have into account the fact that USNOB probably see as many stars as LSST in the Galactic plane, but LSST will go deeper out of the galactic plane, which means the density will be higher." Which in summary says: If you are going to fake data out the galactic center (sorry I wrote in the original mail plane), you better do it right or at least not randomly. > > You well intend to do every thing in memory but a 2 GB test when reality > > is 20 GB will prove nothing. We may think that by 2013 we will have lots > > of memory ... but what if not? I'm very aware of the fact that we will not > > have "realistic" simulations any time soon. But the little we do got to be > > sensible, meassurable, and scalable so we can drive reasonsable > > conclusions. > > > > I'm worry that if we do unrealistic tests it will add high risk in any > > design decissions we take. This applies equally for unrealistic > > distributions, unrealistic number of objects and unrealistic row sizes. > > Please suggest an alternative if you don't like the proposed approach > so much, and we will discuss it. Do consider the hardware constraints > we have. I already did. http://www.lsstmail.org/mailman/private/lsst-data/2006-December/002115.html As I said in that e-mail "These tests will not let us know what happens when we have to read 10 million objects, but they will give us a good starting point about how the two different partitioning approaches compare. Once we have done these tests, we can play with densities and make more realistic simulations as I did for ADASS." It is not me who is pushing for testing the 10 million objects case for a whole horizontal strip vs regions today. Today, you know that your objects will be 2k and that's something you got to include in your tests. Working with 10 million object 200 B each doesn't tell you very much. You got to compare apples to apples. I'm not trying to be difficult just trying to contribute my experince in the matter. I'm off to the airport in a couple of hours. I'll be back Dec 30th. Have a nice holidays Thanks Maria _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
