Keywords: DataAccWG
Hi Maria,
As I said before, the more realistic distribution of objects you have in
the sky, the more realistic distribution of objects you will have in the
disk and in memory. Indexes will be different and therefore behave
differently depending on the data distribution.
Is really only me who see a problem here?
It is all true what you said. Notice however, that if we pick
a distribution (e.g. galactic center as you suggested), it will
tell us how to partition data for the region(s) with similar
distribution, but it will not necessarily be the best partitioning
for other regions, and perhaps on average we could be better off
choosing different partitioning. My point is that finding this
balance is really tricky, and we can't expect to solve all these
problems in a single test. Would you be happier if we did a serious
of tests using several regions with different distributions?
You well intend to do every thing in memory but a 2 GB test when reality
is 20 GB will prove nothing. We may think that by 2013 we will have lots
of memory ... but what if not? I'm very aware of the fact that we will not
have "realistic" simulations any time soon. But the little we do got to be
sensible, meassurable, and scalable so we can drive reasonsable
conclusions.
I'm worry that if we do unrealistic tests it will add high risk in any
design decissions we take. This applies equally for unrealistic
distributions, unrealistic number of objects and unrealistic row sizes.
Please suggest an alternative if you don't like the proposed approach
so much, and we will discuss it. Do consider the hardware constraints
we have.
thanks,
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
