Keywords: DataAccWG
On Wed, 06 Dec 2006 14:19:36 -0800, Maria A. Nieto-Santisteban
<[EMAIL PROTECTED]> wrote:
Hi Jacek,
> Good luck with the distribution, especially if you want to
> run a meaningful x-match at some point.
This test is not for x-match, so in other words I don't
want to run a meaningful x-match using this data.
As I said before, the more realistic distribution of objects you have in
the sky, the more realistic distribution of objects you will have in the
disk and in memory. Indexes will be different and therefore behave
differently depending on the data distribution.
Is really only me who see a problem here?
>> Let's do the test with 2 different row sizes: 200 bytes, and 2K.
>> Small row size stresses the large number of rows in database.
>> Larger row size stresses the system in a different way as Maria
>> pointed out (smaller number of objects per page).
>>
>> Let me know if you have any comments/questions.
>
> What is your plan to put 20 GB (10 million objects * 2KB) of data in
> memory?
For 2KB-row size, I suggested to read 1 million rows (see mail sent
on 12/01). That is 2 GB, and the server has 16 GB.
If the text below is what you are referring to as "read mail sent on
12/01", you mention disk-space limitations not memory limitations.
You well intend to do every thing in memory but a 2 GB test when reality
is 20 GB will prove nothing. We may think that by 2013 we will have lots
of memory ... but what if not? I'm very aware of the fact that we will
not
have "realistic" simulations any time soon. But the little we do got to
be
sensible, meassurable, and scalable so we can drive reasonsable
conclusions.
I'm worry that if we do unrealistic tests it will add high risk in any
design decissions we take. This applies equally for unrealistic
distributions, unrealistic number of objects and unrealistic
My 2 cents: I don't think anybody is claiming we are in a position to make
a set-in-stone decision on partitioning. We do however need to make a
decision on how to proceed for DC2.
Here's a list (incomplete) of depressing facts:
- non final HW (small amount of RAM, disk space, disk bandwidth, and
processor speed relative to what we will probably need)
- non final DBMS (MySQL today, ??? tomorrow)
- bogus data (spatial distribution not ideal, non spatial data
completely made up)
- can't simulate the whole nightly pipeline (no DIA Sources which in the
current plan share the same partitioning as the object catalog)
- unrealistic row size *or* unrealistic row count (due to limitations of
test HW)
- unclear how many updates we will be doing
We should reevaluate our decisions when the factors above change, and each
of them will. So while I agree that a better spatial distribution is
desireable there are so many other factors that will change that I don't
think we need a really accurate spatial distribution right this second. To
be clear: I am absolutely in favor of doing the work to generate a more
realistic distribution and starting on that work now - it seems very
similar to what we need to do to generate DIA Sources (does Sergei's
simulator do any of this for us?). However, we can also do Jacek's
proposed test now, which even without such a distribution will give us a
feel for what happens with our approaches and perhaps reveal hidden
pitfalls. Besides, even after the initial test, we still need to continue
testing both approaches to see how they do with updates and with small
numbers of inserts over lengthy periods of time. We can roll in a better
distribution as soon as it's available. Until then we can test a few
different distributions to make sure they don't result in massive
performance differences between the 2 approaches.
Cheers,
Serge
P.S. As for the "will we have lots of memory in 2013" question: I can buy
a 2 socket 4 core Intel Woodcrest box with 16 FB-DIMM slots and 4GB RAM
sticks -> 64GB of RAM today (price for such a system would be roughly
$35,000). AMD Opteron scales to 8 sockets gluelessly as far as I know (so
128GB should be possible today, but I didn't check). Memory capacity is
going to scale with socket count in an SMP machine - at the extreme end of
the spectrum (with a couple hundred million dollars) you can buy an SGI
Altix with 128 terabytes of globally shared memory today. It's extremely
reasonable to expect 256-512GB of RAM capacity in a commodity server box
by 2013. If worst comes to worst, we switch to something more
high-end/esoteric with the required capacity.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
