Keywords: DataAccWG
Hi Ani,
We have 16 GB of RAM per server available.
DC2 is defined as 10% of real LSST numbers, but I'd
shoot higher: I'd go as far as available hardware allows.
thanks,
Jacek
Ani Thakar wrote:
Keywords: DataAccWG
so to summarize (mostly for my own benefit ;-) ), we have 2 GB memory
available, and the question is: does a 1/10 scale simulation make sense:
- 400k - 1M objects per partition (instead of 4M - 10M)
- so this is 1 M x 2k = 2 GB as jacek said
- an all-sky distribution for realistic partitioning
- incremental update of 10k existing objects and add 100 new objects
per visit (instead of 100k existing & 1000 new)
- or shd we try to do the original 100k - 1000 update?
- is all this achievable with the USNO dataset?
i think such a test may not necessarily be scalable performance-wise to
the real thing, but it will bring out potential logistical problems with
the planned scenario and help us fine-tune it better.
personally, i dont anticipate availability of 20 GB memory (or even
more) to be problematic by 2010+ ... i'm more worried that we're
over-engineering the problem and asking for headaches/trouble (e.g.
endangering data integrity) by doing this outside the db (in memory)
rather than trusting the db to do it for us, esp. if it turns out to be
well within the db's performance capability to do this in 30sec by 2014
(with judicious partitioning for each visit).
ani
On Wed, 6 Dec 2006, Serge Monkewitz wrote:
Keywords: DataAccWG
On Wed, 06 Dec 2006 14:19:36 -0800, Maria A. Nieto-Santisteban
<[EMAIL PROTECTED]> wrote:
Hi Jacek,
Good luck with the distribution, especially if you want to
run a meaningful x-match at some point.
This test is not for x-match, so in other words I don't
want to run a meaningful x-match using this data.
As I said before, the more realistic distribution of objects you have in
the sky, the more realistic distribution of objects you will have in the
disk and in memory. Indexes will be different and therefore behave
differently depending on the data distribution.
Is really only me who see a problem here?
Let's do the test with 2 different row sizes: 200 bytes, and 2K.
Small row size stresses the large number of rows in database.
Larger row size stresses the system in a different way as Maria
pointed out (smaller number of objects per page).
Let me know if you have any comments/questions.
What is your plan to put 20 GB (10 million objects * 2KB) of data in
memory?
For 2KB-row size, I suggested to read 1 million rows (see mail sent
on 12/01). That is 2 GB, and the server has 16 GB.
If the text below is what you are referring to as "read mail sent on
12/01", you mention disk-space limitations not memory limitations.
You well intend to do every thing in memory but a 2 GB test when reality
is 20 GB will prove nothing. We may think that by 2013 we will have lots
of memory ... but what if not? I'm very aware of the fact that we will
not
have "realistic" simulations any time soon. But the little we do got to
be
sensible, meassurable, and scalable so we can drive reasonsable
conclusions.
I'm worry that if we do unrealistic tests it will add high risk in any
design decissions we take. This applies equally for unrealistic
distributions, unrealistic number of objects and unrealistic
My 2 cents: I don't think anybody is claiming we are in a position to make
a set-in-stone decision on partitioning. We do however need to make a
decision on how to proceed for DC2.
Here's a list (incomplete) of depressing facts:
- non final HW (small amount of RAM, disk space, disk bandwidth, and
processor speed relative to what we will probably need)
- non final DBMS (MySQL today, ??? tomorrow)
- bogus data (spatial distribution not ideal, non spatial data
completely made up)
- can't simulate the whole nightly pipeline (no DIA Sources which in the
current plan share the same partitioning as the object catalog)
- unrealistic row size *or* unrealistic row count (due to limitations of
test HW)
- unclear how many updates we will be doing
We should reevaluate our decisions when the factors above change, and each
of them will. So while I agree that a better spatial distribution is
desireable there are so many other factors that will change that I don't
think we need a really accurate spatial distribution right this second. To
be clear: I am absolutely in favor of doing the work to generate a more
realistic distribution and starting on that work now - it seems very
similar to what we need to do to generate DIA Sources (does Sergei's
simulator do any of this for us?). However, we can also do Jacek's
proposed test now, which even without such a distribution will give us a
feel for what happens with our approaches and perhaps reveal hidden
pitfalls. Besides, even after the initial test, we still need to continue
testing both approaches to see how they do with updates and with small
numbers of inserts over lengthy periods of time. We can roll in a better
distribution as soon as it's available. Until then we can test a few
different distributions to make sure they don't result in massive
performance differences between the 2 approaches.
Cheers,
Serge
P.S. As for the "will we have lots of memory in 2013" question: I can buy
a 2 socket 4 core Intel Woodcrest box with 16 FB-DIMM slots and 4GB RAM
sticks -> 64GB of RAM today (price for such a system would be roughly
$35,000). AMD Opteron scales to 8 sockets gluelessly as far as I know (so
128GB should be possible today, but I didn't check). Memory capacity is
going to scale with socket count in an SMP machine - at the extreme end of
the spectrum (with a couple hundred million dollars) you can buy an SGI
Altix with 128 terabytes of globally shared memory today. It's extremely
reasonable to expect 256-512GB of RAM capacity in a commodity server box
by 2013. If worst comes to worst, we switch to something more
high-end/esoteric with the required capacity.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
