Keywords: DataAccWG
so to summarize (mostly for my own benefit ;-) ), we have 2 GB memory
available, and the question is: does a 1/10 scale simulation make sense:
- 400k - 1M objects per partition (instead of 4M - 10M)
- so this is 1 M x 2k = 2 GB as jacek said
- an all-sky distribution for realistic partitioning
- incremental update of 10k existing objects and add 100 new objects
per visit (instead of 100k existing & 1000 new)
- or shd we try to do the original 100k - 1000 update?
- is all this achievable with the USNO dataset?
i think such a test may not necessarily be scalable performance-wise to
the real thing, but it will bring out potential logistical problems with
the planned scenario and help us fine-tune it better.
personally, i dont anticipate availability of 20 GB memory (or even
more) to be problematic by 2010+ ... i'm more worried that we're
over-engineering the problem and asking for headaches/trouble (e.g.
endangering data integrity) by doing this outside the db (in memory)
rather than trusting the db to do it for us, esp. if it turns out to be
well within the db's performance capability to do this in 30sec by 2014
(with judicious partitioning for each visit).
ani
On Wed, 6 Dec 2006, Serge Monkewitz wrote:
> Keywords: DataAccWG
>
> On Wed, 06 Dec 2006 14:19:36 -0800, Maria A. Nieto-Santisteban
> <[EMAIL PROTECTED]> wrote:
>
> > Hi Jacek,
> >
> >> > Good luck with the distribution, especially if you want to
> >> > run a meaningful x-match at some point.
> >>
> >> This test is not for x-match, so in other words I don't
> >> want to run a meaningful x-match using this data.
> >
> > As I said before, the more realistic distribution of objects you have in
> > the sky, the more realistic distribution of objects you will have in the
> > disk and in memory. Indexes will be different and therefore behave
> > differently depending on the data distribution.
> >
> > Is really only me who see a problem here?
> >
> >> >> Let's do the test with 2 different row sizes: 200 bytes, and 2K.
> >> >> Small row size stresses the large number of rows in database.
> >> >> Larger row size stresses the system in a different way as Maria
> >> >> pointed out (smaller number of objects per page).
> >> >>
> >> >> Let me know if you have any comments/questions.
> >> >
> >> > What is your plan to put 20 GB (10 million objects * 2KB) of data in
> >> > memory?
> >>
> >> For 2KB-row size, I suggested to read 1 million rows (see mail sent
> >> on 12/01). That is 2 GB, and the server has 16 GB.
> >
> >
> > If the text below is what you are referring to as "read mail sent on
> > 12/01", you mention disk-space limitations not memory limitations.
> >
> > You well intend to do every thing in memory but a 2 GB test when reality
> > is 20 GB will prove nothing. We may think that by 2013 we will have lots
> > of memory ... but what if not? I'm very aware of the fact that we will
> > not
> > have "realistic" simulations any time soon. But the little we do got to
> > be
> > sensible, meassurable, and scalable so we can drive reasonsable
> > conclusions.
> >
> > I'm worry that if we do unrealistic tests it will add high risk in any
> > design decissions we take. This applies equally for unrealistic
> > distributions, unrealistic number of objects and unrealistic
>
> My 2 cents: I don't think anybody is claiming we are in a position to make
> a set-in-stone decision on partitioning. We do however need to make a
> decision on how to proceed for DC2.
>
> Here's a list (incomplete) of depressing facts:
> - non final HW (small amount of RAM, disk space, disk bandwidth, and
> processor speed relative to what we will probably need)
> - non final DBMS (MySQL today, ??? tomorrow)
> - bogus data (spatial distribution not ideal, non spatial data
> completely made up)
> - can't simulate the whole nightly pipeline (no DIA Sources which in the
> current plan share the same partitioning as the object catalog)
> - unrealistic row size *or* unrealistic row count (due to limitations of
> test HW)
> - unclear how many updates we will be doing
>
> We should reevaluate our decisions when the factors above change, and each
> of them will. So while I agree that a better spatial distribution is
> desireable there are so many other factors that will change that I don't
> think we need a really accurate spatial distribution right this second. To
> be clear: I am absolutely in favor of doing the work to generate a more
> realistic distribution and starting on that work now - it seems very
> similar to what we need to do to generate DIA Sources (does Sergei's
> simulator do any of this for us?). However, we can also do Jacek's
> proposed test now, which even without such a distribution will give us a
> feel for what happens with our approaches and perhaps reveal hidden
> pitfalls. Besides, even after the initial test, we still need to continue
> testing both approaches to see how they do with updates and with small
> numbers of inserts over lengthy periods of time. We can roll in a better
> distribution as soon as it's available. Until then we can test a few
> different distributions to make sure they don't result in massive
> performance differences between the 2 approaches.
>
> Cheers,
> Serge
>
> P.S. As for the "will we have lots of memory in 2013" question: I can buy
> a 2 socket 4 core Intel Woodcrest box with 16 FB-DIMM slots and 4GB RAM
> sticks -> 64GB of RAM today (price for such a system would be roughly
> $35,000). AMD Opteron scales to 8 sockets gluelessly as far as I know (so
> 128GB should be possible today, but I didn't check). Memory capacity is
> going to scale with socket count in an SMP machine - at the extreme end of
> the spectrum (with a couple hundred million dollars) you can buy an SGI
> Altix with 128 terabytes of globally shared memory today. It's extremely
> reasonable to expect 256-512GB of RAM capacity in a commodity server box
> by 2013. If worst comes to worst, we switch to something more
> high-end/esoteric with the required capacity.
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
>
--
Aniruddha R. Thakar, Research Scientist
Center for Astrophysical Sciences, JHU, Bloomberg 375
3701 San Martin Drive, Baltimore MD 21218-2695
410-516-4850, Fax: 410-516-5096
[EMAIL PROTECTED], http://www.sdss.jhu.edu/~thakar
-----------------------------------------------------------------------
It is dangerous to be right when the government is wrong. [Voltaire]
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
