Hi Kirk
I reshuffled queries based the disk io estimates so the
classification is based on what I found EXCEPT the one query
which really is "impossible" at the moment and needs a special
group (a.la. super=high-volume). I recall Kem saying this is going
to be a the most popular temporal query, so I think we need to
make that query "possible" again (by reclustering data and/or
rewriting query), not simply by classify it differently.
I'll fix the multiplicativity in a sec.
thanks,
Jacek
Kirk Borne wrote:
Jacek: can you tell me ... when you sent around the "top 20" LSST
queries document, you labeled the queries as Low Volume or
High Volume... were these labels the ones that Kem and I used,
or were these labels re-assigned to the queries after you learned
just how big or how small the queries actually were? I think we
should label them according to your estimated sizes, not
according to Kem and my guesses about their sizes. This is
particularly relevant for those cases where (I know, for my
own use cases) I misunderstood the gravity of joining to
some very large tables or joining to all sky partitions....
this made some of my low-volume queries into high-volume.
And so these queries should be labeled appropriately as
high-volume.
So, I am just wondering if your list included those updated
assignments (low versus high)?
p.s. I also noted that one of the positional queries still
had the multiplicative cos(DECL) factor in the R.A. condition,
when it should be divided by that factor. This was in the
high-volume query "Find variable stars close to hot stars".
Thanks.
- Kirk
Date: Mon, 03 Jul 2006 19:46:12 -0700
From: Jacek Becla <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] database disk io estimates
To: LSST Data Management <[email protected]>
Keywords: DataAccWG
Hello,
1)
I found one mistake in the original estimate for non-split Object table.
Fixing the problem brought down the number of disk heads:
- indexes not in memory 166K --> 140K
- indexes in memory: 44K --> 16K
2)
I redid the disk io spreadsheet assuming non-split Object table
and measurements for Galaxies per frame.
- indexes not in memory: 208K
- indexes in memory: 18K
That is "only" less than 50% worse comparing to
the non-split object table. (especially if we put index in memory).
3)
I can bring down the numbers even further, e.g. down to 6K disks
by choosing a larger data block (512K).
All the numbers below assume disabled "select time series data
for a given cone" query (this query can still drive the numbers
unbelievably high, even with all indexes in memory)
I need few more days to better validate everything, but I know
some of you really want the numbers ASAP, so there they are,
a little better version than the one before.
Happy 4th of July!
Jacek
Jacek Becla wrote:
Hi Jeff
I promised to give you the disk io estimates by the end of Friday.
I think I am slowly getting there, there is one query (classified as
"low volume"!) that is causing troubles: "select time series data
for a given cone". It highly depends on index selectivity, and
if we choose a reasonable selectivity (50% of one partition is
very reasonable, but even few % is causing problems), we end up
fetching many millions of data rows which drives the number
or required disks up many times (how many times - depends
what selectivity we pick).
If I pick very, very low selectivity for that query (effectively
disable it) the spreadsheet suggests we need 166 K disk heads
assuming we want to run all 300 low volume queries and 10 high
volume queries, no super high volume query. (Reminded, the
spreadsheet is for DR2).
We can drive the numbers down by choosing to run less
queries concurrently (in the current model all low volume
queries will be done after 10 sec and all high volume
after 10 min).
It should also be possible to drive the io down by tuning
the queries.
I assumed disk random reads, block size 64K, realistically
for a system with hundreds of simultaneous queries we should
not count much on nice sequential reads.
General comment:
- none of the queries use Source table, they use DIASource
table instead. DIASource is 5% of the Source table, so
if someone chose the Source table, the numbers would get
much worse. I don't feel comfortable with it: why did we
reserved 100s of TB for Source table in the Database
Size Estimations if we don't have any queries that would
use that?
- the numbers will get (much) worse if I redo the spreadsheet
using the latest sizes (non-split object table...)
Here is the most recent version of the spreadsheet and the doc:
http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.xls
http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.doc
It will take me few more days to clean up the documentation
and QA the spreadsheet better.
I guess we knew the numbers would not be pretty....
To put a positive spin on all this, I think there is a solution:
if we purchase 2.5 TB of RAM (very reasonable in 2012), we can
pin all indexes used for these queries in memory, which will drive
the number of required disk heads down to 44 K. BTW, this still
does not solve the problem with time-series query which wants
to touch petabytes of data rows... If this idea sounds attractive
(think "PetaCache" =), I will do a more complete estimate
of RAM needed. I did a first sketch in the spreadsheet,
and you can turn it on and off to see the effect on various
queries.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
-----------------------------------------------------------------------------
Dr. Kirk D. Borne
NASA Goddard Space Flight Center, SSDOO Program Manager, QSS Group Inc.
and George Mason University, Associate Research Professor, College of Science
<mailto:[EMAIL PROTECTED]> Tel. +1-301-286-0696 Fax: 301-286-1771
Staff page: http://rings.gsfc.nasa.gov/~borne/
US Virtual Observatory: http://www.us-vo.org/
Large Synoptic Survey Telescope: http://www.lssto.org/
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
