Hi,
This time I talked Andy H., here is a summary.
We need a signal from the camera before the image is taken
(for instance 15-30 sec before the ingest/join occur, for sure
the coordinates are known at that time). This will give us
enough time to preload needed archived data to memory
(5GB of data will fit very easily).
We can
a) divide the 10 TB into many small tables, or
b) index the whole 10 TB.
many small tables:
==================
Let's divide the 10 TB into say 5000 tables, each 2GB in size
and cluster data spatially: each table will contain a section
of the sky and have a unique id. Index each table.
Let's have a table that maps "coordinates of currently taken image"
to corresponding list of "2GB tables", and always keep this
table in memory. If we assume we need ~5 GB of data, each
coordinate will map to one table that is completely inside
the image, and several neighbor-tables.
Once we know the coordinates of currently taken image,
identify the 2GB tables
a) load to memory the table(s) that are fully
within the observed area
b) using indexes select the rows of interest from
the neighbor-tables and load to memory
That would require going (sequentially =) through 5+GB of data
- at 100 MB/sec that would take 50+ sec
- at 250 MB/sec that would take 20+ sec
That is the best we can do.
Given we will need to load 5+GB every 30 sec (once per a pair
of images) 100 MB/sec is not enough, so we may need to replicate
the 10 TB catalog on two sets of disks to improve speed.
Of course in 2012 we may have much faster disks (someone
else should decide...)
indexing 10TB
=============
Assuming 100 bytes per row, 10TB --> 10^13 objects, so will need
~39 "hops" to identify the right leaf in a btree, which is < 0.5 sec.
Then will need to fetch the 5GB of data _through index_.
That is an extra 750 MB comparing to the previous approach:
- 5GB/100 = ~50 million rows
- (4 bytes of index + 6 bytes of index pointer) / 0.67 = 15 bytes
- 15 bytes x 50 million rows = 750 MB
I like the first approach better, but feel free to argue =)
Jacek
Jeffrey P Kantor wrote:
Hello all,
I raised the a question with the Telescope/Control System team this morning:
Is there any constraint on operations by the scheduler such that we know
ahead of time the telescope won't go to certain fields that night so that we
do not have to have the entire sky template and catalog pre-cached each
night?
The answer was probably not, so we should assume we need it all pre-cached.
From an archive to base transfer standpoint, that means only what has
changed since the last transfer of course.
Jeff
From: Jacek Becla <[EMAIL PROTECTED]>
Organization: Stanford Linear Accelerator Center
Reply-To: LSST Data Management <[email protected]>
Date: Thu, 04 May 2006 12:22:28 -0700
To: LSST Data Management <[email protected]>
Subject: [LSST-data] Base Camp Database requirements
Keywords: DataAccWG
Hi,
I just had a quick word with Kem, here is the summary plus
some estimates/comments I added.
1) What is the expected size of pre-cached archived data
at the base camp?
We are expecting to have roughly 100TB of image data,
corresponding catalog data will probably be ~10% of that
So it is ~10TB (upper bound).
2) How much of that data will be needed on average per image?
~5GB (upper bound)
3) Is it likely we will re-use the same data for the next images?
Two adjacent images will always need the same data (images
will be taken "in pairs"). Beyond that, there might be
~3-5% overlap, so it is almost negligible.
4) Is there a natural way of splitting the "10TB catalog"
so that we can partition it and spread across multiple servers?
yes, we could split it across two axes:
a) could do it per filter, but we would need to re-cluster
data while precaching it. Data not spread uniformly
across filters: most observations taken in 2 filters (R, I),
very few in G. So this is not a good partitioning.
b) can partition data spatially: based on position in the sky.
Have to deal with edges, so probably will need to have some
overlap between different partitions, but other than that,
it is nice, can answer the join query by going directly
to one partition.
5) what is the accepted latency the join of Ingested Data
with precached archived data can take?
We will need <2-3 sec to ingest data. Probably it is ok
if the join for the first image from given pair takes ~10 sec,
the second join should be much faster (archived data in memory),
so assume 10 sec for the first image and 1 sec for the second
image from a given pair.
===
6) In practice, we will have 201 independent ingest tables
(one per CCD), so we will deal with 201 joins. Each join
will need ~1/201 of 5GB. If we partition data spatially,
all 201 joins will need the same partition. It would be
MUCH more efficient if they used multiple partitions.
So the question is how to partition archived data so that
different ccds will be joined with different partitions?
(per Kem, it is not obvious...)
===
If we use the above assumptions, database will have to
a) locate the 5GB in the 10TB data set
b) crunch through 5GB of data
all that in 10 sec.
Locating 5GB in 10TB:
- if we use index that is 30% of data size (optimistic),
index would be 3TB. A single scan through such
index in 10 sec would require 300GB/sec.
That is obviously not cheap.
Crunching through 5GB of data
- that is 500MB/sec
===
Summary: as Jim Gray correctly guessed, we have an
IO problem (yes, even at the base camp).
1) We need to come up with a way to partition data such
that different ccds use different partitions
2) It would be best if the ccds that use the same
partition would reuse the same pieces of index
3) we have to very carefully tune/cluster indexes.
(e.g., we better avoid full index scan)
4) I would really like to see the query that we will
run at the base that will join ingested data with
archived data. Can someone send it asap?
Comments? Corrections?
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
