Hello all, I raised the a question with the Telescope/Control System team this morning:
Is there any constraint on operations by the scheduler such that we know ahead of time the telescope won't go to certain fields that night so that we do not have to have the entire sky template and catalog pre-cached each night? The answer was probably not, so we should assume we need it all pre-cached. >From an archive to base transfer standpoint, that means only what has changed since the last transfer of course. Jeff > From: Jacek Becla <[EMAIL PROTECTED]> > Organization: Stanford Linear Accelerator Center > Reply-To: LSST Data Management <[email protected]> > Date: Thu, 04 May 2006 12:22:28 -0700 > To: LSST Data Management <[email protected]> > Subject: [LSST-data] Base Camp Database requirements > > Keywords: DataAccWG > > Hi, > > I just had a quick word with Kem, here is the summary plus > some estimates/comments I added. > > > 1) What is the expected size of pre-cached archived data > at the base camp? > We are expecting to have roughly 100TB of image data, > corresponding catalog data will probably be ~10% of that > So it is ~10TB (upper bound). > > 2) How much of that data will be needed on average per image? > ~5GB (upper bound) > > 3) Is it likely we will re-use the same data for the next images? > Two adjacent images will always need the same data (images > will be taken "in pairs"). Beyond that, there might be > ~3-5% overlap, so it is almost negligible. > > 4) Is there a natural way of splitting the "10TB catalog" > so that we can partition it and spread across multiple servers? > yes, we could split it across two axes: > a) could do it per filter, but we would need to re-cluster > data while precaching it. Data not spread uniformly > across filters: most observations taken in 2 filters (R, I), > very few in G. So this is not a good partitioning. > > b) can partition data spatially: based on position in the sky. > Have to deal with edges, so probably will need to have some > overlap between different partitions, but other than that, > it is nice, can answer the join query by going directly > to one partition. > > > 5) what is the accepted latency the join of Ingested Data > with precached archived data can take? > We will need <2-3 sec to ingest data. Probably it is ok > if the join for the first image from given pair takes ~10 sec, > the second join should be much faster (archived data in memory), > so assume 10 sec for the first image and 1 sec for the second > image from a given pair. > > === > > 6) In practice, we will have 201 independent ingest tables > (one per CCD), so we will deal with 201 joins. Each join > will need ~1/201 of 5GB. If we partition data spatially, > all 201 joins will need the same partition. It would be > MUCH more efficient if they used multiple partitions. > So the question is how to partition archived data so that > different ccds will be joined with different partitions? > (per Kem, it is not obvious...) > > === > > If we use the above assumptions, database will have to > a) locate the 5GB in the 10TB data set > b) crunch through 5GB of data > all that in 10 sec. > > Locating 5GB in 10TB: > - if we use index that is 30% of data size (optimistic), > index would be 3TB. A single scan through such > index in 10 sec would require 300GB/sec. > That is obviously not cheap. > > Crunching through 5GB of data > - that is 500MB/sec > > === > > Summary: as Jim Gray correctly guessed, we have an > IO problem (yes, even at the base camp). > > 1) We need to come up with a way to partition data such > that different ccds use different partitions > > 2) It would be best if the ccds that use the same > partition would reuse the same pieces of index > > 3) we have to very carefully tune/cluster indexes. > (e.g., we better avoid full index scan) > > 4) I would really like to see the query that we will > run at the base that will join ingested data with > archived data. Can someone send it asap? > > > Comments? Corrections? > Jacek > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
