Keywords: DataAccWG Hi,
I just had a quick word with Kem, here is the summary plus some estimates/comments I added. 1) What is the expected size of pre-cached archived data at the base camp? We are expecting to have roughly 100TB of image data, corresponding catalog data will probably be ~10% of that So it is ~10TB (upper bound). 2) How much of that data will be needed on average per image? ~5GB (upper bound) 3) Is it likely we will re-use the same data for the next images? Two adjacent images will always need the same data (images will be taken "in pairs"). Beyond that, there might be ~3-5% overlap, so it is almost negligible. 4) Is there a natural way of splitting the "10TB catalog" so that we can partition it and spread across multiple servers? yes, we could split it across two axes: a) could do it per filter, but we would need to re-cluster data while precaching it. Data not spread uniformly across filters: most observations taken in 2 filters (R, I), very few in G. So this is not a good partitioning. b) can partition data spatially: based on position in the sky. Have to deal with edges, so probably will need to have some overlap between different partitions, but other than that, it is nice, can answer the join query by going directly to one partition. 5) what is the accepted latency the join of Ingested Data with precached archived data can take? We will need <2-3 sec to ingest data. Probably it is ok if the join for the first image from given pair takes ~10 sec, the second join should be much faster (archived data in memory), so assume 10 sec for the first image and 1 sec for the second image from a given pair. === 6) In practice, we will have 201 independent ingest tables (one per CCD), so we will deal with 201 joins. Each join will need ~1/201 of 5GB. If we partition data spatially, all 201 joins will need the same partition. It would be MUCH more efficient if they used multiple partitions. So the question is how to partition archived data so that different ccds will be joined with different partitions? (per Kem, it is not obvious...) === If we use the above assumptions, database will have to a) locate the 5GB in the 10TB data set b) crunch through 5GB of data all that in 10 sec. Locating 5GB in 10TB: - if we use index that is 30% of data size (optimistic), index would be 3TB. A single scan through such index in 10 sec would require 300GB/sec. That is obviously not cheap. Crunching through 5GB of data - that is 500MB/sec === Summary: as Jim Gray correctly guessed, we have an IO problem (yes, even at the base camp). 1) We need to come up with a way to partition data such that different ccds use different partitions 2) It would be best if the ccds that use the same partition would reuse the same pieces of index 3) we have to very carefully tune/cluster indexes. (e.g., we better avoid full index scan) 4) I would really like to see the query that we will run at the base that will join ingested data with archived data. Can someone send it asap? Comments? Corrections? Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
