Tim: As a database guy, these are "easy" numbers. The data lossless-compresses 2.5x to ~3TB raw and 1.2TB objects or about 5TB/night. In 2012 disks will be in the 5TB range so we can triplicate the data and consume one disk per nite (200$)
The raw data arrives at 200MBps The pipelines will be a challenge but the database aspects (once the data is processed) is easy. The raw data compresses to 100MBps which is comparable to disk bandwidth in that era. Dealing with ~150k objects every 30 seconds is 5k objects/second is easy. My laptop can do 8k "banking" transactions per second -- and I think the catalog object-lookups and inserts are simpler than that. So, this begins to look like a very doable project. The one challenge is getting the 100MBps data stream from the mountain to your office. It is probably a done deal, but that is the part that I do not see as a sure thing. It is less than 1Gbps so the LAN hardware is trivial, but the WAN will be a challenge. (yes, we are doing 8Gbps from CERN to Tokyo, but only sometimes and with heroes at each end). Jim Gray Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 8222 fax: 425 706 7329 [EMAIL PROTECTED] http://research.Microsoft.com/~gray -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tim Axelrod Sent: Thursday, March 16, 2006 7:10 AM To: Jacek Becla Cc: LSST Data Management Subject: [LSST-data] Re: prestaged data - size? Hi Jacek, Here's a rough calculation. I assume a typical night of 10 hours, which corresponds to about 1000 observations of distinct fields (sky positions). For each of those sky positions, we need at the base: a) Template images for the fields. At roughly 6 GB/image, that gives us 6TB b) Recent catalog data for all objects in the fields brighter than about V=25. How much data that is will vary quite a bit, as a previous email I sent out explains in some detail. For present purposes, I assume that there are 150000 objects/deg^2 that qualify. I assume that for each object we need summary information, which I estimate at 200 bytes (very rough), and the latest 10 epochs of measurements at 50 bytes each. This then gives us 1000 fields * 150000 * 10sq deg * (200 + 10*50) = 1.5TB. I'd say at present this is no better than a factor of two estimate. c) calibration images - this is maybe 100GB, negligible. So a rough guess is 7.5TB. Cheers, Tim Jacek Becla wrote: > Jeff/Tim/Ray > > I'd like to start thinking about details of pre-staging data at the > base / partitioning it, but before I do that I need to understand how > much data we are talking about per night (the question was brought up > at today's Database telecon). > > If you could find that out and give me even a rough estimate whether > it is a terabyte or a hundred terabytes, that would help a lot. > > Thanks, > Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
