Jim
I agree with you that the database _loading_ part should not
be very challenging in 2012, but I would not say that
(at least not yet) about providing efficient access to the data.
Thanks,
Jacek
Jim Gray wrote:
Tim:
As a database guy, these are "easy" numbers.
The data lossless-compresses 2.5x to ~3TB raw and 1.2TB objects or
about 5TB/night.
In 2012 disks will be in the 5TB range
so we can triplicate the data and consume one disk per nite (200$)
The raw data arrives at 200MBps
The pipelines will be a challenge but the database aspects (once the
data is processed) is easy.
The raw data compresses to 100MBps which is comparable to disk
bandwidth in that era.
Dealing with ~150k objects every 30 seconds is 5k objects/second is
easy.
My laptop can do 8k "banking" transactions per second -- and I think
the catalog object-lookups
and inserts are simpler than that.
So, this begins to look like a very doable project.
The one challenge is getting the 100MBps data stream from the mountain
to your office.
It is probably a done deal, but that is the part that I do not see as a
sure thing.
It is less than 1Gbps so the LAN hardware is trivial, but the WAN will
be a challenge.
(yes, we are doing 8Gbps from CERN to Tokyo, but only sometimes and
with heroes at each end).
Jim Gray
Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778
8222 fax: 425 706 7329 [EMAIL PROTECTED]
http://research.Microsoft.com/~gray
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Tim Axelrod
Sent: Thursday, March 16, 2006 7:10 AM
To: Jacek Becla
Cc: LSST Data Management
Subject: [LSST-data] Re: prestaged data - size?
Hi Jacek,
Here's a rough calculation. I assume a typical night of 10 hours,
which corresponds to about 1000 observations of distinct fields (sky
positions). For each of those sky positions, we need at the base:
a) Template images for the fields. At roughly 6 GB/image, that gives us
6TB
b) Recent catalog data for all objects in the fields brighter than about
V=25. How much data that is will vary quite a bit, as a previous email
I sent out explains in some detail. For present purposes, I assume
that there are 150000 objects/deg^2 that qualify. I assume that for
each object we need summary information, which I estimate at 200 bytes
(very rough), and the latest 10 epochs of measurements at 50 bytes each.
This then gives us 1000 fields * 150000 * 10sq deg * (200 + 10*50) =
1.5TB. I'd say at present this is no better than a factor of two
estimate.
c) calibration images - this is maybe 100GB, negligible.
So a rough guess is 7.5TB.
Cheers,
Tim
Jacek Becla wrote:
Jeff/Tim/Ray
I'd like to start thinking about details of pre-staging data at the
base / partitioning it, but before I do that I need to understand how
much data we are talking about per night (the question was brought up
at today's Database telecon).
If you could find that out and give me even a rough estimate whether
it is a terabyte or a hundred terabytes, that would help a lot.
Thanks,
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
