Hi,
A few thoughts following our telecon:
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
I just had a quick word with Kem, here is the summary plus
some estimates/comments I added.
1) What is the expected size of pre-cached archived data
at the base camp?
We are expecting to have roughly 100TB of image data,
corresponding catalog data will probably be ~10% of that
So it is ~10TB (upper bound).
2) How much of that data will be needed on average per image?
~5GB (upper bound)
I think we need a combination of deep object data, which does not have
time history (it is constructed from deep stacks of images over a long
time span), and shallower object data which does have time history. For
the deep object data, if we assume 50 galaxies per arcmin**2, we have
about 2e6 objects per image. We need to work on the object data
definition, but let's suppose that for each filter we keep three
different kinds of magnitudes and their associated errors. We will
have some kind of classification, and a variety of data quality flags.
There will also be detailed shape information, but we likely don't need
this at base, so I ignore that here. If each of those mags/errors is a
32 bit float, the flags take 16 bits, and the classification 8 bits, and
we have 5 bands (U will not go deep), then we need 2e6*5 filters*(3
mags*2 quantities/mag * 4 bytes/quantity + 3 bytes for flags) = 2.7 GB.
The time dependent data will basically be limited to stars at roughly
25th in V. The density on the sky is highly variable, but an average
value of about 2 per arcmin**2 is probably not too far off. This gives
us roughly 1e5 stars per image on average. I think we will need summary
information for each star, plus perhaps the most recent ten measurements
(in whichever bands we have them). Again, we suffer from not having
defined our object data well. But suppose we summarize the star with
10 floats per band, and then have for each time point 2 floats plus 2
bytes of quality flags. Then for each star we need 5*10*4 + 10*(2*4 +
2) = 300 bytes. The total for stars is then 0.07 GB.
So, given the assumptions about what we need, I certainly think that
Kem's number is in the right ballpark. However, I wonder if we really
need to prestage all this data for every image. After all, we are
operating on difference images at base, and even granted that the
detection are supplemented by some list of objects to always measure, we
will be needing object information for only a tiny fraction of the stars
in the image - perhaps a few percent. So, we need to fetch info for a
few * 1e3 objects from the total of 2e6 total (mostly deep) objects for
which we have data. Does this change our strategy?
Tim
begin:vcard
fn:Tim Axelrod
n:Axelrod;Tim
org:;Large Synoptic Survey Telescope
email;internet:[EMAIL PROTECTED]
title:Data Management Project Scientist
tel;work:520-322-8735
version:2.1
end:vcard
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
