Hello all,
This fits very well into the approach I suggested (split
the sky into 5000 2GB-tables), the only change would be
adding an extra step that selects small number of objects
and loads them to memory, instead of blindly loading
one 2GB table and parts of few adjacent tables. This further
reduces required disk IO at the Base - very nice :-)
Jacek
Kem Cook wrote:
Hi,
Got it. Staging the queries that way probably would be faster and more
efficient.
cheers,
Kem
Hi Kem,
I am in complete agreement! I guess I wasn't clear enough in what I
was proposing. I am not suggesting that we ignore the deep detection
data in alerting at base. What I *am* suggesting is that it may be
more efficient to query that data for the relatively small number of
detected transients rather than to blindly load it all into memory for
every frame.
Tim
Kem Cook wrote:
Hi,
So, we may not need all the static data, but I thought that it would be
value added to be able to say that a new transient occured at the
position
of a known, but non-variable object (say a distant, barely resolved
galaxy), or that a new transient appeared x arcsec offset from the
nucleus of a rather well resolved galaxy. or that there was a transient
at
the position of a known M dwarf (perviously non-variable).
Kem
Hi,
A few thoughts following our telecon:
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
I just had a quick word with Kem, here is the summary plus
some estimates/comments I added.
1) What is the expected size of pre-cached archived data
at the base camp?
We are expecting to have roughly 100TB of image data,
corresponding catalog data will probably be ~10% of that
So it is ~10TB (upper bound).
2) How much of that data will be needed on average per image?
~5GB (upper bound)
I think we need a combination of deep object data, which does not have
time history (it is constructed from deep stacks of images over a long
time span), and shallower object data which does have time history. For
the deep object data, if we assume 50 galaxies per arcmin**2, we have
about 2e6 objects per image. We need to work on the object data
definition, but let's suppose that for each filter we keep three
different kinds of magnitudes and their associated errors. We will
have some kind of classification, and a variety of data quality flags.
There will also be detailed shape information, but we likely don't need
this at base, so I ignore that here. If each of those mags/errors is a
32 bit float, the flags take 16 bits, and the classification 8 bits, and
we have 5 bands (U will not go deep), then we need 2e6*5 filters*(3
mags*2 quantities/mag * 4 bytes/quantity + 3 bytes for flags) = 2.7 GB.
The time dependent data will basically be limited to stars at roughly
25th in V. The density on the sky is highly variable, but an average
value of about 2 per arcmin**2 is probably not too far off. This gives
us roughly 1e5 stars per image on average. I think we will need summary
information for each star, plus perhaps the most recent ten measurements
(in whichever bands we have them). Again, we suffer from not having
defined our object data well. But suppose we summarize the star with
10 floats per band, and then have for each time point 2 floats plus 2
bytes of quality flags. Then for each star we need 5*10*4 + 10*(2*4 +
2) = 300 bytes. The total for stars is then 0.07 GB.
So, given the assumptions about what we need, I certainly think that
Kem's number is in the right ballpark. However, I wonder if we really
need to prestage all this data for every image. After all, we are
operating on difference images at base, and even granted that the
detection are supplemented by some list of objects to always measure, we
will be needing object information for only a tiny fraction of the stars
in the image - perhaps a few percent. So, we need to fetch info for a
few * 1e3 objects from the total of 2e6 total (mostly deep) objects for
which we have data. Does this change our strategy?
Tim
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
