This sort of "prestaging" will be important not only for the object
table.
At some time before an exposure (either many slews before or, at
worst, when the shutter opens!), we will know the "rough" (to within
pointing accuracy) coordinates of the image. At that point all sorts
of prestaging can start, including...
- prestaging the template image(s), and starting whatever
manipulation is necessary to account for possible gross differences
in image rotation,
- prestaging astrometric catalogs and other input needed for data
reduction
and, as you point out,
- prestaging catalogs and other input needed for post-reduction
transient analysis, including previous transients, expected moving
objects, etc.
Cheers,
Chris
On Mar 16, 2006, at 10:41 AM, Jacek Becla wrote:
Hi all,
If we know so well which fields will be observed in the next few
minutes, why don't we query the object table few minutes earlier
and load all needed data to memory? That should be simple, unless
I am missing something (am I)?
thanks,
Jacek
Tim Axelrod wrote:
Hi Kem,
Yes, I agree with all your points. On the other hand...there
should be a significant overlap between the information needed
tonight and the information needed tomorrow night. The op-sim
output should give us a good idea of what that overlap factor is,
but certainly it should significantly ease the average amount of
data needed by the base each night.
Tim
Kem Cook wrote:
Hi Tim and Jacek,
Just to play devil's advocate, we may need quite a bit more than
6 TB of
template images. While we may only take 1000 images between
astronomical
twilights, we may not be able to accurately predict which fields and
filters these will be. Variable conditions may push us to
observe fields
which are lower priority. We may also do z and y observing in
twilight
conditions which gives us a few hundred more field/filter
combinations. We may also need color information for diffraction
correction
calculations. Might we also need our templates to be better than
integer
precision?
Baseline with twilight observing:
1200 images x 6.4 GB/image X 0.80 unique area = 6.1 TB perfect
planning
x2 for color information
x2 for float
x1.3 for spare field/filters for unpredicted weather/seeing backup
gulp, 31.7 TB (! upper limit, I hope)
Kem
Hi Jacek,
Here's a rough calculation. I assume a typical night of 10 hours,
which corresponds to about 1000 observations of distinct fields
(sky
positions). For each of those sky positions, we need at the base:
a) Template images for the fields. At roughly 6 GB/image, that
gives us
6TB
b) Recent catalog data for all objects in the fields brighter
than about
V=25. How much data that is will vary quite a bit, as a
previous email
I sent out explains in some detail. For present purposes, I
assume
that there are 150000 objects/deg^2 that qualify. I assume
that for
each object we need summary information, which I estimate at 200
bytes
(very rough), and the latest 10 epochs of measurements at 50
bytes each.
This then gives us 1000 fields * 150000 * 10sq deg * (200 +
10*50) =
1.5TB. I'd say at present this is no better than a factor of two
estimate.
c) calibration images - this is maybe 100GB, negligible.
So a rough guess is 7.5TB.
Cheers,
Tim
Jacek Becla wrote:
Jeff/Tim/Ray
I'd like to start thinking about details of pre-staging
data at the base / partitioning it, but before I do that
I need to understand how much data we are talking about
per night (the question was brought up at today's Database
telecon).
If you could find that out and give me even a rough estimate
whether it is a terabyte or a hundred terabytes, that
would help a lot.
Thanks,
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
========================================================================
=
Dr. R. Chris Smith EMAIL: [EMAIL PROTECTED]
National Optical Astronomy Observatory
Cerro Tololo Inter-American Observatory
Casilla 603 P.O. Box 26732
La Serena, CHILE Tucson, AZ 85726-6732
Office Phone: +56-51-205200/205214 FAX number: 520-318-8259
FAX number : +56-51-205212 WWW: http://www.ctio.noao.edu/
~chris .
========================================================================
=
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
