Hi All,
Following up on today's database telecon, I've spent a bit more time
with Jacek's database storage estimating spreadsheet. There has been
an enormous increase in the estimated size of the databases over the
last few weeks! This is apparently due to two factors, one of which we
have explicitly discussed and one of which has appeared without much
discussion. The one we have discussed is of course the effect of
observing the galactic plane. At the beginning of our estimating, the
assumption was that the plane would be avoided, as the cadence simulator
runs have done. All our discussions recognized the sensitivity of the
databases sizes to this assumption. We have apparently decided to
observe through the plane, including the galactic center, and this
increases the number of stars by a large factor, which directly affects
the size of the object and source databases. I am comfortable with
this, though I think there will be some implications for our photometry
algorithms.
I'd like to highlight the second factor, which we have not really
discussed. As background, one of the outcomes of the photometry
working group we convened late last year was that photometry at base
camp would be performed on the difference image formed from an
individual LSST image and a high signal-to-noise template for the
field. The resulting source database would then naturally include only
sources that varied significantly from the template. We recognized,
however, that this would unduly hamper transient science, and we decided
that it was necessary to photometer a pre-identified set of objects of
particular interest regardless of their signal-to-noise in a particular
difference image. This set would include all known variables as well
as objects flagged as interesting from a-priori information such as
their position in color-magnitude space or from observations in other
wavelengths such as the IR. The assumption was that the total size of
this set would likely be only a little larger than the set of variables,
say a few percent of the total objects detectable in a single LSST
image. Photometry of all objects in a field, in particular galaxies,
would be performed only infrequently, on stacks of images. The
resulting design of the detection and association pipeline was
presented in one of our AAS posters at the end of last year.
The current database sizing calculation is now assuming that we detect
and measure *all* sources in each image, including galaxies. This
increases the number of entries in the source database by a factor of
between 20 and 50. Further, including extended object properties, such
as spatial moments of the intensity distribution, has increased the size
of each entry by a significant factor. This is the major contributor
to the growth of the database size from about 15TB for the first year to
about 600TB! We should carefully consider whether we wish to make this
change in strategy. It is a major one, which affects not only database
sizes but the requirements for the photometry pipeline at base camp and
the cpu required to run it.
Cheers,
Tim
Jacek Becla wrote:
Keywords: DataAccWG
Hello all,
I published an updated version of database storage estimates
in docushare, same place as before: Collection-413. It captures
the assumptions proposed by Zeljko related to galactic plane.
Due to assuming closer proximity to galactic plane the numbers
went up roughly by 50% - much less than we feared.
The disk IO still uses the old numbers, I expect to have that
updated later this week.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
begin:vcard
fn:Tim Axelrod
n:Axelrod;Tim
org:;Large Synoptic Survey Telescope
email;internet:[EMAIL PROTECTED]
title:Data Management Project Scientist
tel;work:520-322-8735
version:2.1
end:vcard
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
