Keywords: DataAccWG
It was 1 1/2 day meeting in Tucson, AZ.
Two main topics:
- schema
- cross matching
Attendees:
LSST:
Jeff Kantor
Tim Axelrod
Ani Thakar
Maria Nieto-Santisteban
Kem Cook
Sergei Nikolaev
Jacek Becla
IPAC/Caltech:
Deborah Levine
Serge Monkewitz
Bruce Berriman
[These notes cover the schema discussion only]
decisions
=========
(most important close to the top)
- Source table ideally should contain individual measurements.
Because of unrealistic database size (over 1 exabyte in 2023
based on first rough estimates, but in fact we know it will
go down ~x4 if we apply some pending changes). We need to find
ways to bring the size further down even if it is at the expense
of adding extra complexity. Things to do/consider:
a) update db spreadsheets: proximity to galactic plane has
changed recently (should reduce db size, not sure how much)
b) keep individual measurements for detections if above certain
signal-to-noise threshold, and keep an average for
detections below the threshold
c) consider splitting source table into 2 tables: one for stars
and one for galaxies. Hide these implementation details from
users. If we split, the source-galaxy table should contain
stellar information (either through pointer or by containment,
containment means loosing space due to replication, pointer
means loosing performance)
d) use data compression, in particular for less frequently
accessed data
e) don't keep in deep storage indexes for older releases
f) maybe keep sources with low signal-to-noise in separate
table(s) and use less bytes (reduce precision)
g) don't keep data in deep storage if it is on disk (last
two releases)
[loosing possibility to bring data from tape if disk fails,
but have another copy at other center, so can bring it
from there]
i) reducing release frequency should help, see next point
- we should release data once per year (used to be twice).
Exception: the first couple of releases: DR1 after the first
6 months, DR2 6 months later, then once per year
- yes, we should still keep 2 most recent
releases on disk
- we should run deep detection twice more frequently
than releases (e.g. after 3rd month, then for DR1,
then 3 months later, then DR2, then every 6 months...)
- we should keep DIASource table at the Base Camp.
It should contain data from ~the last 6 months
("sliding window" or "since the last deep detection").
This is <10TB is size, so no big impact on hardware
- alerts will have embedded database object ids pointing
to other tables, therefore we must maintain the same
ids between catalogs produced at the base camp and
catalogs produced at the archive center. Some implications:
we may not rely on random numbers (like auto_increment),
pipelines should generate unique ids
- we should send updates of the Object Catalog to
Main Archive each night for QA purposes
- introduce specialized object table for moving objects.
There might be more specialized object table (e.g. for
Deep Detection, Difference Image...)
- we should preserve DIASource data for ever. This is because
of use case: "better light curves in crowded regions" (?)
- We need to reprocess all DIASource once per release.
This is because each release will use different set
of templates, and new templates may "invalidate" DIASources
done with older templates
- we should have VarObj table, it should contain
a copy of variable objects. The big Object table should
contain the variable objects as planned before
- we do not allow orphan sources (sources without a
corresponding object)
- the changes in the "proposed schema changes" document
discussed at the meeting are "approved": ok to put them
in the schema (unless they conflict with what these
notes say)
- other schema changes needed which are not captured by
the document mentioned above:
a) schema should capture what template image was used
to produce difference image
b) need to store the source classification persistently
(in DIASource table). This is the table shown by Tim,
it had columns: cosmic ray/negative excursion,
positive excursion, fast movers, flash
and rows: present in both visits, shape differs
in two visits, elliptical after PSF deconvolve, positive
flux excursion, association pl action)
Schema should be flexible enough to support future changes
c) alert should contain cut outs (postagestamps) of the
corresponding image and template
- we will store these cut outs for some time (year?),
not for ever. They can be regenerated in the future
if needed
d) provenance of the coadded image is different from provenance
of Difference Image
=== end of schema changes
- we should periodically verify whether our prototype schema
is compatible with mysql and sql server. The master
schema should be kept in ascii in docushare, and also
loaded to EA
- we need to change the baseline: should say "measurement
per visit", not "per exposure". DB spreadsheet does
not need to change
- we should continue this discussion with the same
group of people. Will do half-day phone meetings
once per month (likely Friday 9:00am - 12:00 or 1pm PDT)
Other important points discussed
(but no decisions made)
================================
- do we need to ensure object ids remain the same
across different runs of deep detection pipeline?
- how is the Image table different from ImageWCS, ImagePSF?
- Object table: should there be more specialized object
tables e.g. one for Difference Image Objects,
one for Deep Detection Objects
- how do we implement link between objects and sources?
E.g. do we need an extra table for keeping these links?
- orbit is a property of Object, but it takes lots of space,
should we keep it in Object table?
- is it worth to have a requirement for "standing query"?
At the moment a user need to query alerts periodically
to find what she/he is looking for.
- we need to review the schema from the perspective of
storing provenance (not high priority at the moment,
but it is a big/important topic)
- we need to better understand native db support for
overlapping partitions (needed for neighbor queries)
some other things
=================
- we have 3 more years for r&d (so don't panic if
you see "exabytes")
- we will need to re-associate frequently in the first year
(Source->Object will change)
- will will know ~30 sec ahead of time what field will
be observed next. Could use that to warm up db caches
- db spreadsheets should use scientific notation
Thanks,
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
