Hello all, My notes regarding the action items from the meeting:
Spatial/temporal indexing and partitioning/clustering approach - Jacek, Serge, Maria - 11/15/06 Incremental object/source table update approach - Jacek, Ani - 11/15/06 Update disk size, I/o models - Jacek - 11/3/06 Prototype database records - Sergei - per LLNL schedule Data quality analysis proposal - Deborah - 10/31/06 It was a great meeting! Jeff > From: Jacek Becla <[EMAIL PROTECTED]> > Organization: Stanford Linear Accelerator Center > Reply-To: LSST Data Management <[email protected]> > Date: Fri, 20 Oct 2006 13:55:29 -0700 > To: LSST Data Management <[email protected]>, Deborah Levine > <[EMAIL PROTECTED]>, Bruce Berriman <[EMAIL PROTECTED]>, > [EMAIL PROTECTED] > Subject: [LSST-data] notes from "schema/cross-match" meeting > > Keywords: DataAccWG > > > It was 1 1/2 day meeting in Tucson, AZ. > Two main topics: > - schema > - cross matching > > > Attendees: > LSST: > Jeff Kantor > Tim Axelrod > Ani Thakar > Maria Nieto-Santisteban > Kem Cook > Sergei Nikolaev > Jacek Becla > IPAC/Caltech: > Deborah Levine > Serge Monkewitz > Bruce Berriman > > [These notes cover the schema discussion only] > > > decisions > ========= > (most important close to the top) > > > - Source table ideally should contain individual measurements. > Because of unrealistic database size (over 1 exabyte in 2023 > based on first rough estimates, but in fact we know it will > go down ~x4 if we apply some pending changes). We need to find > ways to bring the size further down even if it is at the expense > of adding extra complexity. Things to do/consider: > a) update db spreadsheets: proximity to galactic plane has > changed recently (should reduce db size, not sure how much) > b) keep individual measurements for detections if above certain > signal-to-noise threshold, and keep an average for > detections below the threshold > c) consider splitting source table into 2 tables: one for stars > and one for galaxies. Hide these implementation details from > users. If we split, the source-galaxy table should contain > stellar information (either through pointer or by containment, > containment means loosing space due to replication, pointer > means loosing performance) > d) use data compression, in particular for less frequently > accessed data > e) don't keep in deep storage indexes for older releases > f) maybe keep sources with low signal-to-noise in separate > table(s) and use less bytes (reduce precision) > g) don't keep data in deep storage if it is on disk (last > two releases) > [loosing possibility to bring data from tape if disk fails, > but have another copy at other center, so can bring it > from there] > i) reducing release frequency should help, see next point > > > - we should release data once per year (used to be twice). > Exception: the first couple of releases: DR1 after the first > 6 months, DR2 6 months later, then once per year > - yes, we should still keep 2 most recent > releases on disk > > > - we should run deep detection twice more frequently > than releases (e.g. after 3rd month, then for DR1, > then 3 months later, then DR2, then every 6 months...) > > > - we should keep DIASource table at the Base Camp. > It should contain data from ~the last 6 months > ("sliding window" or "since the last deep detection"). > This is <10TB is size, so no big impact on hardware > > > - alerts will have embedded database object ids pointing > to other tables, therefore we must maintain the same > ids between catalogs produced at the base camp and > catalogs produced at the archive center. Some implications: > we may not rely on random numbers (like auto_increment), > pipelines should generate unique ids > > > - we should send updates of the Object Catalog to > Main Archive each night for QA purposes > > > - introduce specialized object table for moving objects. > There might be more specialized object table (e.g. for > Deep Detection, Difference Image...) > > > - we should preserve DIASource data for ever. This is because > of use case: "better light curves in crowded regions" (?) > > > - We need to reprocess all DIASource once per release. > This is because each release will use different set > of templates, and new templates may "invalidate" DIASources > done with older templates > > > - we should have VarObj table, it should contain > a copy of variable objects. The big Object table should > contain the variable objects as planned before > > > - we do not allow orphan sources (sources without a > corresponding object) > > > - the changes in the "proposed schema changes" document > discussed at the meeting are "approved": ok to put them > in the schema (unless they conflict with what these > notes say) > > > - other schema changes needed which are not captured by > the document mentioned above: > > a) schema should capture what template image was used > to produce difference image > > b) need to store the source classification persistently > (in DIASource table). This is the table shown by Tim, > it had columns: cosmic ray/negative excursion, > positive excursion, fast movers, flash > and rows: present in both visits, shape differs > in two visits, elliptical after PSF deconvolve, positive > flux excursion, association pl action) > Schema should be flexible enough to support future changes > > c) alert should contain cut outs (postagestamps) of the > corresponding image and template > - we will store these cut outs for some time (year?), > not for ever. They can be regenerated in the future > if needed > > d) provenance of the coadded image is different from provenance > of Difference Image > > === end of schema changes > > > - we should periodically verify whether our prototype schema > is compatible with mysql and sql server. The master > schema should be kept in ascii in docushare, and also > loaded to EA > > > - we need to change the baseline: should say "measurement > per visit", not "per exposure". DB spreadsheet does > not need to change > > > - we should continue this discussion with the same > group of people. Will do half-day phone meetings > once per month (likely Friday 9:00am - 12:00 or 1pm PDT) > > > > > > Other important points discussed > (but no decisions made) > ================================ > > - do we need to ensure object ids remain the same > across different runs of deep detection pipeline? > > - how is the Image table different from ImageWCS, ImagePSF? > > - Object table: should there be more specialized object > tables e.g. one for Difference Image Objects, > one for Deep Detection Objects > > - how do we implement link between objects and sources? > E.g. do we need an extra table for keeping these links? > > - orbit is a property of Object, but it takes lots of space, > should we keep it in Object table? > > - is it worth to have a requirement for "standing query"? > At the moment a user need to query alerts periodically > to find what she/he is looking for. > > - we need to review the schema from the perspective of > storing provenance (not high priority at the moment, > but it is a big/important topic) > > - we need to better understand native db support for > overlapping partitions (needed for neighbor queries) > > > some other things > ================= > > - we have 3 more years for r&d (so don't panic if > you see "exabytes") > > - we will need to re-associate frequently in the first year > (Source->Object will change) > > - will will know ~30 sec ahead of time what field will > be observed next. Could use that to warm up db caches > > - db spreadsheets should use scientific notation > > > Thanks, > Jacek > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
