In article <[EMAIL PROTECTED]>, "Sergei Nikolaev" <[EMAIL PROTECTED]> wrote:
> Nicole and Russ, > > I looked at your DB schema. Here are my initial ideas/notes on how to > merge it with the prototype schema: > > 1) Your WCS tables (NOAO4mWCS, TangentWCS, APO2_5mWCS, MSO50inWCS) can be > "merged" into a single "entity" in the prototype schema (unless we envision > multiple WCS solutions for LSST). I used the word "entity" and not > "table", because we need to think more about whether to keep the resulting > WCS info in a separate table or as part of the image table. I totally agree with you. In fact one WCS table would probably have sufficed in any case if all our data sources had used standard parameters. But SDSS needed its own model (the standard WCS model isn't sophisticated enough to handle drift-scanned data) and we just went with one table per data source at that point. As to where to put the WCS, I strongly suggest you don't couple it *too* tightly to the images. Remember that you may resolve the WCS many times for a given image! In fact for LSST I would suggest adding an AlgRunID field to the WCS table. I didn't do that because we plan to use existing WCS parameters for all our images. > 2) Same goes for PSF model tables (DophotPSFModel, DaophotPSFModel, > PhotoPSFModel) -- a single table/entity can be made from these three with > attributes relevant to LSST PSF algorithm. I completely agree. > 3) Your PipelineRun and AlgRun tables capture provenance data, and are > very useful. I will try to incorporate most of their attributes into the > prototype schema. Your TieImageToAlgRun table appears redundant: couldn't > Image and AlgRun be tied directly (AlgRun will have pointer to imageID)? I'm doubful. I believe the tie table is required because an image can be used for many different algorithm runs. For us an algorithm may have photometry run from many algorithms. For LSST an image may be used to compute WCS (more than once) and find sources (more than once)., Also, we may want to count image ingest as an algorithm run. > 4) Your Image table is the one I looked at more or less in detail, and > have more specific comments about it: > - I assume your image table relates to a single amp (it has ccd and amp > attributes). It might be advantageous to have a separate table for the > whole FOV assembly -- that way one does not need to duplicate attributes > common to the whole mosaic (like ExpTime, UT1, etc) I agree. > - A bunch of varchar fields related to names of files are better taken out > to a separate table and linked to this one by a foreign key. The reason is > (variable) text fields slow down the query performance. Ah yes, all that calibration provenance that we may end up never even using. I didn't spend much time on that; it's just placeholders for us in case we start calibrating data. I think the cleanest design for LSST (and perhaps us) would be: - Use the Image table to save metadate for calibration images as well as for calibrated data. - For calibrated data, save the ImageID of each calibration image used for calibration (in the appropriate field). Then it's a smallish number of BIGINTs instead of huge varchar fields. Of course you may want to break this up into more tables for performance reasons (e.g. calibration images in one table, calibrated in another). > - Run/Rerun attributes seems redundant -- maybe Run only is enough? These are both SDSS-isms and I doubt LSST will need either, though I confess I'm not really sure what run number. Anyway, we only have them because of SDSS data. - ReRun is associated with a run of the photo pipeline--it has nothing to do with the image itself and really belongs in the WCS table and/or the AlgRun table. - I'm not sure whether Run # tells us anything that MJD (see next) doesn't, but it was handy for us to keep it around, in any case. We would like to add an MJD field to the Image table. This would be the MJD day of the *beginning* of the night's run. Thus a unique integer to identify data from a particular night. > - HTMIDLL/LR/UL/UR appear to be an attempt for spatial indexing (using HTM). > I do not want to say anything bad about HTM, but there are also built-in > efficient spatial indexes in most DB engines (using e.g. geometry data > types and R-trees). > - Az, Alt encode telescope pointing. Was there a specific reason you used > them, as opposed to RA, Dec? Fundamentally we were trying to make it easy to figure out if a particular object *should* appear in a particular image, so we could keep track of missing detections. Thus we recorded the positions of the corners of the CCD. We chose to use HTMID instead of RA, Dec, but I have no idea yet whether this was a good choice. In any case, Az/Alt were intended to be "in addition" to, not "instead" of mean position on the sky. > - There are a few attributes that I do not understand (and comments did > not help either): camcol, TAI, BinX, BinY TAI is International Atomic Time. I *STRONGLY* recommend that LSST use this as the basis for all high precision timekeeping. It has the same unit second as UTC (and is readily available from most GPS clocks, though perhaps as GPS time which has a fixed offset from TAI), but does not have the incredibly annoying leap seconds. Using TAI, one can take the difference of two times and know *exactly* how much time has elapsed--something that is impossible with UTC. In fact if there's two things I hope the LSST learns from our database and my experience with telescopes, they are: - Use TAI, not UTC for all time-critical applications. This is especially crucial for a telescope used to study time-varying phenomena. - Record UT1 (universal time) as as part of image provenance; it's a royal pain to work it out later! BinX, BinY are the binning factor in X and Y. I suggest that you at least record one bin factor (perhaps it's safe to assume the same binning in X and Y). > - An interesting point to me is that you keep as an attribute the name of > the WCStable. While this is ok, it does not allow you to use a table JOIN > on this attribute, because JOIN cannot operate on table names (I think). > It seems to me you have to query this field and interpret it outside of > the database before you can use it to form a JOIN query. I agree. It's not ideal and I didn't know how to do it cleanly. However: - The tables are so different (one WCS to another, one PSF model to another) that it's hard to imagine much useful querying going on where you didn't already tune the query for the particular table in mind. - It was optimized for the kind of work we were doing. For instance the WCS tables are solely used to compute RA, Dec during photometry algorithm runs. And the PSF tables are intended solely to keep the data around in case we figure out some use for it. (Even if we do generate WCS info ourselves, it would be done programmatically, and then table names present no problem at all.) - Queries can be done by looking at multiple WCS tables and looking for the appropriate ImageID, AlgRunID, etc. LSST won't have this issue, thank heavens because you'll only need one WCS table and one PSF model table. Thank you very much for your thoughtful comments and insights! -- Russell _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
