Keywords: DataAccWG
Attendees:
Nicole Silvestri
Russell Owen
Tim Axelrod
Ani Thakar
Maria Nieto-Santisteban
Kem Cook
Mark Huber
Sergei Nikolaev
Ray Plante
Jacek Becla
EA issues related to generating SQL
- Nicole will send bullets today
- Jacek will post in the plone
UW:
- have not stressed the database: using small amounts of data
- focused on capturing science info
- initially did not pay attention to uml
UW schema:
- object table which contains info about detections (sources),
also contains astro objects that are clusters
- tie table (many to many relations): detections to objects
- image table, image metadata
- objects know which image it came from
- images tie back to algorithm table
- wcs table
- psf model tables, storing an unparsed text
- ingesting lots of different kinds of data,
different tables for different data sources (sdss most complex)
- could not use precursor, headers missing lots of
critical info
- because of missing things in the model
--> Ray will look into it once we post UW schema
- largely successful
- main issue: object classification non-existing
- have not found one simple uniform classification scheme
SQL and EA
- painful and slow to do low-level db design in EA
- much faster in standard text editor than in EA
- if we use standard editor, not EA to generate schema,
then how to make sure EA is consistent with SQL?
- can try to import ascii SQL to EA, still how
to reliably compare 2 versions in EA?
- will have "many eyes checking it"
- also, EA not very useful to generate schemas
for different databases
- lots of manual work, changing types, etc.
--> decision: build SQL schema outside of EA
- still useful to document EA bugs
Compatibility issues, database-specific tricks
- using "basic", standard SQL would reduce number
of compatibility issues across different database
systems
- but we have to take advantage of db specific tricks
to get performance
- baseline schema in MySQL
- issue: how to convert schema to SQL Server, or DB2?
- [ hot discussion here =) ]
- many compatibility issues: unsigned, foreign keys,
decimal/float...
- automatic conversion non trivial
- could use special keywords in comment section
for attributes in question to help automatically convert
- conversion will not be done frequently,
once the bulk conversion done then only need to
convert changes/additions, so maybe automatic
conversion not so crucial
- need to keep track of schema changes to know what
needs to be converted (among other reasons...)
(UW is using Subversion already)
- conclusion:
- Ani/Maria will try to convert existing precursor
schema "by hand", document encountered problems
Merging schemas
- have 3 schemas
a) UW schema
b) precursor schema
c) lsst schema that can be derived from UML model
- how to merge them?
1) UW + precursor: use precursor schema as base and
add missing things by looking at UW schema
[Sergei+Nicole+Russell]
2) independently, look at UML model and derive schema
from it [Sergei + Jacek]
- number 1 is much simpler, number 2 requires lots of work
DC1 schema
- DC1: bandwidth test, not algorithmic, want to show
we can move bytes and ingest into tables
- For the DC1 test will need yet another schema
- data used for DC1: from sextractor, simulated data
- probably UW schema closest to it
- need to discuss more (next meeting)
Action items:
- Nicole: send EA bugs
- Jacek: post in plone
- Nicole/Russell: publish UW schema on UW website
- Jacek: put precursor schema in plone, add link
from plone to UW schema
- Ray: check what is missing in headers in UML model
that UW needed
- Ani/Maria: migrate precursor schema to SQL-Server
compatible format
- Sergei+Nicole+Russell: merging UW + precursor schema
- Sergei+Jacek: derive schema from UML
Have a lot to discuss, so next LSST Database meeting
NEXT WEEK (same time), not in two weeks. Will discuss:
- logistics how to track schema changes
(cvs? subversion? which server to use?)
- DC1 schema
- progress/issues/update:
- merging UW with precursor schema
- deriving schema from UML
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We have a Database telecon tomorrow (Wednesday) at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Main topic: merging schemas (UW and "LSST prototype").
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
