Keywords: DataAccWG
Attendees:
Kem Cook
Ani Thakar
Maria Nieto-Santisteban
Nicole Silvestri
Sergei Nikolaev
Mark Huber
Ghaleb Abdulla
Russell Owen
Ray Plante
Jacek Becla
schema
======
- ultimately want one version of a clean sql schema
in EA that is not specific to any database.
So have to work towards 3 goals:
1) in EA
2) clean sql
3) one schema
"in EA" goal
- precursor and UW schemas in EA already
- merging: precursor+UW done outside of EA, but
- will import to EA once ready (as a separate package)
- importing SQL to EA not too bad, see how-to in docushare
- DC1 schema: need quickly, one-time "branch"
- create separate package in EA (likely a copy
of merged schema), tweak, then generate sql from EA
- schema derived from uml use cases:
- in EA from the very beginning
- This will be "the" LSST baseline schema
- need to "look at" the merged schema when building it
"clean sql" goal
- precursor/uw/merged/dc1 schemas: all specific to mysql
- derive schema from uml - "a clean start":
- keep it clean: no mysql
- use post-processing to map to particular db
(need to research how)
"one schema" goal
- precursor schema will be replaced by merged uw+precursor
- uw can't use merged schema. uw-schema lifetime: depends
how satisfied with tests. ~3 years
- dc1 schema will be a temporary "branch" (will preserve in
docushare, but will not improve after dc1)
- clean schema derived from uml, eventually should replace
all other schemas (think "longer term", not the next
few months)
work on schema: progress/update
- merging: have draft version, sent to uw, getting comments,
relatively straightforward
- almost none progress with deriving schema from uml
ingest tests
============
plan (for DC1):
- March: performance of single stream
- April: multiple streams (merging small tables, indexing)
- May/June: tests on teragrid
- July: final DC1 tests
slac:
- will start tests in the next few days
- performance: gave up on tuning the existing (old)
hardware: these sun disk arrays and Redhat Linux don't
go together very well, managed to get good performance
during io tests (sequential reads) by setting read-ahead
to 1MB (v large). Sort of cheating.
- will get two dual-CPU v240 with (probably) 5TB/server,
Sun hardware + Solaris OS, should have ~end of March
jhu:
- run some first tests: measured ingest speed, decimal/float
- Maria will send summary/numbers
- need to do the tests outlined in ingest doc in docushare
teragrid machines are running Linux, slac tests will use Sun Solaris
- not an issue, in the worst case will get worse performance on Linux
Need to measure pure disk i/o
- Sergei used iozone and bone++
- slac/jhu planning to use simple test program written at slac
docushare
=========
Will eventually need to migrate from plone to docushare.
First steps done:
- EA bug list in docushare
- ingest tests in docushare
- how-to import SQL to EA in docushare
- need to add UW schema/proto pipeline/db links to docushare
- what about permissions? who has write access?
- controlled per file or directory bases, simply give access
to everybody so that others can change ingest docs and such
provenance (doc from group in Britain sent to mailing list by Ray)
===========
"managing actor receives notifications" - do we need
something similar?
- there will be some kind of processing that we will run
regularly, perhaps based on some conditions that is reached,
eg 2 or 3 passes on trigger then do co-add. Perhaps this
will happen by monitoring provenance
"acknowledgment (msg received and recorded info from
provenance recording service)"
- we can do it through "monitoring event": each component
can send such events to inform others what it is doing
and what it accomplished
"overwriting, force deletion"
- yes, we will need that
coding standards
================
- yes, we should write it down and adopt it ASAP
- foreign keys: do after creating tables through "alter table"
-that is what EA is doing. Useful because then don't need to
worry about order of tables (SQL Server is sensitive to that),
and easy to extract if we decide to enable foreign keys later,
after ingest.
AOB:
===
Will discuss file systems and replication at DataAccWG this Friday
HTM/geometry algorithms question
- have htm id in current precursor schema, borrowed from sloan
- db evolved, now geom algorithms/indices much better
- e.g. SQL Server's yukon have add-on geom types,
mysql have openGIS based geometry classes and r-tree indices,
and are re-working/improving it
- need to be tested what works best, not urgent
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
Reminder: we have "an extra" Database telecon
tomorrow (Wednesday) at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- schemas (merging, EA, generic schema?)
- testing db ingest
- migrating from plone to docushare(?)
- nvo provenance doc
- coding standards
Jacek
p.s. I know we will run out of time,
let's try to cover as much as possible...
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
