Keywords: DataAccWG
Attendees:
Tim Axelrod
Maria Nieto-Santisteban
Ani Thakar
Ray Plante
Sergei Nikolaev
David Fleming
Kem Cook
Jacek Becla
Agenda for Database Schema meeting
==================================
- proposed agenda ok
- "schema changes" (documented in Docushare: Document-2407)
will be covered by Jacek
- 1/2 hour for each presentation, then discussion
- on Oct 18 we will start ~1:30
- on Oct 19 we will stop no later than ~5:00 pm
- the meeting will be held at LSST headquarters
precursor data for testing
==========================
- Most precursor data we have is in form of images
- will have pipeline that will covert to db by mid of next year
- Sergei's simulated data
- C code, simulates supernova, var stars, asteroids
- produces simple flat files
- both Source and Object tables
- good data set for near term tests
- will publish it in Docushare
Spatial query support
=====================
Use cases
---------
usecase 1: astronomers and general public
usecase 2: association pipeline running spatial queries
in real time for alert generation
Main question: if we observe the same object more than once
during the same night, do we want the alerting system to know
about the history of previous visits from current night?
- yes, there are usecases where it would be useful
- particularly interesting are "orphans": objects that
are observed for the very first time
- this drives the database design: technically it requires
re-indexing entire data set per image
- proposed solution:
a) don't reindex entire data set per image,
keep it read-only throughout the night
b) keep "new data" in separate, small catalog.
The size of such catalog would be ~1% of the large one
c) once every 24 hours (during the day):
- reindex the big catalog and add new data
- purge the small catalog
- likely we will visit the same field at least twice per night
(2 visits separated by 30 min, in the same band),
- so we will be hitting the small table fairly frequently
- also, notice that for each generated alert we will send
special package with data for that alert, it will include
database-type data. Amount of data fairly small.
Could keep data related to alerts in separate, small db
Any other use cases using spatial queries?
- yes: deep detection pipeline (run few times a year)
- large volumes of data, will be stressing database
- too early to discuss, let's come back in few months
(at All Hands meeting)
Queries
-------
what type of spatial queries (expressed in generic form)
accurately describe the spatial-query-related load?
- most common queries are circles
- circles are simpler to implement than rectangles
- very few cases where rectangles are truly needed,
- rectangles are often used because it is easiest
to formulate
- there are some usecases that need elliptical queries
(for example for elliptical or spiral galaxies)
- these are harder to implement, so we need to
understand how frequently they will be needed
- if infrequent, might e.g. use circular
and do post-processing outside of database
- if frequent, this will drive the spatial index design
--> need to understand expected distribution of
different types (circular, rectangle, elliptical)
- Ghaleb did some research in the past
when/where to continue this discussion?
=======================================
- no telecons next week,
- Wed: Tim unavailable
- Thu/Fri: Maria/Ani unavailable
--> make progress by email
--> continue at Tucson during ADASS
- Tim, Maria, Kem, Ray, Jacek, Jeff there
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a DataAccWG telecon tomorrow, Friday at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
The agenda:
1) DC1 ingest related (if there is anything new discuss)
2) precursor data for testing
-) database schema. Possible topics include:
a) review definition of Source and Object tables
b) postage stamp jpegs table
c) support for template images
d) "orbit catalog"
e) persisting policies and provenance/metadata
-) spatial query support
a) use cases (e.g. real-time for alert generation)
b) requirements (e.g. how often we re-index)
c) strategies/libraries to consider
e) documenting what we know
A relevant, highly recommended reading for
the last topic: Docushare Document-2459
I expect we will not manage to cover all of it.
We can use the Wednesday's database telecon to continue.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
