Jacek, et al,
I'm back at Google Pittsburgh, and have been talking with them about
ways that Google can help LSST. Now that Google is an institutional
member, the possibilities seem to be opening up. One technology that
came up is "bigtable", which sounds interesting to me in the context of
our issues with updating the object table during the night. Take a
look at http://labs.google.com/papers/bigtable-osdi06.pdf and see what
you think.
They would like from us a test problem that would give them a concrete
example of what we need to do with the object table updates, and I told
them we would come up with something at our meeting next week.
Tim
Jacek Becla wrote:
Keywords: DataAccWG
Attendees:
Tim Axelrod
Maria Nieto-Santisteban
Ani Thakar
Ray Plante
Sergei Nikolaev
David Fleming
Kem Cook
Jacek Becla
Agenda for Database Schema meeting
==================================
- proposed agenda ok
- "schema changes" (documented in Docushare: Document-2407)
will be covered by Jacek
- 1/2 hour for each presentation, then discussion
- on Oct 18 we will start ~1:30
- on Oct 19 we will stop no later than ~5:00 pm
- the meeting will be held at LSST headquarters
precursor data for testing
==========================
- Most precursor data we have is in form of images
- will have pipeline that will covert to db by mid of next year
- Sergei's simulated data
- C code, simulates supernova, var stars, asteroids
- produces simple flat files
- both Source and Object tables
- good data set for near term tests
- will publish it in Docushare
Spatial query support
=====================
Use cases
---------
usecase 1: astronomers and general public
usecase 2: association pipeline running spatial queries
in real time for alert generation
Main question: if we observe the same object more than once
during the same night, do we want the alerting system to know
about the history of previous visits from current night?
- yes, there are usecases where it would be useful
- particularly interesting are "orphans": objects that
are observed for the very first time
- this drives the database design: technically it requires
re-indexing entire data set per image
- proposed solution:
a) don't reindex entire data set per image,
keep it read-only throughout the night
b) keep "new data" in separate, small catalog.
The size of such catalog would be ~1% of the large one
c) once every 24 hours (during the day):
- reindex the big catalog and add new data
- purge the small catalog
- likely we will visit the same field at least twice per night
(2 visits separated by 30 min, in the same band),
- so we will be hitting the small table fairly frequently
- also, notice that for each generated alert we will send
special package with data for that alert, it will include
database-type data. Amount of data fairly small.
Could keep data related to alerts in separate, small db
Any other use cases using spatial queries?
- yes: deep detection pipeline (run few times a year)
- large volumes of data, will be stressing database
- too early to discuss, let's come back in few months
(at All Hands meeting)
Queries
-------
what type of spatial queries (expressed in generic form)
accurately describe the spatial-query-related load?
- most common queries are circles
- circles are simpler to implement than rectangles
- very few cases where rectangles are truly needed,
- rectangles are often used because it is easiest
to formulate
- there are some usecases that need elliptical queries
(for example for elliptical or spiral galaxies)
- these are harder to implement, so we need to
understand how frequently they will be needed
- if infrequent, might e.g. use circular
and do post-processing outside of database
- if frequent, this will drive the spatial index design
--> need to understand expected distribution of
different types (circular, rectangle, elliptical)
- Ghaleb did some research in the past
when/where to continue this discussion?
=======================================
- no telecons next week,
- Wed: Tim unavailable
- Thu/Fri: Maria/Ani unavailable
--> make progress by email
--> continue at Tucson during ADASS
- Tim, Maria, Kem, Ray, Jacek, Jeff there
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a DataAccWG telecon tomorrow, Friday at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
The agenda:
1) DC1 ingest related (if there is anything new discuss)
2) precursor data for testing
-) database schema. Possible topics include:
a) review definition of Source and Object tables
b) postage stamp jpegs table
c) support for template images
d) "orbit catalog"
e) persisting policies and provenance/metadata
-) spatial query support
a) use cases (e.g. real-time for alert generation)
b) requirements (e.g. how often we re-index)
c) strategies/libraries to consider
e) documenting what we know
A relevant, highly recommended reading for
the last topic: Docushare Document-2459
I expect we will not manage to cover all of it.
We can use the Wednesday's database telecon to continue.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
