Keywords: DataAccWG
Present: Maria, Ani, Tim, Ray, Arun, David
Executive summary:
We discussed how to organize spatial and temporal data in the
object database schema so that the base camp database can quickly get
the data it needs to generate alerts. Tim suggested the object db be
partitioned temporally to ensure fast querying. However this may not be
a big issue, i.e. the spatial partitioning will largely take care of
good query performance, even for temporal data.
Action Item
Sergei or Kem to come up with query/use-case for base camp request
for info from object db required for alert generation
Notes:
David said he will send status by Monday
Spatial & Temporal data
------------------------
What we need from object db
- at base camp (for alerts), need info from object db that is current
in time (best estimate of ra,dec etc.)
- shd be partitioned based on time axis
- need time history of the object
- need only the latest detailed information along with the summary of
the time history
How this info gets into object db
- part of ingest/publication process is generating this
- as new object comes in, match it with previous observations
- spatial match basically
- association process (pipeline) takes care of this
A query should return this information, hopefully
- a spatial query (?)
- object db will have latest info on some area of sky
- only need this info when that area of sky is re-observed
- Tim thinks this query might take too long
- but it only needs to query for last 30 days
- can we organize the data to optimize for this (partition in time)?
- does this mean partitioning in both space and time?
- each object will already have time attribute, just index on it
- time-based query from base camp will be much less frequent (once
per day) than spatial queries from users, so need to optimize
for latter
We may already have the data partitioned the way we need
- because of our modular schema
- but data needs to be partitionied spatially to resolve multiple
obervations of the same object
Important thing is not what is the latest observation, but the latest
*analyzed* observation
- latest list of detections gets fed to association pipeline
- comes out as list of associations with known objects
- this will be most relevant for base camp db (alerts)
- this list will always be about the same (size?)
We need to distinguish between summary info (time history) and detailed
info
- there are two distinct processes
- one is assoc. pipeline that matches latest with history of
observations
- second process is deciding if any of these objects has behavior
that warrants generating an alert
- second part requires both recent and summary info
Does time partitioning mean that data from different time periods is in
different tables?
- it may turn out that way
Does the UML model think of alert generation as part of association
pipeline?
- we didnt have the answer to that
Can Sergei or Kem come up with query that requests the information
needed for alert generation (i.e the time history and detailed recent
information)?
- since they were not present, this action item was assigned to
them ;-)
- should be quite easy to write such a query
- is there already one in the use cases? - probably not
Query efficienncy will depend on how schema is organized
- small amount of data needed per night at base camp db
- indicates usefulness of prototype schemas
- do either of the current prototype schemas (UW and sergei's)
have time partitioning in mind?
- we shd start posting these schemas
- is the precursor schema in docushare? dont know, tho' it is
in the plone
Conclusion:
- since the data in the object db will be spatially partioned, and
each night's data will be a certain part of the sky, the query
from the base camp db will in fact be limited to a small number of
partitions
- it may not be necessary to further partition based on time because
the schema can be organized in such a way as to make the query
run quite fast on the spatial partition(s) involved
- there will be a object history table which will record the history
of observations for each object (like the current SDSS match
table)
- this will be the time history summary
- finding the recent detailed info for a given object within a
partition should be very quick (dbs are very good at this)
Other agenda items:
- how will LSST handle multiple targetings for spectroscopy?
- not an issue, since LSST doesn't have spectroscopy ;-)
- questions on ingest test results on mysql
- sergei and jacek not present
That's it.
Ani
On Fri, 24 Feb 2006, Ani Thakar wrote:
>
> Keywords: DataAccWG
>
> Hi All,
>
> Sorry for the short notice, but if we have a quorum, we will have a
> DataAccWG
> telecon today at 11:00 PST.
>
> Phone number: 866 330 1200
> passcode: 300 2363
>
> Agenda to include:
> Spatial and Temporal Aspects (Jacek suggested this)
> How do we propose to handle multiple targetings for spectroscopy?
> Ingest tests with MySql (I know Jaeck isn't here, but had some
> questions for Sergei)
>
> Thanks.
>
> Ani
>
>
--
Aniruddha R. Thakar, Research Scientist
Center for Astrophysical Sciences, JHU, Bloomberg 375
3701 San Martin Drive, Baltimore MD 21218-2695
410-516-4850, Fax: 410-516-5096
[EMAIL PROTECTED], http://www.sdss.jhu.edu/~thakar
-----------------------------------------------------------------------
Ask not for whom the bell tolls, and you will pay only the
station-to-station rate.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
