I have recently become involved with the LSST project as postdoc
working for Stanford University. Returning from the ADASS 2006
conference in Tucson I realized that one area where I may be able to
make a contribution is the database design. Jacek encouraged me to
post to this mailing list. The purpose of this posting is to find out
which topics are well covered and where a contribution would be
welcome. Maybe the ideas below will also trigger some discussion.
One of the most impressive numbers relating to the LSST data base is
the enormous expected size of the data base during the lifetime of the
experiment. The sources table for example is expected to contain
8*10^12 rows. While storage of such a dataset will be a big challenge
calling for a distributed GRID/SRB like architecture, one of the main
concerns will be the efficient querying of such a dataset. It will not
be possible to query 8*10^12 rows within a reasonable time with a
centralized indexing or sequential approach.
More specifically:
1) Some questions that will be asked to a scientific dataset like ours
cannot easily be expressed in terms of an SQL query, or at least it
will be difficult to parallelize an arbitrary SQL query on a large
cluster. A more procedural and parallelizable approach to
filtering/processing may yield better results. This approach has been
taken by the ADR project ("active data repository"), as well as by
google's bigtable/sawzall projects, and it is also commonplace in
other fields such as high energy physics where we have to deal with
billions of recorded events. Should such an approach be evaluated for
LSST?
2) Has work been done towards a distributed, hierarchical, or
two-level approach to indexing? Which technologies are appropriate for
LSST? I am especially thinking of existing work like the above
mentioned ADR or its successors. The general idea is to not only
distribute the data but also distribute the index and hence
parallelize queries onto many distributed servers. One model could be
to cluster data spatially, and have local indices on each data server
that search only a small part of the total data volume. Results are
then aggregated by a master and delivered to the client.
3) I have found in various documents in the LSST document archive
discussions about spatial R-Tree based indices and secondary B-Tree
based indices, or mapping of spatial coordinates to a B-Tree
(HTM). Looking at the typical expected queries on our dataset it is
clear however that our dataset is inherently multidimensional.
Indexing high dimensional datasets is difficult and many times
indexing is slower than a sequential scan on the data. There are two
solutions to this problem: a) make the sequential scan a relatively
fast operation using a parallelized approach (see item 1), or
b) use advanced multidimensional indexing like telescopic vector trees
(TV-Trees), X-Trees (modified R-Trees), sh-Trees, etc. Best results
would be obtained with a combination of a) and b).
I apologize in advance if these items have already been discussed
previously. I would appreciate if one of you can point me to the
corresponding presentations, discussion board threads, etc. so that I
can catch up!
Cheers,
- Leif ([EMAIL PROTECTED])
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
