Hi,
first of all welcome to the team.
Now, some quick comments
> I have recently become involved with the LSST project as postdoc
> working for Stanford University. Returning from the ADASS 2006
> conference in Tucson I realized that one area where I may be able to
> make a contribution is the database design. Jacek encouraged me to
> post to this mailing list. The purpose of this posting is to find out
> which topics are well covered and where a contribution would be
> welcome. Maybe the ideas below will also trigger some discussion.
>
> One of the most impressive numbers relating to the LSST data base is
> the enormous expected size of the data base during the lifetime of the
> experiment. The sources table for example is expected to contain
> 8*10^12 rows. While storage of such a dataset will be a big challenge
> calling for a distributed GRID/SRB like architecture,
I doubt the Storage Resource Broker will be of much help for handeling
the Source table. SRB aims to manage collections that can be distributed
across multiple organizations and heterogeneous storage systems. Not quite
the case for the Source table. I hope!
> one of the main concerns will be the efficient querying of such a
> dataset. It will not be possible to query 8*10^12 rows within a
> reasonable time with a centralized indexing or sequential approach.
>
> More specifically:
>
> 1) Some questions that will be asked to a scientific dataset like ours
> cannot easily be expressed in terms of an SQL query, or at least it
> will be difficult to parallelize an arbitrary SQL query on a large
> cluster. A more procedural and parallelizable approach to
> filtering/processing may yield better results. This approach has been
> taken by the ADR project ("active data repository"), as well as by
> google's bigtable/sawzall projects, and it is also commonplace in
> other fields such as high energy physics where we have to deal with
> billions of recorded events. Should such an approach be evaluated for
> LSST?
So far, I think we all agree that a massive parallel approach will be
necessary. However, I cannot agree with the statement that SQL cannot make
it as I have already done things in the area
http://research.microsoft.com/research/pubs/view.aspx?tr_id=786
I am not as naive as to think that it will be easy to efficiently query
trillions of rows on a RDBMS using SQL, but I strongly believe that the
general query, the flexible query, the next user query will be easier to
support if we have a RDBMS behind.
> 2) Has work been done towards a distributed, hierarchical, or
> two-level approach to indexing? Which technologies are appropriate for
> LSST? I am especially thinking of existing work like the above
> mentioned ADR or its successors. The general idea is to not only
> distribute the data but also distribute the index and hence
> parallelize queries onto many distributed servers. One model could be
> to cluster data spatially, and have local indices on each data server
> that search only a small part of the total data volume. Results are
> then aggregated by a master and delivered to the client.
Yeap. This is what I'm working on. See paper above and
http://research.microsoft.com/research/pubs/view.aspx?msr_tr_id=MSR-TR-2005-169
> 3) I have found in various documents in the LSST document archive
> discussions about spatial R-Tree based indices and secondary B-Tree
> based indices, or mapping of spatial coordinates to a B-Tree
> (HTM). Looking at the typical expected queries on our dataset it is
> clear however that our dataset is inherently multidimensional.
> Indexing high dimensional datasets is difficult and many times
> indexing is slower than a sequential scan on the data. There are two
> solutions to this problem: a) make the sequential scan a relatively
> fast operation using a parallelized approach (see item 1), or
> b) use advanced multidimensional indexing like telescopic vector trees
> (TV-Trees), X-Trees (modified R-Trees), sh-Trees, etc. Best results
> would be obtained with a combination of a) and b).
Yes. We have a multidimensional dataset where multi correspond to a very
big number. It will be a challenge to implement the indexing system
that makes most popular queries efficient and every body happy.
> I apologize in advance if these items have already been discussed
> previously. I would appreciate if one of you can point me to the
> corresponding presentations, discussion board threads, etc. so that I
> can catch up!
I guess you already have looked at them but if not,
Documents 2572, 2573 and 2571 may be of your interest as well
As I said before
welcome to the team
Cheers
Maria
> Cheers,
> - Leif ([EMAIL PROTECTED])
>
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
>
--
------------------------------------------------
Maria A. Nieto-Santisteban ([EMAIL PROTECTED])
Johns Hopkins University
3400 N. Charles St.
Physics & Astronomy Department
Baltimore, MD 21218 (USA)
Tel: 1 410 516-7679 Fax: 1 410 516-5096
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
