Thought some of you would be interested to see what ASTRO-Wise is doing
for their databases. If you're not familiar with the project, it would
be worth checking out the various web-links supplied below.
Tim
--------------
Dear Tim,
good to hear from you.
We currently use Oracle 10g in a Oracle Real Application Cluster .
The pro's are that by connecting the db to our own developed compute
GRID we could achieve scalable processing and could fully integrate db
+ grid processing and storage in a single environment accessible by both
operators and end-users. (Unlike the classical GRID projects eg
GLOBUS). Since we can connect enough cpus, several 30-200 node
clusters, + storage to our system we are fully in businesses.
Choosing/evaluating the db engine is a major enterprise as you know ,
and I have to cut this short. After working with Objectivity we made a
deal with Oracle and are a reference programme. A major consideration
was that we want to be operational now.
A few quickies:
Oracle partitioning works fine and provides good (linear) db scalebility
(for catalogues we use an Oracle index on top of the htm index), but
SQL loader is in our experience a bit slow. An by now quit old report
from us can be found at
http://www.astro-wise.org/docs/reports/Oracle_Review.html
Oracle components like Spatial turned out not to be useful, in most
cases simple own Python scripts worked much faster.
Oracle Enterprise manager is archaic and we had to build our own GUIs etc.
So in the end we had to build a lot ourselves, but nevertheless our
teams are working happily. Using the Oracle facilities we could
relatively quick (1-2 manyear) build all the privilege and security
stuff, including things like scratch user db space, upgrade scratch to
persistent , upgrade private to public, upgrade to VO etc etc.
Mean issue with Oracle is that we want to operate several Oracle servers
over Europe, at our different National datacenters, in a p2p network.
This is simply because we want to share and distribute responsibilities
and not provide all the resources here in Groningen for the rest of
Europe. Also internet speeds is a consideration for some remote sites.
We want to keep the servers in sync, or transparent to each other, more
or less in real time. Neither Oracle Advance Replication nor Oracle
STREAMS can do this for user defined types (and this is how objects are
stored). We are working with Headquarters solving this. In the mean
time we have made our own workaround which has good prospects. This is
an important issue when you want to operate in a federation and we are
hitting walls of current technology here!
Next to this, we have some developing lines as an alternative to
Oracle, since they might become to expensive for small research
groups, or unnecessary complex. We are involved in exploring streaming
databases for Lofar, and in porting the algorithms we run now in
Python and C on the LINUX clusters to within the db kernel. We use
MONET/db for this development, which is build in the Netherlands, open
source, and build by our National center for CWI Amsterdam:
http://monetdb.cwi.nl
Ok thats it for now,
best regards
Edwin Valentijn
Tim Axelrod wrote:
Dear Edwin,
Thanks very much for following up. We remain interested in your
project as well. I do have an immediate question: It would be good
to know more about what sort of database technology you are using, and
what issues you expect to encounter as you scale up to the full
throughput of OmegaCAM. We see this as an area of considerable
uncertainty for LSST.
Best regards,
Tim
--
----------------------------------------------------------------------
Prof Dr Edwin A. Valentijn
Head OmegaCEN www : www.astro.rug.nl/omegacen
Kapteyn Astronomical Institute tel : +31 (0)50 3634011
P.O.Box 800 mobile: +31 (0)6 48276416
NL-9700 AV Groningen e-mail: [EMAIL PROTECTED]
The Netherlands www : www.astro.rug.nl/~valentyn
----------------------------------------------------------------------
begin:vcard
fn:Tim Axelrod
n:Axelrod;Tim
org:;Large Synoptic Survey Telescope
email;internet:[EMAIL PROTECTED]
title:Data Management Project Scientist
tel;work:520-322-8735
version:2.1
end:vcard
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
