FYI
------ Forwarded Message
From: Jacek Becla <[EMAIL PROTECTED]>
Organization: Stanford Linear Accelerator Center
Date: Fri, 19 May 2006 21:57:46 -0700
To: Tim Axelrod <[EMAIL PROTECTED]>, Ray Plante <[EMAIL PROTECTED]>,
Cristina Beldica <[EMAIL PROTECTED]>, Jeffrey Kantor <[EMAIL PROTECTED]>
Cc: Russell E Owen <[EMAIL PROTECTED]>
Subject: DC1 status update (database/ingest)
Hi,
Here is the DC1 status report related to database/ingest.
Proposed database architecture:
- 2 "ingest nodes"
- 2 "archive nodes" (nodes serving pre-cached archived data)
- don't use federated engine as originally planned,
use a combination of replication and data partitioning
to build a distributed system
Each ingest node will handle 1/2 of ingest streams.
Each archive node will serve 1/2 of the archived data and a copy of
all ingested data.
Data query will be done on archive nodes.
Ingested data will be transferred to archived nodes through replication:
both archive nodes are slaves for each ingest node, which means data
from both ingest nodes will be transferred to both archive nodes (in
real time).
Such setup has 4 major advantages over federated-engine approach:
- no collisions between ingest and readers
- we have up-to-date backup
- the ingest node tables do not need to be indexed,
so no need to worry about disable/enable indexes
- fast and reliable (as opposed to federated engine
which is not optimized at all yet, it is a new feature
released few months ago)
Archived data partitioned into many small tables,
data in each table clustered spatially. There should
be enough overlap to avoid having to query data from
more than one archive node.
Russell will implement Data Ingest Service (DIS)
in python, Jacek will implement Data Ingest
Converters (DIC) in C++.
We assume data from pipeline will be sent to DIS
through "queuing package". We need to agree on
exact format of data.
We are guessing that signals/events between DIS, DIC and the rest
of the middleware will be done through the queuing
package as well. The signals/events include:
- "start loading data",
- "stop loading data".
We are also expecting DIS will be "created", and
"destroy"...
Questions:
- Are we planning to record events in Event Database for DC1?
(we are assuming: no)
- are we going to get ingest node assignment from
policies for DC1? (we are assuming: no, they will
be assigned statically)
- are we going to get database schema from an official
place, e.g. through policies, or we should hardcode
it (by using a file with a fixed path) for DC1?
(we are assuming: hardcode)
- how should we exchange data between DIC and DIS?
We are assuming a shared file system
- how much archive data will we have to deal with for DC1?
(We are assuming <1TB)
- we need to have enough disk space on archive nodes
to host the archived data...
Summary of open issues:
- need to decide how to partition/divide archived data
so that any image maps naturally to a small number
of tables with archived data
- need to decide on format of data sent from pipeline to DIS
We are assuming file ingest is not our responsibility
and are leaving it for Ray and his group.
Jacek and Russell
------ End of Forwarded Message
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
