Here is DC1 ... a work in progress.
M.
See attached work in progress.
Ray
Scalability Data Challenge 1. General Description
In this challenge, we plan to assemble a mock-up of the LSST Data
Management System (DMS) operating at three geographically distributed
sites which demonstrates how data from nightly observations will flow
and be processed from mountain top to archive center. We will use
three sites of the Teragrid network to represent the three main
components of the LSST "grid": mountaintop, base camp, and archive
center. The focus of this challenge is primarily on exercising the
throughput capabilities of the middleware design rather than on
performance or accuracy of application algorithms.
1.1. Specific Goals
1) Verify that we can operate an automated data handling and
pipeline processing system that is representative of LSST data
transfer and computing requirements at a scaled rate of
nominally 2% of what is expected during production operation.
2) Assemble evidence through this demonstration that our reference
design can be scaled up to meet the required operational data
rate.
3) Answer (at least in part) key questions regarding how we will scale
the system up in various areas, including:
o How do we deploy and utilize relational databases for use by
pipelines?
o What is the anticipated data ingest rate for detected sources
on a per source basis?
o How do we design application components so that they scale
well in a parallel mode?
o Will grid-based data transfer technologies scale via
parallelism to the anticipated volumes and rates?
o Where and how should error detection and recovery be
incorporated into the system? Can we automatically handle
most errors in an automated way without significant loss of
real-time results?
o Which technologies are most appropriate for inclusion in the
DMS reference design that handle:
* data replication and staging
* workflow management
4) Test robustness and algorithm performance of a few key stages in
the Calibration Pipeline include WCS generation.
5) Illustrate key aspects of the DMS UML model, including the
integration of application components into a middleware
framework.
1.2. Description
This challenge will be run on three sites of the Teragrid, each one
representing one of the three main sites of the LSST "grid". At the
mountaintop site, we will feed in our raw data. That data will be
transfered automatically to the base camp site where the prototype
pipeline will be run in real time. The raw data will simulataneously
be forwarded to the archive center where the same pipeline will be run
again, and the resulting data will be ingested into archive storage
and persistent databases.
The "raw data" that will drive processing will preferably be simulated
LSST data resulting from the efforts of the LSST simulation effort led
by Phil Pinto. If simulation data is not available or otherwise is
not appropriate for use in this challenge, we will use data from the
Precursor Archive (e.g. Subaru data). Prior to receiving the actual
simulated data used in the challenge, we expect to obtain or construct
stand-in fake data that will enable development of the DC1 system.
The processing pipelines will be a collection of application modules
wrapped in OGRE workflow "scripts". Initially, the application
modules will be in the form of "resource consumers"--CPU cycle burners
that behave similarly to real astronomical modules in terms of
interface, input and output data, and system load. As DC1 is
developed, we will replace some of these consumers with actual
astronomical implementations based either on existing astronomical
codes or new but simple implementations.
The application code that makes up the pipeline will focus on early
operations from the Calibration Pipeline that get run at both the base
camp and the archive center. They will not include modules related to
deep detection, classification, and photometric calibration. Nor
shall we consider the case of reprocessing raw data.
When executed, the challenge will follow this sequence:
1) The DC1 system will be deployed on Teragrid sites, including:
o the DC1 software stack will be deploy at each site
o the raw input data will be delivered to the mountaintop
site.
2) Execute data flow simulation
a. Pre-stage any needed "pre-existing" data to the processing
platforms required as part of the input data.
b. Start Pipeline processes at base camp and the archive center.
c. Start data flow
3) Clean up and analyze results.
Each of the above number items represent individual, human-initiated
actions. Within step 2, we expect that the sub-steps can be coupled
together and launched as single action in such a way to mimic the
expected behavior of the Data Management system when delivering and
processing new data from the telescope.
2. Infrastructure Components
2.1. Teragrid as Processing Platform
roles
NCSA, SDSC, & ??
use of storage, mass storage
use of network, networking tools
use of standard services (e.g. gridFTP)
Greg -- architecture
Chris -- performance tools, TG contact
2.2. NCSA Data Cluster as Archive Platform
roles
use of storage
Chris -- deployment
3. Middleware Components
3.1. Data Access Framework
3.1.1. Inter-site Replication: the Data Transport System
DTS vs. stageCollection.
Lead: Ramon, assistance from David
3.1.2. Data Staging: stageCollection
Lead: David, assistance from Steve
Todo: + finish stageCollection
3.1.3. Data Ingest
3.1.3.1. Database Ingest
Lead: Jacek, deployment support from Ramon
3.1.3.2. Image Ingest
Lead: Steve, assistance from Ramon
3.2. Pipeline Framework
Mostly repackaging for closer correspondence to UML model
3.2.1. Pipeline Control and Management (PCM)
Lead: Greg, assistance from Steve
Todo: Portlet for pipeline launching, Condor queue interaction
Pipeline Manager component
+ get staging policy, call stageCollection; triggered
as first action in DC1 "engage button" after
deployment.
+ tell PCS to target pipeline
3.2.2. Pipeline Execution
Todo: improved exception handling. (torpedos?)
Lead: Greg, assistance from Steve
3.2.3. Pipeline Construction System
Lead: Greg, Steve
Todo: PCSTarget: configure, deploy pipeline from Policy file
4. Application Components
Lead: Sagitarrius
4.1. Resource Consumers
4.2. Replacement Astronomical modules
4.3. OGRE wrapping
5. LSST Software Stack Deployment
Deploying pre-configured software onto Teragrid
Lead: Ramon
_______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
