DC1 Status report from R. Allsman 18 May 2006
Resource Consumer
Goal
Develop a code which mimics the resource consumption of a pipeline
including file I/O, DB I/O, memory utilzation, FLOP activity.
Provide a configurable method to define diverse pipeline analogies.
Non-Parallel Processing Resource Consumer Pipeline
Accepts policy statements which drive the consumer actions:
Pipeline.policy : defines stage policy files, disk policy file
Stage(i).policy : defines resource consumption of a stage-i for
file I/O, DB I/O, flops
Disk.Policy : filesystem locators
Performs one of three operations:
1) setup environment prior to run - builds all input files/DB
2) consume the resources as defined by pipeline policy
3) garbage collect output files
Status
Code is operational
Remaining to be done
Instrument the code to ensure the Resource Consumer accurately
consumes the resources that a known pipeline consumes, i.e.
the prototype WCS pipeline. RAA will work with TSA to define
metrics; RAA will implement.
Parallel Processing Resource Consumer
Using standalone resource consumer as a base,
consumer was updated to ensure file I/O does not clash with
other peers in stage.
Using the Pipeline Processing model below,
feature allowing sequencing of 'setup' operation over all stages
prior to 'consumption' operation within the **same** PBS job
was implemented.
Status
A non-PBS initiated, parallel processing version works.
However, integrating this code into the PBS Pipeline Processing
model described immediately below is not straightforward. The
next upgrade awaits completion of the PBS Pipeline implementation
before it can be completed.
Remaining to be done
DB I/O is not currently being handled. Since DC1 will use a
separate server for DB creation and update, this feature has
low priroty. Enabling queries from a separate network DB
server has not been implemented but should be.
RAA will implement.
See also: 'Remaining to be done' from above.
PBS Pipeline Processing
Goal
Initiate a pipeline job via PBS which automatically configures the
parallel run to efficiently use as many nodes as available. Each node
will have all stages running. All stages defined on a single node will
process the same set of CCD segments.
Communication Model
===================
Pipeline Initiator
Sets up PBS job submission defining individual mpiruns for each
stage. Uses policy statements to drive the initiator's actions:
Batch.Policy : defines PBS characteristics
Pipeline.policy : defines stage policy files, disk policy
file, Input/Output queues (per stage)
Stage(i).policy : defines resource consumption of stage
file I/O, DB I/O, flops
Disk.Policy : filesystem locators
Submits the PBS job. THe PBS job starts the Pipeline Manager as
an mpi process running on ONE node (provided by PBS).
Pipeline Manager
The Pipeline Manager orchestrates the start/stop of the pipeline.
The Pipeline Manager, in turn, spawns an mpi process on ONE node
(provided by PBS) running multiple Stage Leaders, one for EACH
pipeline stage.
Stage Leader
Each Stage Leader orchestrates the transition of data through
its nearest neighbor stages, i.e. stage(i-1) and stage(i+1).
Each Stage Leader spawns an mpi process running StageSlices
on EACH node provided by PBS.
StageSlice
Each StageSlice orchestrates the processing of a designated
number of CCDs; in the optimal case, there would be one CCD
designated to each StageSlice.
StageSlice uses policy statements to derive the stage's processing
requirements
Stage(i).policy : defines resource consumption of stage
file I/O, DB I/O, flops
Disk.Policy : filesystem locators
The root node for each mpi peer StageSlice defines the CCDs on
which each node will operate.
The resource consumer pipeline & stages are being used during
development of the Pipeline Processing scenario.
Status
Sequential nature of stage processing is being implemented in this
model.
Remaining to be done:
Orchestration of Mosaic Image processing via stage-sequenced
parallel processing of individual CCDs. The inter-stage communication
protocol needs to be implemented. The data movement will occur through
the I/O queues shared between adjacent stages.
This is currently being: designed by TSA, prototyped by RAA,
implemented by LSSTCorp team.
Logging of error and status messages by pipeline. Being designed and
prototyped by Michelle Miller, implemented by LSSTCorp team.
Installation of source code into the DC1 source repository.
Repository being setup by NCSA team, code will be installed by
the implementer.
WCS Prototype Pipeline
Goal
Provide a 'real' pipeline for DC1
Status
Non-parallel processing pipeline is functional.
Parallel processing version requires:
1) integration into the PBS Pipeline Processing model above.
RAA and TSA will integrate.
2) installation onto TG systems of the support environment (libraries,
tools, catalogs, etc).
Ramon is installing the software via PACMAN.
RAA will provide the catalogs which need distribution.
3) install and integrate FIC FITS handling routines into prototype
WCS CVS repository or DC1 CVS repository.
Fleming should install.
4) implementation of data delivery into the Image Input Queue for
pipeline processing. WCS DC1 Pipeline is assuming anything in
its Input queue should be processed. ???
5) test data for DC1. ???
--
Roberta (Robyn) Allsman
Large Synoptic Survey Telescope phone: 520-322-8741
Data Management FAX: 520-881-2627
4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED]
Tucson, AZ 85712 USA
--
Roberta (Robyn) Allsman
Large Synoptic Survey Telescope phone: 520-322-8741
Data Management FAX: 520-881-2627
4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED]
Tucson, AZ 85712 USA
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
