DCers,
Attached for your review and comment is a write up of the development
needed for realizing the glue required by the new Pipeline model.
(Pardon the outline form.) You will notice that two items listed require
learning more details regarding the teragrid environment before I can
flesh them out to the same level of detail. Comments/alternate
suggestions are greatly apprecidated.
cheers,
Ray
Data Challenge 1
Plan for the Pipeline Glue Layer
Framework Constraints:
Processing Stage
* long-lived process
* processes streaming data
1. check input queue
2. if empty, sleep and repeat 1.
3. process data
4. place products on output queue
* messaging events for communicating with other stages.
* each Stage is a separate MPI application.
* implemented in python
Processing Slice:
* long-lived shared memory (in the form of ramfs)
* worker thread of the MPI application is permanently assigned a
CCD
* all stage applications will have identical CCD assignments.
No pipeline "glue" language is envisioned for DC1. Each stage will
run as independent processes communicating with each other via the
queues.
Needed components:
1. Input and output queues
2. Assignment of CCDs to slices, slices to nodes
3. Messaging mechanism for communicating the progress of pipeline
4. Mechanism for setting up and cleaning up ramfs on each node
5. A distributable mosaic format and associated library
Proposal
1. Queues:
* Each Stage will be assigned a directory to monitor as its input queue
and a directory to write output data.
* Queue directories may be in ramfs.
(do we need to allow more than one input/output queue for a stage;
e.g., one in ramfs, one in on real disk?)
* Each queue is a pair of directories in the same partition,
representing output from one stage connected to the input to another.
* Stage writes data to output queue directory; when done, application
give signal move to input queue for next stage.
* In move, datafile name is prepended with 2-digit number to
represent the order they arrive in. (Is this needed?)
* Next stage scans input queue by reading the file contents of directory
and sorting the names alpha-numerically. The first file
represents the next image in the queue.
* If directory is empty, the Stage sleeps.
2. Assignment of CCDs
* A Pipeline level policy file encodes mapping of one or more CCD
numbers to rank number.
* As part of the call that engages parallel processing, the CCD
numbers to work on are passed from the master node the
appropriate worker nodes.
* The details for managing multiple MPI jobs, determining the
mapping of rank to nodes will require some consultation with
Teragrid staff.
3. Messaging mechanism for reporting progress
* the messaging mechanism for communicating with the manager should
be as possible as it will be replaced by a real messaging
framework.
* stages can write a specially tagged message to stdout which are
continually captured. A monitoring pipeline manager process can
run monitoring all stdout streams.
4. Starting and stopping ramfs
5. Distributed mosaic format and associated library
* we re-use Dave Flemings FITS Image Collection library, a thin
wrapper around FITSIO.
* add support for mosaic level metadata.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
