Hi Ray, I've integrated the pipeline MPI-harness with the resource consumer. Tomorrow, I should have the documentation describing how to setup and run the Consumer. Perhaps this skeleton could be added to the testing reservation next week.
To recap: The consumer is supposed to use the same physical resources as the proposed image processing pipeline. The resources were derived from the image processing pipeline UML description. The resources being used include file I/O and memory/flop usage. Resource consumption is gathered into atomic processing units: e.g. stage0 = AssembleImage, stage1=Xtalk, stage3= calibration, etc. Status: Tested on lsstarchive, but not on TG, are DB creation and access and FITS I/O handling. These features have not been tested on TG. Since DB ingest is handled external to the image processing pipeline (other than generating data input), the image processing Consumer will not use the DB hooks. The current version of the Consumer is not totally done: 1) only text files are currently being read/written; I need to check the fits handler (worked on lsstarchive but haven't tried on TG). 2) The new logger is not integrated. At the moment, lots of separate log files are created in order to track progress/status. 3) the 'real' consumer values for MemFlops, # of files read/written have not been installed. There are some issues we need to iron out: 1) I installed an input queue to simulate receiving an image from the mountain. This queue causes (stage0) processing to hang until data is available. However, I don't really use the queue in the Consumer other than for the hang. Notably, I don't currently remove the file from the queue. (Code is in but not tested.) 2) A HARD limit permits only 3 images to be processed. This limit allows the PBS job to complete successfully within the short time period needed for the debug batch queue. Note, however, a 'qdel' on the PBS job is very effective at terminating all live processes on the cluster so the limit can be taken out for the DC1 Phase 3 stress test. 3) All policy files for the harness and the consumer need to be loaded into a single directory. How this is arranged for production is for Greg or Steve to figure out. I will provide instructions to enable you to run short tests of the harness integrated consumer. -- Roberta (Robyn) Allsman Large Synoptic Survey Telescope phone: 520-322-8741 Data Management FAX: 520-881-2627 4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED] Tucson, AZ 85712 USA _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
