Keywords: DataChallenge1
Following is a summary of the Resource Consumer Pipeline being prepared for
DC1 use. The overall structure is implemented with the exception of invoking
mpi (which has been tested in isolation). Resource setup and consumption
for Flops and File I/O (both text and FITS files) are implemented.
Neither Database I/O nor end-of-test Cleanup are implemented.
The implementation will morph to use the LSST Middleware API as that interface
is developed.
I hope to acquire Jacek's database ingest test files/scenario in order to
rework it into the required characteristics for the DB Resource Consumer.
(OK, Jacek?)
--
Roberta (Robyn) Allsman
Large Synoptic Survey Telescope phone: 520-322-8741
Data Management FAX: 520-881-2627
4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED]
Tucson, AZ 85712 USA
============================================================================
Resource Pipeline Overview
============================================================================
The Resource Pipeline is a process which consumes resources based on a
preliminary characterization of a to-be-developed LSST pipeline process.
The characterization includes the files read/written, databases
created/updated/queried, mpi configuration, computer cycles spun during
analysis. File or database resources initially used in an 'input' role
by the to-be-developed LSST pipeline process will be created prior to
invoking the stand-in Consumer pipeline.
Resource consumption will be characterized by the following attributes:
file location: {gpfs, RAMfs}
file type: {text, fits}
file mode: {read, write}
file size: for text/binary: number of bytes; for fits: number of pixels
DB type: {Lsst collection, lsst image metadata}
DB mode: {create, ingest, update, query}
DB size: # attributes, # of rows
memuse: object size, number of objects,
flops: # of floating point operations on 'memuse' objects
mpi nodes: number of nodes to spray the stand-in Consumer
mpi barrier type: {none, completion}
mpi xfer: # of bytes to transfer amongst all nodes
============================================================================
Resource Pipeline Setup
============================================================================
Create and load any file or database resource initially used by the
stand-in Consumer as an input resource. Input is indicated by 'init'
in the resource usage policy defined below:
=====================================================================
Pipeline Characterization
=====================================================================
segment = [segmentPolicy,diskPolicy]
segmentPolicy = policy file defining pipeline processing segments
diskPolicy = policy file defining disk params
=====================================================================
Pipeline Processing: Disk Characterization
=====================================================================
ram = root pathname for all files created on RAMfs
gpfs = root pathname for all files created on gpfs
=====================================================================
Pipeline Processing Segment: MPI Characterization
=====================================================================
nodes = number of Nodes to be used for MPI processes
format = integer; default = 4
type = type of mpi barrier to simulate
type = {completion || none}
format = text string; default = none
xfercnt = number of FLOATs to send between all mpi processes
format = integer; default = 0
=====================================================================
Pipeline Processing Segment: Node Characterization
=====================================================================
PPS Node: File I/O Characterization
------------------------------------------------------------------------
file = [name,init,location,type,mode,size,loops]
name = root for local name for Consumer
init = {init,noinit}: init resource before sim run
location = {gpfs,ram}
type = {text,binary,fits}
mode = {read,write}
size = for 'text'&'binary': number of bytes to in/out-put;
for 'fits': number of bytes of pixel data
loops = number of times the file operation will occur
eg 2 -> activity processed twice during simulation
------------------------------------------------------------------------
PPS Node: FLOP Characterization
------------------------------------------------------------------------
memobj = [size, dim, ops,loops]
size = # of FLOATs per memory object
dim = # of memory objects per array
ops = # of MULT operations to perform on each MemObj
loops = # of times to repeat the ops
------------------------------------------------------------------------
PPS Node: DB Characterization
------------------------------------------------------------------------
db = [name,init,location,type,complexity,mode,cnt,aux]
name = root for local name for Consumer
init = {init,noinit}: init resource before Sim run
location = {gpfs,ram}
type = {collection(op: member-in-region),md(op: joins)}
mode = {create,ingest,query,update}
for mode=create:
cnt = total number of attributes
aux = not used
for mode=update:
cnt = number of attributes altered per row
aux = number of rows being altered
for mode=ingest:
cnt = total number of rows being added
aux = not used
for mode=query:
cnt = number of rows returned
aux = names of tables to be joined (comma separated)
============================================================================
Resource Pipeline Consumer
============================================================================
Consume the resources characterized in the Consumer resource usage policy.
if mpi, spray the specified Resource Pipeline process across nodes:
if file, perform specified disk I/O
read
write
if db, perform specified DB I/O
create:
ingest:
query:
update:
if memobj, perform specified computation
============================================================================
Software Requirements
============================================================================
Current software requirements are minimal:
python and standard distribution modules
mysql
additional python modules:
numarray
mysql
mpi
Installing the existing prototype pipeline infrastructure would provide
the complete environment required for the Consumer.
============================================================================
Command line interface
============================================================================
# initialize required pre-existing objects
python Consumer.py --pipeline=./pipeWcsConsumer.policy --op=setup
# consume the resource indicated in the pipeline policy
python Consumer.py --pipeline=./pipeWcsConsumer.policy --op=consume
# cleaup the residual resources
python Consumer.py --pipeline=./pipeWcsConsumer.policy --op=clean
============================================================================
Sample Policy Statements
============================================================================
The following policy statements have the current structure of the
existing WCS prototype pipeline. However, the resource consumption values
are place holders until the computational expense of the pipeline is properly
estimated.
pipeWcsConsumer.policy
------------------------
# Define the individual mpi segments within a Resource Consumer pipeline
segment = mpiAssembleImage.policy,diskRoot.policy
segment = mpiXtalk.policy,diskRoot.policy
segment = mpiPixelProcessing.policy,diskRoot.policy
#segment = mpiAssociation.policy,diskRoot.policy
-------------------------------------------------------------------------
diskRoot.policy
-----------------------
# Root pathnames on various devices for Consumer R/W operations
gpfs = /gpfs_sanscratch/rallsman/Consumer
ram = /dev/shm/Consumer
-------------------------------------------------------------------------
mpiAssembleImage.policy
-----------------------
# Define MPI organization
nodes = 4
type = completion
xfercnt = 100
# Simulate 'Assemble Image' processing on each node
# write Raw CCD Image to disk
file = FOne,noinit,gpfs,fits,write,100000,1
# copying CCD pixel data into memory
memobj = 20,1,2,1
# query the Mountain RawImageMetadata DB Table
db = DBOne,init,gpfs,md,query,1,0
# create Base RawImageMetadata DB Table
db = DBTwo,noinit,gpfs,md,create,100,0
# write metadata to Base RawImageMetadata DB Table
db = DBTwo,noinit,gpfs,md,ingest,1,0
-----------------------------------------------------------------------
mpiXtalk.policy
-----------------
# Define MPI organization
nodes = 4
type = completion
xfercnt = 100000
# Simulate 'Xtalk' processing on each node
# read Raw CCD Image from disk
file = FOne,init,gpfs,fits,read,100000,1
# read crosstalk matrix from disk
file = FTwo,init,gpfs,fits,read,100,5
# perform the crosstalk calculation
memobj = 20,1,2,1
# write xtalked Image to disk (in prep for next pipeline stage)
file = FThree,noinit,gpfs,fits,write,100000,1
# create Base CalImageMetadata DB Table
db = DBOne,init,gpfs,md,create,100,0
# write metadata to Base CalImageMetadata DB Table
db = DBOne,noinit,gpfs,md,ingest,1,0
# * * T H I S * * Policy File's Characterization
clientPolicyName = mpiXtalk
clientPolicyVersion = '1.0'
serverPolicyVersion = '1.0'
logFile = Consumer.log
diagnose = 2
#------------------------------------------------------
mpiPixelProcessing.policy
-------------------------
# Simulate 'Pixel Processing' on each node
# Define MPI organization
nodes = 4
type = completion
xfercnt = 10
# ------------------Calibration
# read Xtalk'ed Image from disk
file = FOne,init,gpfs,fits,read,100000,1
# read bad pixel image from disk
file = FTwo,init,ram,fits,read,10000,1
# read flat image from disk
file = FThree,init,gpfs,text,read,10,25
# read dark image from disk
file = FFour,init,gpfs,fits,read,100,5
# read bias image from disk
file = FFive,init,gpfs,fits,read,10000,1
# read fringe image from disk
file = FSix,init,gpfs,fits,read,10000,1
# perform the various calibration operations
# bad pixel
memobj = 20,1,2,1
# flat
memobj = 20,1,2,1
# dark
memobj = 20,1,2,1
# bias
memobj = 20,1,2,1
# fringe
memobj = 20,1,2,1
# write Calibrated Image to disk (in prep for next pipeline stage)
file = FSeven,noinit,gpfs,fits,write,100000,1
# ------------------WCS
# calculate crude WCS from in-core RawImageMetadata.Geometry
memobj = 20,1,2,1
# use sextractor to build the in-core image source collection
memobj = 20,1,2,1
# acquire the reference sources from the Base referenceSourceCatalog DB
db = DBOne,init,gpfs,md,query,50000,0
# calculate the 'adequateWCS' from: refSources, imgSources, telGeom, calImage
memobj = 20,1,2,1
# ------------------PSF
# acquire PSF reference source from Base referenceSourcecatalog DB
db = DBOne,noinit,gpfs,md,query,50000,0
# use WCSlib to calculate adequatePSF from: PSF refSources, imageSources,
# rawImageMetadata.wavefront, calImage, adequateWCS
memobj = 20,1,2,1
# ------------------PhotoZ
# acquire photometric reference source from Base referenceSourcecatalog DB
db = DBOne,noinit,gpfs,collection,query,50000,0
# use zero-point fitting algorithm to calculate photZ from in-core:
# imageSourceCatalog, photometric refSources, rawImageMetadata.?,
# adequateWCS
memobj = 20,1,2,1
# ------------------Image Subtraction
# read reference Image from disk
file = FEight,init,gpfs,fits,read,100000,1
# read CalImage from disk
file = FSeven,noinit,gpfs,fits,read,100000,1
# use Alard/Lupton image subtraction algorithm utilzing: refImage, calImage,
# refImage.adequatePsf, calImage.adequateWCS
memobj = 20,1,2,1
# write Subtracted Image to disk (in prep for next pipeline stage)
file = FNine,noinit,gpfs,fits,write,100000,1
# ------------------Source Detection
# read (calibrated or subtracted) image from disk
file = FNine,noinit,gpfs,fits,read,100000,1
# acquire 'interesting objects' from referenceSourceCatalog DB
db = DBOne,noinit,gpfs,collection,query,50000,0
# create sky image
memobj = 20,1,2,1
# write sky image to disk
file = FTen,noinit,gpfs,fits,write,100000,1
# create Base imageSourceCatalog DB
db = DBTwo,noinit,gpfs,collection,create,15,0
# ingest each source's photometric, astrometric & shape in imageSourceCatalog DB
db = DBTwo,noinit,gpfs,collection,ingest,50000,0
# ------------------Astrometric Calibration
# create refined WCS
memobj = 20,1,2,1
# Photometric Calibration
# create refined photo-Z
memobj = 20,1,2,1
# Save Metadata Permanently
# update flat/dark/bias/fringe/bad pixel to Base CalImageMetadata DB
# update refinedWCS to Base CalImageMetadata DB
# update refinedPSF matrix to Base CalImageMetadata DB
# update photoZ to CalImageMetadata DB -or- ImageSourceCatalog DB
db = DBThree,init,gpfs,md,update,20,1,0
# ------------------Object Detection
...TBD...
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
