Hi Ray, This is a great summary of the Policy concept. I've embedded a few comments.
Robyn On Mon, Mar 13, 2006 at 05:21:45PM -0600, Ray Plante wrote: > Keywords: MiddleWG, DataAccWG > Policy Objects > Ray Plante <snip> > 2. Our Starting Point > > Here are some facts/assumptions that can be gleaned from the UML model > (as of 3/9/06): > > 2.1. A Policy is a collection of parameters and rules that control the > behavior of a processing step. > > 2.2. A Policy is stored in a file within a Policy Library. > > 2.3. A Policy Rule has a condition and an action. > > 2.4. Policy parameters are collected into a named Parameter Set > within the Policy object. Likewise, Policy Rules are > collected into a Policy Rule Set within the Policy. It > is not documented what purpose these sets serve other than as > a container. > > 2.5. It must be possible to change policy data at least up until > the pipeline is executed. > > 2.6. The provenance service will record all parameters used by each > Processing Step and its constituent Processing Components. > > 2.7. During reprocessing, the PCS will use the provenance data to > create a Policy file for the pipeline that will recreate the > desired data products (see Create Re-processing Strategy use > case). > > The LSST WCS Prototype Pipeline produced a prototype Policy class > and file format based on keyword-value pairs. Some useful features > that would be good to incorporate in the model which are not there now > include: > > 2.8. Each Processing Component can have a Policy object/file > associated with it. > > 2.9. The file format allows parameter values to be "calculated". > (This could include the insertion of values of from other > parameters.) The allowable calculation does NOT include any reference to other parameters. It simply allows simple arithmetic ops on constant values. > > 2.10. Policy objects will need to be verified before use; at minimum, > it must ensure that the values are the proper type. > > Finally, the following use case that has been discussed in the > Middleware Working Group as being important: > > 2.11. A Processing Component should have the ability to listen for > events issued by other components up- or down-stream in a > pipeline and alter its behavior as a result. > > This last item can be considered as changing the value of a parameter > at run-time. This is very similar to the role of a Policy Rule. It > should be noted that such changes to a pipeline configuration should > only be allowed at controlled times (e.g. between processing Mosaic > frames) and for a well-defined scope in a pipeline. 2.11 needs to be restated. Having a component listen for events "up- or down-stream in a pipeline" is adding too much complexity to each individual component. Having the pipeline controlling manager listen for events and then moderating future actions of its components via their Policy statements, is more efficient. > > 3. The Policy Model > > The initial model outlined above is a good starting point and paints a > reasonably consistent picture so far. This section attempts to expand > the model to a level sufficient for the reference design. > > 3.1. Clarifications > > Some definitions are not currently spelled in the model, and so I > spell out a few of these here. In addition, I point out some other > modifications to the basic model and its semantics to support their > role in pipelines as described in section 3.2. > > 3.1.1. Parameter > > A Parameter is a piece of named data used to configure or control some > part of a pipeline. It consists of an identifier and a value. The > identifier semantically identifies the control being set and has > defined value type associated with it. The type of the value must > match the type associated with the identifier for the parameter to be > valid. A parameter value can be an expression that can evaluate to > the proper type and which may reference other named parameter values. Is the added complexity of allowing reference other named parameters needed? > > 3.1.2. Policy Rule > > A Policy Rule consists of an identifier, a condition and an action, such > that if the condition is evaluated to be true, the action should be > taken. There are two types of actions: an event type and a parameter > type. The event type indicates that an event of a specified type > should be raised with a specified message if the condition is true. > The parameter type includes a list of parameters that should be set if > the condition is true. I haven't figured out why the event type bothers me, but it does. Perhaps because I don't know how it could be used in a generic sense. How about a use case. > > Components that are interested in whether a rule applies are > responsible for evaluating the rule. Thus, the Policy Rule class will > include an operation called evaluate(). > <snip> > An important role of the Processing Step Policy will be to provide to > the Components pointers to input data (which may be in memory or on > disk) and pointers to where output data should go. Thus, an important > role of the Processing Step Policy is to direct the flow of data and > information between the components that make up a Processing Step. > In > general, this may well be a two-way conversation between the > Processing Step glue code and the Component itself; Explain. > this is discussed > more in section 3.2.3. <snip> > 3.2.2. Policies That Drive the Data Access Framework > > The current design for the Data Access Framework (DAF) attempts to use > context-specific intelligence for transfering data over distributed > machines to maximize both efficient transfer and efficient I/O by the > application. That is, to transfer data, one needs to provide three > pieces of information: what data one wants (via collection IDs), > where the they are wanted, and why the data is wanted. The "why" > might be to "reprocess nightly observations". Each "why" would map to > a configuration file that describes how to transfer the data in the > most efficient way for that particular purpose. > > For example, if nightly processing is done on a per-CCD basis, the > configuration file would be set up to copy each CCD data to a > different node on the target platform. So, common mpi-tools won't be used on a cluster to spray the CCDs to the next available nodes? Each node will be individually mapped? Is this because we're unsure that a cluster with enough nodes would be available to process an image and hence need the services of a globus resource manager? > The DAF could enlist multiple > transfer servers on the archive side to enable parallel transfers for > high throughput. > > This document proposes to model these configuration files as a > specific form of Policies. These would be "included" in Pipeline > Policies that handle the coordination of Processing Steps. > > 3.2.3. Policy Updating and Information Sharing > > As mentioned in 3.2.1, an important role of the Processing Step Policy > is to configure the flow of data and information between components in > a pipeline. For example, one Component will produce some data > products that are to be used as input into the next Component. Each > Component, in principle, has been developed independently, so it is up > to the glue code to direct that flow. > Furthermore, there may be > pieces of information calculated by a Component, e.g. the calculated > seeing in an image, that would useful to pass to other components. I > propose that the Policy could be the container for passing output > information back out to the Processing Step glue code from the > Component. Wouldn't DB's be suitable for this purpose? > > This could be enabled by defining in the Policy special output > Parameters. These would be Parameters that a component would update > as a result of execution. This could include a parameter that > contains the calculated seeing. More importantly would be the > Parameters that point to each of the output products created by the > Component. The job of the Processing Step glue code would be to copy > the output parameters it needs from one Component into the input > Parameters of another. > > I note that that in a multi-threaded streaming model for processing, > each call to a Component will need to have its own copy of these > volatile Parameters. This would be easier to do if Policy objects > could be easily decomposed into is hierarchical components. > > Finally, it is worth noting that a particular workflow system on > which our Processing Step framework will be based may provide a > mechanism already for passing information between components. In this > case, this role for Policy objects would not be needed. What workflow system is this? <snip> > > 3.3.3. Pipeline Execution > > If a Pipeline is being run for the first time on a collection > (e.g. the Calibration pipeline on new incoming raw data), then the > Pipeline will be configured by the PCS using the default Policy Files > in the Policy Library. If the Pipeline will be recreating or > otherwise being rerun on a previously processed collection, the PCS > system will use the Provenance database to create the Policy Files. > These two cases will be encapsulated within the PCS interfaces. In > either case, the PCS will need override various parts of the Policy > for the particular input collection and/or processing platform being > targeted. > > When the PCS is ready to deploy the Pipeline, it will copy all > necessary Processing Step and Component Policy Files to the nodes > where the Processing Steps will run. Then, when the Processing Step > is executed, the Processing Step framework code will read the > Policy Files into memory for use. Is this for parallelizing major constructs/groups (e.g. a night's worth of observation data) or for parallelizing intrinsic structures (e.g. all the CCDs' data for an observation). _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
