Keywords: MiddleWG, DataAccWG
Hi MWGers and DAWGers,
As promised, attached is a document discussing the Policy and related
classes (also in docushare as Document-1397). It contains all I've
gleaned from the UML Model and the LSST WCS Prototype Pipeline and expands
on it to clarify its interactions with 3 of our high-level components
(PCS, DAF, and PE). For example, I want to explore modeling data staging
strategies as Policy objects. Pending feedback from the working groups, I
plan to fold these ideas back into the UML model.
So the main purpose for sharing this document is to make sure my thinking
about Policies is consistant with others (particularly those that created
the concept and have been prototyping it).
We discussed Policies last week in the DataAccWG telecon, and one of the
concerns brought up was about adjusting parameters at run-time. In
particular, the MiddleWG talked about (at the Middleware-Apps
interface video-con meeting) having the ability for one Component to
subscribe to an event from another Component up- or down-stream in a
pipeline, and that the Component would as a result of receiving an event
change its behavior. This brought up some concerns about when the change
takes effect and how that would be captured in the Provenance such that
any data products could be reproduced. This document tries to collect
those issues and ramifications. So, do we really need this feature?
If so, did I capture the right approach for handling this?
Your comments would be appreciated.
cheers,
Ray
Policy Objects
Ray Plante
Contents:
1. Introduction
2. Our Starting Point
3. The Policy Model
3.1. Clarifications
3.2. Use of Policies within Pipelines
3.3. A Scenario the Use of Policies
1. Introduction
I have been digging deeper into questions of how input paramemters--
especially input datasets--get passed into processing steps. This is
an issue that affects three of our high-level components: Pipeline
Construction System (PCS), Data Access Framework (DAF), and Pipeline
Execution (PE). It also touches on provenance management. Our
current model defines a Policy class which can provide input
parameters, and this class has been prototyped in the LSST WCS
Prototype Pipeline. However, in order to address in further detail
how the PCS targets pipelines for a particular collection of data and
platform, how the DAF stages data, and how PE manages processing
steps, further detail regarding the use of Policy objects is needed.
The purpose of this document is to summarize the current understanding
of Policy files and to propose extending it to address its
interactions within our high-level components. By collecting this
information in a single document, we have a reference for discussion
as the new ideas are integrated into the UML model.
2. Our Starting Point
Here are some facts/assumptions that can be gleaned from the UML model
(as of 3/9/06):
2.1. A Policy is a collection of parameters and rules that control the
behavior of a processing step.
2.2. A Policy is stored in a file within a Policy Library.
2.3. A Policy Rule has a condition and an action.
2.4. Policy parameters are collected into a named Parameter Set
within the Policy object. Likewise, Policy Rules are
collected into a Policy Rule Set within the Policy. It
is not documented what purpose these sets serve other than as
a container.
2.5. It must be possible to change policy data at least up until
the pipeline is executed.
2.6. The provenance service will record all parameters used by each
Processing Step and its constituent Processing Components.
2.7. During reprocessing, the PCS will use the provenance data to
create a Policy file for the pipeline that will recreate the
desired data products (see Create Re-processing Strategy use
case).
The LSST WCS Prototype Pipeline produced a prototype Policy class
and file format based on keyword-value pairs. Some useful features
that would be good to incorporate in the model which are not there now
include:
2.8. Each Processing Component can have a Policy object/file
associated with it.
2.9. The file format allows parameter values to be "calculated".
(This could include the insertion of values of from other
parameters.)
2.10. Policy objects will need to be verified before use; at minimum,
it must ensure that the values are the proper type.
Finally, the following use case that has been discussed in the
Middleware Working Group as being important:
2.11. A Processing Component should have the ability to listen for
events issued by other components up- or down-stream in a
pipeline and alter its behavior as a result.
This last item can be considered as changing the value of a parameter
at run-time. This is very similar to the role of a Policy Rule. It
should be noted that such changes to a pipeline configuration should
only be allowed at controlled times (e.g. between processing Mosaic
frames) and for a well-defined scope in a pipeline.
3. The Policy Model
The initial model outlined above is a good starting point and paints a
reasonably consistent picture so far. This section attempts to expand
the model to a level sufficient for the reference design.
3.1. Clarifications
Some definitions are not currently spelled in the model, and so I
spell out a few of these here. In addition, I point out some other
modifications to the basic model and its semantics to support their
role in pipelines as described in section 3.2.
3.1.1. Parameter
A Parameter is a piece of named data used to configure or control some
part of a pipeline. It consists of an identifier and a value. The
identifier semantically identifies the control being set and has
defined value type associated with it. The type of the value must
match the type associated with the identifier for the parameter to be
valid. A parameter value can be an expression that can evaluate to
the proper type and which may reference other named parameter values.
3.1.2. Policy Rule
A Policy Rule consists of an identifier, a condition and an action, such
that if the condition is evaluated to be true, the action should be
taken. There are two types of actions: an event type and a parameter
type. The event type indicates that an event of a specified type
should be raised with a specified message if the condition is true.
The parameter type includes a list of parameters that should be set if
the condition is true.
Components that are interested in whether a rule applies are
responsible for evaluating the rule. Thus, the Policy Rule class will
include an operation called evaluate().
3.1.3. Drop Parameter Set and Policy Rule Set
If their role is nothing more than a container for more than one
Parameter and Policy Rule, then the Parameter Set and Policy Rule Set
should be dropped from the Domain model. Both sets feature a "name"
attribute, suggesting that they may have some semantic role for
partitioning policies. This is indeed needed: it will be useful to group
Policy data based on where (i.e. which component) they are needed. In
this case, it's not so useful to separate Parameters and Policy Rules into
different containers. See next section.
3.1.4. Policy
The basic definition remains the same; however, we define its contents
as having Parameters, Policy Rules, and/or other Policies, each
accessible via a keyword look-up operation, get(). Thus, policy data
are hierarchical and should be accessible and reference-able via a
hierarchical name. The levels of hierarchy should reflect the depth
within the pipeline that a policy datum is used (e.g. see section 3.3.1).
3.1.5. Policy File
A Policy File is the persistant stored version of a Policy. A Policy
File can capture hierarchical data either via hierarchical names or by
"including" other Policy Files.
3.2. Use of Policies within Pipelines
3.2.1. Functional Relationship between Policies and Pipeline Components
Recalling the model for Pipelines--that a Pipeline is made up of a
collection of Processing Steps (ordered via a directed acyclical
graph, or DAG), each composed of Processing Components--Policies can
be associated with all three levels.
In particular, any Processing Component can have a default Policy File
registered for it in the Policy Library. This Policy File would
typically be written by the author of the Processing Component. Much
of the values encoded won't depend on the context in which the
Component is used. An example from the WCS Prototype Pipeline is the
MosaicImage.conf's ccdGeom parameter; this is set by the physical
layout of the focal plane, and isn't expected to change over time.
The best value for other parameters may depend on the context the
component is being used--i.e. the processing step. For example, a
component that adds a stack of arrays, parameters that control that
addition or the creation of the varience image resulting from the
addition. In this case, the author of a Processing Step will need to
be able to override the default values for such parameters.
A Processing Step also will have a Policy associated with it as well.
This policy will include not only parameters that will be used within
the glue code that stitches components together but also will contain
parameters that override the default values given in the individual
Component Policy Files. This is done by first including the
Component's Policy File and then resetting the parameters.
An important role of the Processing Step Policy will be to provide to
the Components pointers to input data (which may be in memory or on
disk) and pointers to where output data should go. Thus, an important
role of the Processing Step Policy is to direct the flow of data and
information between the components that make up a Processing Step. In
general, this may well be a two-way conversation between the
Processing Step glue code and the Component itself; this is discussed
more in section 3.2.3.
It is also worth noting that some of the parameters that will control
a Processing Step will be specific to an instance of the Pipeline and
the data it operates on. This points to the role of the Pipeline
Construction System in configuring, or "targeting", a pipeline
instance.
Finally, we can consider Pipeline as having a Policy associated with
it, as well. These parameters would be used to coordinate the various
Processing Steps, and would likely have less to do with the scientific
functionality of the Pipeline as it does the deployment and execution
on a particular platform. For example, it may indicate the preferred
or maximum number of processing nodes to engage in executing a
particular Processing Step. It would also include the strategy for
deploying input data onto pipeline platforms as well as pulling
products back to the archive. This special Policy function is
described next.
3.2.2. Policies That Drive the Data Access Framework
The current design for the Data Access Framework (DAF) attempts to use
context-specific intelligence for transfering data over distributed
machines to maximize both efficient transfer and efficient I/O by the
application. That is, to transfer data, one needs to provide three
pieces of information: what data one wants (via collection IDs),
where the they are wanted, and why the data is wanted. The "why"
might be to "reprocess nightly observations". Each "why" would map to
a configuration file that describes how to transfer the data in the
most efficient way for that particular purpose.
For example, if nightly processing is done on a per-CCD basis, the
configuration file would be set up to copy each CCD data to a
different node on the target platform. The DAF could enlist multiple
transfer servers on the archive side to enable parallel transfers for
high throughput.
This document proposes to model these configuration files as a
specific form of Policies. These would be "included" in Pipeline
Policies that handle the coordination of Processing Steps.
3.2.3. Policy Updating and Information Sharing
As mentioned in 3.2.1, an important role of the Processing Step Policy
is to configure the flow of data and information between components in
a pipeline. For example, one Component will produce some data
products that are to be used as input into the next Component. Each
Component, in principle, has been developed independently, so it is up
to the glue code to direct that flow. Furthermore, there may be
pieces of information calculated by a Component, e.g. the calculated
seeing in an image, that would useful to pass to other components. I
propose that the Policy could be the container for passing output
information back out to the Processing Step glue code from the
Component.
This could be enabled by defining in the Policy special output
Parameters. These would be Parameters that a component would update
as a result of execution. This could include a parameter that
contains the calculated seeing. More importantly would be the
Parameters that point to each of the output products created by the
Component. The job of the Processing Step glue code would be to copy
the output parameters it needs from one Component into the input
Parameters of another.
I note that that in a multi-threaded streaming model for processing,
each call to a Component will need to have its own copy of these
volatile Parameters. This would be easier to do if Policy objects
could be easily decomposed into is hierarchical components.
Finally, it is worth noting that a particular workflow system on
which our Processing Step framework will be based may provide a
mechanism already for passing information between components. In this
case, this role for Policy objects would not be needed.
3.3. A Scenario the Use of Policies.
3.3.1. Pipeline Definition
Pipeline definition will involve several different developers playing
different roles.
An author of a Processing Component will, as part of the component
development, create a Policy File that contains the default
configuration for the Component. The Policy File syntax will support
mark-up for fully documenting each parameter and rule. The
Component's interface will include a constructor that allows a Policy
object to be passed in.
Some Components may be composed out of other components. Such an
aggregating Component would also have a Policy File written for it.
It would include the default Policy Files of the constituant
Components and then override the Parameters they define as needed. In
addition, the aggregating Component would have its own set of
Parameters defined. The hierarchical naming convention would
distinguish the Parameters for the different components. For example,
suppose a "Co-add" module calls a "Stats" module; the Co-add Policy
File might refer to a Stats Parameter as "stats.threshold". Within
the aggregating Component's implementation, it would "slice off" the
part of the Policy that corresponds to the constituent's parameters
when that constituent is constructed.
If the information flow mechanism described in 3.2.3 is required, the
Component's interface will also need to allow a Policy object for
input and output parameters that are needed for that particular call
to the Component. The passing of Component output Parameters to input
Parameters for other Components is written into the Processing Step
glue code.
When the Component development and testing is finished, the
Component's Policy File is given a version number and loaded into the
Policy Library.
The job of the Pipeline author is to stitch together Components into
Processing Steps which are then organized into a logical Pipeline.
For each Processing Step defined, the author will create a Policy File
that first "includes" its components Policy Files and then overrides
specific Parameters and rules. The Processing Step glue code will be
responsible for passing the right parts of the Policy to each
Component (via constructors and/or method calls); this will be
straight-forward to do using hierachical naming and the ability to
"slice off" Policy data.
Finally, a Pipeline Policy File is created to coordinate all the
Processing Steps. This includes a Policy File that the Data Access
Framework will use to stage input data to the processing platform and
to pull products back to the archive. (The details for data staging
may depend on whether the Pipeline will be reprocessing data or being
run on new data from the telescope.) If transfers are needed to move
data between Processing Steps, those are configured as well. These
high level Policy Files are versioned and loaded into the Policy
Library. As part of this loading, the Policy Files should be
validated at least for adherence to typing rules.
3.3.2. Data Deployment
In advance of Pipeline's execution, the PMC will load the Pipeline's
Policy in order to stage the (existing) input data.
3.3.3. Pipeline Execution
If a Pipeline is being run for the first time on a collection
(e.g. the Calibration pipeline on new incoming raw data), then the
Pipeline will be configured by the PCS using the default Policy Files
in the Policy Library. If the Pipeline will be recreating or
otherwise being rerun on a previously processed collection, the PCS
system will use the Provenance database to create the Policy Files.
These two cases will be encapsulated within the PCS interfaces. In
either case, the PCS will need override various parts of the Policy
for the particular input collection and/or processing platform being
targeted.
When the PCS is ready to deploy the Pipeline, it will copy all
necessary Processing Step and Component Policy Files to the nodes
where the Processing Steps will run. Then, when the Processing Step
is executed, the Processing Step framework code will read the
Policy Files into memory for use.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
