Robyn,
Thanks for the feedback.
On Wed, 15 Mar 2006, Robyn Allsman wrote:
> 2.11 needs to be restated. Having a component listen for events
> "up- or down-stream in a pipeline" is adding too much complexity to each
> individual component. Having the pipeline controlling manager listen for
> events and then moderating future actions of its components via their
> Policy statements, is more efficient.
I like this better. Tim--since you brought this one up initially, does
this sound in line with what you were thinking?
> Is the added complexity of allowing reference other named parameters needed?
This is easy to implement (I did it for another configuration class) and
it can really be useful for keeping parameters in sync. Where it's
most useful (but which would not apply to us as much) is in specifying
directory paths. One can specify a parameter of a root path and have
other parameters referenced to that rather than repeating the same base.
Then when the base needs to be changed (either by a person or by the PCS),
then only one parameter needs to be updated.
Now we will be deemphasizing the use of directory paths in preference to
collection identifiers, which I would like to see as hierarchical paths.
So the same capability could apply, particularly if we leverage knowledge
about the structure of a collection ID.
This capability, nevertheless, is not restricted to paths.
> > A Policy Rule consists of an identifier, a condition and an action, such
> > that if the condition is evaluated to be true, the action should be
> > taken. There are two types of actions: an event type and a parameter
> > type. The event type indicates that an event of a specified type
> > should be raised with a specified message if the condition is true.
> > The parameter type includes a list of parameters that should be set if
> > the condition is true.
>
> I haven't figured out why the event type bothers me, but it does. Perhaps
> because I don't know how it could be used in a generic sense. How about a
> use case.
I didn't originate the idea of a Policy Rule--I'm just trying to make the
best sense of it given everything recorded about Policies. I agree that a
use case would be helpful here. I will inquire.
> > In
> > general, this may well be a two-way conversation between the
> > Processing Step glue code and the Component itself;
>
> Explain.
Perhaps my wording is misleading. See section 3.2.3 about "output"
parameters. A component tells the glue layer what its outputs are, and
the glue layer copies tells the next component it's inputs by copying
values from output parameters into input parameters.
>
> > this is discussed
> > more in section 3.2.3.
>
> <snip>
>
> > 3.2.2. Policies That Drive the Data Access Framework
> > For example, if nightly processing is done on a per-CCD basis, the
> > configuration file would be set up to copy each CCD data to a
> > different node on the target platform.
>
> So, common mpi-tools won't be used on a cluster to spray the CCDs to
> the next available nodes? Each node will be individually mapped? Is this
> because we're unsure that a cluster with enough nodes would be available
> to process an image and hence need the services of a globus resource manager?
MPI is one possible way to spray data but not necessisarily the only way.
Reasons why it may not be used in general:
o in the case described above, I'm talking about staging a pipeline's
input data to a platform well in advance of its execution. We could
write our stageCollection program as an MPI job; however, there are
grid tools that specialize in fast parallel data transfer.
o An MPI application more or less is predicated on the connected
processes running at the same time--i.e. their execution is tightly
coupled. A looser coupling between parallel activities--as in the
perfectly embarrasingly data-parallel case--can/may be more robust
and flexible way to manage a Pipeline.
The model for a Pipeline in general is being made up of Processing
Steps that related via a directed acyclical graph. It's easier to
recover from failure of one Processing Step do to their loose
couplings. Maybe that will change with advances in OpenMPI's fault
tolerance capabilities.
You alluded to passing data between components in a pipeline, and that's
where MPI could be useful. One example I've been looking at is a
Calibration Pipeline--which is largely parallizable on per-CCD level--that
starts with cross-talk correction which is likely to be done on a per-raft
level. Thus we could set up special cross-talk nodes that pass on their
results to CCD calibration nodes via MPI.
> > Furthermore, there may be
> > pieces of information calculated by a Component, e.g. the calculated
> > seeing in an image, that would useful to pass to other components. I
> > propose that the Policy could be the container for passing output
> > information back out to the Processing Step glue code from the
> > Component.
>
> Wouldn't DB's be suitable for this purpose?
Perhaps, and this implementation is not excluded.
At the moment, we're talking about having dedicated database servers
serving pipelines--i.e. different nodes than where the processing step is
running. How would caching this information in a remote db compare to a
simple mechanism to pass the information locally? Perhaps if we are
putting this info in a db anyway as part of provenance it make sense.
> > Finally, it is worth noting that a particular workflow system on
> > which our Processing Step framework will be based may provide a
> > mechanism already for passing information between components. In this
> > case, this role for Policy objects would not be needed.
>
> What workflow system is this?
We don't know because we have decided on such a system yet. However, as
an example, the Kepler system, I believe, has a built-in way of connecting
outputs to inputs. UIMA may have as well. If we used one of these
systems, then it would make sense to use the connection mechanism it comes
with.
> > When the PCS is ready to deploy the Pipeline, it will copy all
> > necessary Processing Step and Component Policy Files to the nodes
> > where the Processing Steps will run. Then, when the Processing Step
> > is executed, the Processing Step framework code will read the
> > Policy Files into memory for use.
>
> Is this for parallelizing major constructs/groups (e.g. a night's worth of
> observation data) or for parallelizing intrinsic structures (e.g. all the
> CCDs' data for an observation).
In general, this is the choice of the Pipeline author. However, in the
case of nightly processing which must be done in real-time, you clearly
want to parallelize on a smaller scale. That is (putting aside cross-talk
for the moment), you would have one Processing Step per CCD, each deployed
on a different node.
thanks,
Ray
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
