On Thu, Mar 09, 2006 at 12:49:07AM -0600, Ray Plante wrote: > On Wed, 8 Mar 2006, Robyn Allsman wrote: > > As a pipeline implementer, I assume a code can select where to place an > > arbitrary file: > > 1) a node's local disk (which may (gpfs) or may not be visible to other > > nodes); > > 2) a node's RAM disk (not visible to other nodes); or > > 3) the LSST-managed global 'disk' accessible via the middleware library. > > To the extent that these files will be read by other components, then the > processing component itself probably should not be making this choice. > (We might imagine that a component creates a ram disk, uses it to cache > data internally, and then drops it; however, I think our main motivation > for ram disks is as a convenient way to pass data between components > and/or processes on the same node.) > > I think the best place to put the choice for where to write files is in > the Policy object. The settings within the policy are then passed into a > component, effectively telling it "write your data here". These Policies > can contain the knowledge of our components are working together as part > of a particular processing step/pipeline.
Yep, and as an implementor, I write the Policy rules governing my pipeline stages. The flexibility to allow targetted file location enables better performance tuning. > > Stepping away from DC1 for the moment and thinking about the reference > design, I expect that components won't read there data direct from files > nor write them out as this assumes to much about where the data is coming > from or going. Rather, a component will be passed in-memory objects (e.g. > an image object that might read its data from disk on-the-fly when > necessary) and send out in-memory objects. For output, the component glue > then might have to take responsibility for writing out the file. Yes, this corresponds to my dataflow model for the DC1 pixel processing pipeline. > > > Ray said Middleware will provide a black-box interface to LSST storage > > so DC1 pipeline code should assume all files are locally resident. > > I assume this implies the pipeline code will call a Middleware library > > routine to acquire a file which will appear some time later on the > > specified local disk. Depending on when the acquisition is done, this > > operation could be considered a pre-staging operation. > > In general, pre-existing data will be staged to disk prior to the > execution of the pipeline (perhaps hours or days before hand). Raw > data will be delivered to "local disk" as it is available. Only in > special cases or in defense against failures will data transfer be > initiated within a pipeline's execution. This will necessitate a > middleware I/O API. > > For DC1, however, let's just assume that the data will be local prior to a > component's execution and we'll through an exception if this is not the > case. > > > Ray--what are the plans for the middleware file I/O interface? > > Do my comments above elucidate the strategy for DC1? Yes, thanks. > > I'm working on the UML model right now to explain more for the reference > design. To see more detail, have a look within the Middleware Use > Cases, the Pipeline Execution package, and read the notes for Stage Input > Data and Stage a Named Data Collection. > > cheers, > Ray Cheers, Robyn _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
