On Wed, 8 Mar 2006, Robyn Allsman wrote: > As a pipeline implementer, I assume a code can select where to place an > arbitrary file: > 1) a node's local disk (which may (gpfs) or may not be visible to other > nodes); > 2) a node's RAM disk (not visible to other nodes); or > 3) the LSST-managed global 'disk' accessible via the middleware library.
To the extent that these files will be read by other components, then the processing component itself probably should not be making this choice. (We might imagine that a component creates a ram disk, uses it to cache data internally, and then drops it; however, I think our main motivation for ram disks is as a convenient way to pass data between components and/or processes on the same node.) I think the best place to put the choice for where to write files is in the Policy object. The settings within the policy are then passed into a component, effectively telling it "write your data here". These Policies can contain the knowledge of our components are working together as part of a particular processing step/pipeline. Stepping away from DC1 for the moment and thinking about the reference design, I expect that components won't read there data direct from files nor write them out as this assumes to much about where the data is coming from or going. Rather, a component will be passed in-memory objects (e.g. an image object that might read its data from disk on-the-fly when necessary) and send out in-memory objects. For output, the component glue then might have to take responsibility for writing out the file. > Ray said Middleware will provide a black-box interface to LSST storage > so DC1 pipeline code should assume all files are locally resident. > I assume this implies the pipeline code will call a Middleware library > routine to acquire a file which will appear some time later on the > specified local disk. Depending on when the acquisition is done, this > operation could be considered a pre-staging operation. In general, pre-existing data will be staged to disk prior to the execution of the pipeline (perhaps hours or days before hand). Raw data will be delivered to "local disk" as it is available. Only in special cases or in defense against failures will data transfer be initiated within a pipeline's execution. This will necessitate a middleware I/O API. For DC1, however, let's just assume that the data will be local prior to a component's execution and we'll through an exception if this is not the case. > Ray--what are the plans for the middleware file I/O interface? Do my comments above elucidate the strategy for DC1? I'm working on the UML model right now to explain more for the reference design. To see more detail, have a look within the Middleware Use Cases, the Pipeline Execution package, and read the notes for Stage Input Data and Stage a Named Data Collection. cheers, Ray _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
