Keywords: MiddleWG
Hello Jacek at al,
First, some comments on file I/O for DC1; second, embedded commments/questions
within Jacek's missive.
I too have been thinking about the file and database management issues for DC1.
I've been focussing on two areas:
1) integrating the current prototype WCS pipeline; and
2) developing a resource consumer for any pipeline stage for which we do
not have live code.
As a pipeline implementer, I assume a code can select where to place an
arbitrary file:
1) a node's local disk (which may (gpfs) or may not be visible to other nodes);
2) a node's RAM disk (not visible to other nodes); or
3) the LSST-managed global 'disk' accessible via the middleware library.
Ray said Middleware will provide a black-box interface to LSST storage
so DC1 pipeline code should assume all files are locally resident.
I assume this implies the pipeline code will call a Middleware library
routine to acquire a file which will appear some time later on the
specified local disk. Depending on when the acquisition is done, this
operation could be considered a pre-staging operation.
Writing a file to the LSST-managed global disk could be done via
(1) placing a file in a spooling directory; or
(2) output to a previouly opened I/O port managed by the Middleware.
Until Ray provides the middleware interface, I'm working on the assumption
that I'll bind a file I/O port to the LSST-managed global storage driver.
With respect to DB ingest files generated by the computational pipelines,
they could be placed in spooling directories handled by the DB servers or
they could be written to a previously opened file I/O port managed by a
middleware File-Forwarding-Service which would provide them to the DB Servers.
Ray--what are the plans for the middleware file I/O interface?
On Wed, Mar 08, 2006 at 04:02:08PM -0800, Jacek Becla wrote:
> Ray,
>
> I've been thinking more about Catalog Ingest for DC1 and had another
> look at the Ingest use cases in UML. Here are some thought/comments
> that might help us prepare for the skunk meeting.
>
> It is almost all about "how are we going to pass data from
> Pipeline to Data Ingest Service"?
>
>
> Database servers should run on dedicated machines,
> not together with pipelines
>
> Data will be passed in the form of ascii files
> - we need to decide exactly what the format is
> (mysql requirements will drive this)
> - these files will be created by pipeline,
> and deleted by ingest service
>
I guess ascii is pretty universal and avoids any endian issues.
>
> Data will be arriving in multiple streams
> - one stream per CCD?
The pixel processing pipeline works on one CCD at a time (on multiple nodes).
Pixel processing pipeline includes: badpixel/flat/dark/bias/fringe removal,
WCS and PSF calculation, image differencing, image detection. Tim said that
a part of the Association Pipeline will also work on one CCD at at time.
> - in "chunks", a chunk corresponds to a section of sky
> from a single image.
Yes, the CCD's map to a region of the sky so this is another 'natural' chunking
for an ingest file. However, this doesn't imply that the sources or objects
in the ingest file which are derived from the CCD's projection on the sky
will match any sort of segmenting being done on the source/object DB tables.
Which raises a question for the post-DC1 object DB: will it be segmented
based on regions (tiled pattern on sky) or zones (swathes in the sky)?
Will it be monolithic?
> - should we pass each "chunk" in a separate file (?)
To enable best MPI use, the unit used during computation should match the
unit forwarded for ingest.
>
>
> How do we "buffer" the incoming files on database server?
> - One directory per stream? One for all?
> - need to decide on file name convention
I've been sketching out a scheme for the MPI instance, I'll review it to see
if the naming scheme can be generalized for more global use.
> - we should probably allow for buffering many files for given stream
> (in case ingest falls behind for whatever reason)
Definitely.
No more comments on items below here.
Cheers, Robyn
>
> Once the file appears in the buffer, it can be grabbed by ingest,
> loaded to db, then deleted
> - do we need mutexes/locks around files?
>
>
> Do we want to preload data to memory before loading to db,
> or just read from disk?
> - if preload, then how?
> - ram disk? (be aware: Redhat has a limit for RAM disks: 0.5GB,
> we are working with them on this issue)
> - xrootd memory mapped files?
>
>
> If ingest determines during validation step that
> the input file is corrupt/data is formatted incorrectly,
> then it will send signal to pipeline. Then what?
> - should ingest stop?
> - should ingest ignore the file and continue?
>
>
> How serious should we be about recovering from errors, like
> - full-disk failure
> - server crash
> - machine crash
>
>
> I am not sure what the best forum to discuss these details is
> - we could try to make progress at Friday's DataAccWG telecon
> (if we have time)
> - or at the next MiddlewareWG telecon
> - or through emails
> - or simply wait until March 23/24
>
> Any preferences? =)
>
> thanks,
> Jacek
>
>
>
>
> Jeffrey P Kantor wrote:
> >Hi Ray,
> >
> >Just to document my comments from the telecon:
> >
> >This is a good target for the scope of the "skunk works" Middleware/APP
> >meeting. Much of this is modeled at the high level in the UML already, but
> >as you say it needs to be taken down to the next level of detail.
> >
> >Regarding PCS, I assume for DC1 that we will be using "static" pipelines,
> >so
> >instantiation will be minimal, mostly just "targeting". That said, it is
> >appropriate to cover more than that in the Middleware/App meeting. Thanks!
> >
> >Jeff
> >
> >
> >>From: Ray Plante <[EMAIL PROTECTED]>
> >>Reply-To: Ray Plante <[EMAIL PROTECTED]>, LSST Data Management
> >><[email protected]>
> >>Date: Fri, 3 Mar 2006 10:57:40 -0600 (CST)
> >>To: LSST Data Management <[email protected]>
> >>Subject: [LSST-data] problem statement for focusing further modeling
> >>[MiddleWG]
> >>
> >>Keywords: MiddleWG
> >>
> >>Hi MWGers,
> >>
> >>Sorry for the delay on this. Below is my strawman problem statement for
> >>focusing our further modeling activities. Please share your comments on
> >>the list so we can refine it. I recommend we have a call on Wed. to
> >>finalize this and try to list the specific UML use cases we need to
> >>focus on.
> >>
> >>thanks,
> >>Ray
> >>
> >>This statement describes a very broad use case that can elucidate
> >>important functionality and interfaces of the Data Management System.
> >>We will use it to identify the more specific use cases to develop to a
> >>deeper level. We also expect this to guide the design and
> >>implementation of our data challenges.
> >>
> >>This use case covers the transformation raw images into calibrated
> >>images and object detections on a nightly basis. It follows the
> >>dataflow from the point they are recorded onto disk on the Mountaintop
> >>down to the base camp where the Calibration and Object Detection
> >>pipelines are executed. We continue to follow the flow of raw data to
> >>the Archive Center where the same pipelines are run again, and the
> >>results are ingested into the archive.
> >>
> >>As we drill down and model in more detail, we will pay particular
> >>attention to the following activities:
> >>
> >>o operations of the Pipeline Management System
> >> * interfaces and events that trigger activity
> >>
> >>o operations of the Data Access Framework:
> >> * inter-site data replication
> >> * data staging for data-parallel processing supporting
> >> - CCD-indepedent processing
> >> - cross-talk correction
> >> * image ingest
> >> * catalog ingest
> >> * replication of data to base camp as inputs to future
> >> processing
> >>
> >>o operations of the Pipeline Construction System
> >> * instantiating/configuring the Calibration Pipeline
> >> * instantiating/configuring the Detection Pipeline
> >> * use of Policies in configuration
> >> * deployment of pipelines on processing nodes
> >> * determination of necessary input data
> >>
> >>We recognize that the fully modeled Data Management System will
> >>actually cover a lot more activities. For now, we will account for these
> >>at a high level in the model; however, in this exercise we will not
> >>develop these in as deep a level of detail. The functionality we will
> >>*not* focus on in detail include:
> >>
> >>o execution of the association pipeline
> >>o generation and distribution of alerts
> >>o non-daily pipelines, moving object pipelines
> >>o data publishing, public access to data
> >>
> >>
> >>
> >>_______________________________________________
> >>LSST-data mailing list
> >>[email protected]
> >>http://www.lsstmail.org/mailman/listinfo/lsst-data
> >>
> >
> >
> >_______________________________________________
> >LSST-data mailing list
> >[email protected]
> >http://www.lsstmail.org/mailman/listinfo/lsst-data
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
--
Roberta (Robyn) Allsman
Large Synoptic Survey Telescope phone: 520-322-8741
Data Management FAX: 520-881-2627
4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED]
Tucson, AZ 85712 USA
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
