Jacek,
Do we really have to use ascii for ingest? This really ups the needed
I/O bandwidth. Is decimal arithmetic next? :-)
Tim
Tim
Jacek Becla wrote:
Ray,
I've been thinking more about Catalog Ingest for DC1 and had another
look at the Ingest use cases in UML. Here are some thought/comments
that might help us prepare for the skunk meeting.
It is almost all about "how are we going to pass data from
Pipeline to Data Ingest Service"?
Database servers should run on dedicated machines,
not together with pipelines
Data will be passed in the form of ascii files
- we need to decide exactly what the format is
(mysql requirements will drive this)
- these files will be created by pipeline,
and deleted by ingest service
Data will be arriving in multiple streams
- one stream per CCD?
- in "chunks", a chunk corresponds to a section of sky
from a single image.
- should we pass each "chunk" in a separate file (?)
How do we "buffer" the incoming files on database server?
- One directory per stream? One for all?
- need to decide on file name convention
- we should probably allow for buffering many files for given stream
(in case ingest falls behind for whatever reason)
Once the file appears in the buffer, it can be grabbed by ingest,
loaded to db, then deleted
- do we need mutexes/locks around files?
Do we want to preload data to memory before loading to db,
or just read from disk?
- if preload, then how?
- ram disk? (be aware: Redhat has a limit for RAM disks: 0.5GB,
we are working with them on this issue)
- xrootd memory mapped files?
If ingest determines during validation step that
the input file is corrupt/data is formatted incorrectly,
then it will send signal to pipeline. Then what?
- should ingest stop?
- should ingest ignore the file and continue?
How serious should we be about recovering from errors, like
- full-disk failure
- server crash
- machine crash
I am not sure what the best forum to discuss these details is
- we could try to make progress at Friday's DataAccWG telecon
(if we have time)
- or at the next MiddlewareWG telecon
- or through emails
- or simply wait until March 23/24
Any preferences? =)
thanks,
Jacek
Jeffrey P Kantor wrote:
Hi Ray,
Just to document my comments from the telecon:
This is a good target for the scope of the "skunk works" Middleware/APP
meeting. Much of this is modeled at the high level in the UML
already, but
as you say it needs to be taken down to the next level of detail.
Regarding PCS, I assume for DC1 that we will be using "static"
pipelines, so
instantiation will be minimal, mostly just "targeting". That said, it is
appropriate to cover more than that in the Middleware/App meeting.
Thanks!
Jeff
From: Ray Plante <[EMAIL PROTECTED]>
Reply-To: Ray Plante <[EMAIL PROTECTED]>, LSST Data Management
<[email protected]>
Date: Fri, 3 Mar 2006 10:57:40 -0600 (CST)
To: LSST Data Management <[email protected]>
Subject: [LSST-data] problem statement for focusing further modeling
[MiddleWG]
Keywords: MiddleWG
Hi MWGers,
Sorry for the delay on this. Below is my strawman problem statement for
focusing our further modeling activities. Please share your comments on
the list so we can refine it. I recommend we have a call on Wed. to
finalize this and try to list the specific UML use cases we need to
focus on.
thanks,
Ray
This statement describes a very broad use case that can elucidate
important functionality and interfaces of the Data Management System.
We will use it to identify the more specific use cases to develop to a
deeper level. We also expect this to guide the design and
implementation of our data challenges.
This use case covers the transformation raw images into calibrated
images and object detections on a nightly basis. It follows the
dataflow from the point they are recorded onto disk on the Mountaintop
down to the base camp where the Calibration and Object Detection
pipelines are executed. We continue to follow the flow of raw data to
the Archive Center where the same pipelines are run again, and the
results are ingested into the archive.
As we drill down and model in more detail, we will pay particular
attention to the following activities:
o operations of the Pipeline Management System
* interfaces and events that trigger activity
o operations of the Data Access Framework:
* inter-site data replication
* data staging for data-parallel processing supporting
- CCD-indepedent processing
- cross-talk correction
* image ingest
* catalog ingest
* replication of data to base camp as inputs to future
processing
o operations of the Pipeline Construction System
* instantiating/configuring the Calibration Pipeline
* instantiating/configuring the Detection Pipeline
* use of Policies in configuration
* deployment of pipelines on processing nodes
* determination of necessary input data
We recognize that the fully modeled Data Management System will
actually cover a lot more activities. For now, we will account for
these
at a high level in the model; however, in this exercise we will not
develop these in as deep a level of detail. The functionality we will
*not* focus on in detail include:
o execution of the association pipeline
o generation and distribution of alerts
o non-daily pipelines, moving object pipelines
o data publishing, public access to data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
