Jeff
That a very good point. It will also insulate pipelines from other
database-specific issues (for instance format of ascii file for
mysql is slightly different than format of the corresponding file
for sql server). So it means that
- the interface between pipelines and ingest changes,
it is no longer a preformatted ascii file
- we will need a db-specific "converters" that will logically
belong to Data Access Framework. I guess a converter
will need to run on the same machine where pipeline is executed.
I'll wait some time for comments, and if there are none,
I'll update the uml/ingest use cases
Thanks,
Jacek
Jeffrey P Kantor wrote:
Hello all,
One comment regarding ingest:
I think that most pipelines will want to handle their data as floats rather
than ASCII, while the ingest for MySQL apparently works best with ASCII. I
maintain that the pipelines should carry the data in floating point (as they
naturally want it) and any conversion to ASCII should occur on the Data
Access Framework/DBMS loader side of the interface. I think this also
insulates the pipelines from change in case we use another dbms that is more
efficient for floats.
Jeff
From: Ray Plante <[EMAIL PROTECTED]>
Reply-To: Ray Plante <[EMAIL PROTECTED]>, LSST Data Management
<[email protected]>
Date: Wed, 15 Mar 2006 14:46:06 -0600 (CST)
To: LSST Data Management <[email protected]>
Subject: [LSST-data] Minutes 2006-03-15 [MiddleWG] telecon
Middleware Working Group Minutes
2006-03-15 10am PT
Attendees: Russell, Jeff, Tim, Ray, Marcus, Jacek, Arun
Agenda:
o business
+ chair
+ the video option
o updates on high-level review of the model
+ database ingest
+ policy objects
+ ...
o identifying areas for further modeling
Ray requested we have ready access to the UML model via the terminal
server as well as to this "problem statement":
http://www.lsstmail.org/pipermail/lsst-data/2006-March/001007.html
1. Business
1.1. Chair
Due to other demands, Ghaleb asked if he could step down as chair of
this working group, and Ray has agreed to take over. He'll try to get
out agendas ahead of time so non-regular attendees can decide whether
to call in. Suggestions for input to the agenda will be taken at
anytime, including at the start of the meeting; however, ahead of time
is certainly more helpful to everyone.
Action: Ray: review Jeff's edits to the MiddleWG charter in
docushare.
Comments from others are welcome, too.
1.2. Connecting via polycom
Jeff is encouraging people to access telecons via polycom video when
convenient as a way to reduce costs. This requires that someone from
LSSTC connect the phone into the polycom so that those without polycom
still attend.
We determined that enough people in the WG to make it worth supporting
video. Jeff and Tim will be our LSSTC person responsible for patching
in the phone. If on a given week, neither are able to attend,
everyone will revert to the phone.
Action: Ray: arrange a reservation with Iain; send out connection
details to WG.
2. UML Modeling
Jeff introduced this discussion with some guidance on further modeling
through to the "skunk works" workshop. As mentioned in Ray's problem
statement, areas of focus can be an aid to the design of DC1. Thus,
the important areas to think about are:
o pipeline construction
o pipeline execution
o access to data by pipelines
-- staging input data
-- ingesting output data
We plan to push Use Cases from these areas down into Robustness models
using entities from the Domain Model. We currently have Domain
objects for Framework concepts (e.g. Image); however we are light on
interface objects, e.g. data ingest service. So we need to reflect
more these later type of nouns in the Domain model. These should be
stereotyped "interface".
Tim recommend that if we can pick and make progress on some very
specific operations--e.g. executing a pipeline that does
flat-fielding, we would accomplish a lot.
There was the question of how to approach modeling something this
specific (wherein our current use cases are very general). We
concluded that we create a robustness diagram for the very specific
use case. We might then later create a second version that is more
general.
2.1. Review of the state of the model at the high-level
Ray briefly reviewed the discussion on the Policy class, highlighting
two points:
o the proposal for using Policy objects for transfering
dynamic information between components may be replaced with
something more natural to the Processing Step framework we
choose to support glueing components together.
o Creating, tweaking Policy objects will be at the center of
"targeting" a pipeline to a data collection and a platform.
Jacek briefly reviewed the conclusions from the Data Ingest discussion
and doesn't see any major outstanding issues. Ray wants to also bring
up the related use case of "staging database data".
2.2. Modeling Plans
Marcus suggested and it was agreed that the skunk works attedees
should try to generate one robustness diagram prior to next week so
that we can get used to the mechanical and process issues of doing
so.
Action: Ray, Marcus, Jacek: post to the list today a brief statement
of the robustness diagram we will work on this week.
Action: Ray, Marcus, Jacek: begin work on a robustness diagram for
discussion next Wed.
Marcus will do something on targeting a pipeline for some part of
calibration. Jacek will do one on catalog data ingest. Ray will do
one on executing a configured pipeline. We will work these diagrams
inside the Middleware Use Cases folder.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
