Keywords: MiddleWG
Attendees: Greg Daues, Jeff Kantor, Marcus Miller, Michelle Miller,
Russell Owen, Ray Plante
NDDS workshop
=============
Michelle gave a preview of a report she plans to post describing what
she learned at the NDDS system which we are considering as a
technology that deliver our event related use cases. NDDS is a
product that is used particularly by defense contractors building
embedded systems, so robustness and lightness is important. It
provides a fine-grained approach to publish-subscribe data
communications on par with CORBA (but more light-weight). In general
consumers subscribe to data objects and don't necessarily care where
they come from. NDDS appears to provide some fine-grained robustness
and fail-over features. The instructor appeared to bend general CS
parlence a bit, so it was hard to understand some of the differences
from similar products.
Middleware Technology Choices
=============================
Jeff had begun and posted a document setting down some specific
technology choices for the reference design. He reviewed that
document and discussed its role in the modeling/costing effort. He
highlighted three critical TBD choices still remaining: Pipeline
Glue, Event Messaging, and Pipeline Construction. We stepped through
each of these, and made some initial choices.
Event/Messaging
---------------
Broadbrush requirements:
o reliable synchronous and asynchronous messages
o publish/subscribe model that can be made fairly specific
o ability to minimize event traffic between sites to just what is
needed.
o persist-able event queues
o ability to interface with multiple languages and other messaging
systems
* the system used within a pipeline may be different from
what is used by Management and Control.
* the system used by the OCS may be different by Data M&C.
* if there are advantages to do so, a common choice across
these regimes can be used. Eventually, an evaluation
will be done.
Ray proposed that we choose a Mule/JBoss combination for event
messaging at least for M&C. (Mule as the message broker, and JBoss to
provide the JMS implementation and queues.) It looks like a good
match, is currently being used by the NOAO Science Archive
development, and appears to be highly amenable to plugging in
alternate solutions and connecting with heterogenous components.
Ray/Steve were given the action to write up and post these points in a
brief statement.
Ray was given the action to add some basic event use cases to the UML
model.
Pipeline Glue
-------------
Greg brought up the Cactus framework because it handles both
componentization and MPI for the user. Eventually Jeff vetoed it
because because:
- it is not CCA compliant
- it does not handle python (only C, Fortran, etc.)
Jeff stated these language requirements for application layer code:
- new code primarily written in C/C++, python, and possibly Java.
- use of existing libraries written in, say, FORTRAN may require
wrapping in one of the above languages.
- Perl will not be used (Perl code must be rewritten.)
Marcus recommended CCA because it works with many frameworks and
languages and is a de-facto standard in the high performance
computing environment. Jeff accepted that recommendation. Marcus
noted that CCA does not "wrap" components, but rather that a
component is built to be CCA-compliant.
A question was raised whether anyone had tested interoperability of
CCA components in different frameworks. Marcus said that people where
he works do this all the time, but that he did not personally know
the "cost" of doing so (e.g. how much, if any, rewriting of the
components is required and whether there is any overhead involved).
Jeff tasked Marcus, Greg and Michelle with:
- Documenting why we are going to use CCA (for now)
- Picking a CCA-compatible framework
he wants a report by the end of the week. The report will be discussed
at next week's MW WG phone con.
Pipeline Construction
---------------------
(A choice of a CCA technology may enable the use of building tools,
such as an Eclipse plugin.)
Was anything more specific said about this TBD?
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
