Hello all, OK, so according to Marcus it sounds like we are in a promising space with CCA frameworks. The combination of tools Marcus mentioned in the last paragraph sounds like an environment worth evaluating. I have also recently been having Robyn look at MPI 2.0/OpenMPI for pipeline glue as well, and it also looks promising. All of these evaluations make sense in the overall D&D phase.
But we still don't have a good picture of the reference design for the MREFC proposal estimate. I want to identify the gaps as of now. Looking at our current middleware baseline, if I understand things properly we have: DM System Control and Management - OGRE/Elf for facility-level control/workflow - DTS, Gridftp for data transfers between facilities ( I think we also need to look at Alex Szalay's contact at University of Chicago for a UDP-based protocol stack that facilitates high-speed data transfer over long-haul networks. This has been used on internet2 land speed records) - StageCollection/globus-url-copy for data staging within a facility - TBD Middleware for events, messaging, logging across facilities and integration with OCS (evaluating RTI NDDS, MULE, JMS, what else?) Pipeline Execution Middleware - Python Framework Classes (per the UML Model), encapsulating Application Layer calls to the following: - MPI for intra-stage science data communications (within a parallelized processing stage) - TBD middleware for inter-stage science data communications, aka "pipeline glue" (evaluating ZeroC ICE, OpenMPI, RTI NDDS, MULE, JMS, what else?) - TBD Middleware for events, messaging, logging by an executing pipeline (evaluating ZeroC ICE, RTI NDDS, MULE, JMS, what else?) - GridDB descendent for smart re-processing analysis Pipline Construction - Python, C++, Java for creating "raw" components - Eclipse with CCA plug-in, Babel/SIDL, SWIG for component wrapping/ integration with Pipeline Execution Middleware Data Access Framework - MySQL, Xrootd, C++ for DBMS with parallel ingest/query - DCI, SRB, Ibrix (or Lustre? Xrootd?) for image file storage, replication Securit and Administration - NSA/Public Domain Certificate Manager for Security - Other tools, services? User Interface Services - Ccaffeine/GUI for ??? - Mirage/Java for web-enabled scientific plots and graphs, in a multi-window environment - All manner of VO-compliant tools Am I mis-stating or missing anything? My biggest concern is validating this and filling in the TBDs. Is there a CCA framework that fills in TBDs and if so where does it fit? What evaluations can be done by May 20 to resolve TBDs in time for inclusion in the cost estimates/development schedules? Jeff > From: "W. Marcus Miller" <[EMAIL PROTECTED]> > Organization: LLNL > Reply-To: [EMAIL PROTECTED] > Date: Mon, 24 Apr 2006 17:21:11 -0700 > To: Jeffrey P Kantor <[EMAIL PROTECTED]> > Cc: Ray Plante <[EMAIL PROTECTED]>, Michelle Miller <[EMAIL PROTECTED]>, > Tim Axelrod <[EMAIL PROTECTED]>, don Dossa <[EMAIL PROTECTED]>, Ghaleb > Abdulla <[EMAIL PROTECTED]>, [EMAIL PROTECTED] > Subject: Re: UCF Paper > > Hi Jeff, > > It has taken a while to slog thought all my e-mail after last weeks travel, so > this comes a bit delayed. > > SciRun/2 and UCF/Uintah are both Frameworks based on the Common > Components Architecture (CCA) Forum specification. UCF is a components > framework for simulating PDEs in parallel on Structured Adaptive Mesh > Refinement Grids (much like the LLNL SAMRI package). After reading the > attached paper, my sense is that UCF is likely too domain specific (PDE solver > using SAMR) for LSST pipelines. > > Other CCA implementations/tools such as Ccaffeine/GUI, Eclipse with > CCA plugin support and Babel/SIDL (cross language interoperability > components compiler) provide the capability for creating interoperable > components in C/C++/Python/Fortran90/Java without the domain specific > specializations of UCF. A CCA framework that couples LSST Pipeline > Middleware and Astronomy specific domain libraries could provide > both multi-language component support, and LSST specific methods > (pipelline event registration/notifications, bulletin board services/common > address space, etc). > > -Marcus > > On Wednesday 19 April 2006 11:50, Jeffrey P Kantor wrote: >> Hi Ray, Marcus, Michelle, >> >> This UCF framework looks very interesting, lighter weight than UIMA, some >> features for "steering" execution, and is based on SciRun, which Michelle >> worked on. I think it should move up to the top of the eval list, along >> with the rest of the messaging/glue middleware. However, I'm not sure it >> is publicly available, so we may need to see if Michelle can work a >> connection at Utah. Thanks! >> >> Jeff > > -- > W. Marcus Miller, Ph.D. > Lawrence Livermore National Laboratory > Mail Stop L-560 Tel : (925) 424-4147 > 7000 East Avenue FAX : (925) 422-6287 > Livermore, CA 94550 e-mail: [EMAIL PROTECTED] > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
