Hi Suresh, I've been meaning to respond to this, believe it or not, this is my first opportunity to! :-)
Comments inline below: > All of us clearly know what Airavata software is about in varying details, > but at the same time I realize not every one of us on the list have a full > understanding of the architecture as a whole and sub-components. Along with > inheriting the code donation, I suggest we focus on bringing every one to > speed by means of high level and low level architecture diagrams. I will > start a detailed email thread about this task. In short, currently the > software assumes understanding of e-Science in general and some details of > Grid Computing. Our first focus should be to bring the software to a level > any java developer can understand and contribute. Next the focus can be to > make it easy for novice users. > > I thought a good place to start might be to list out the high level goals and > then focus on the first goal with detailed JIRA tasks. I am assuming you will > steer us with a orthogonal roadmap to graduation. I hope I am not implying we > need to meet the following goals to graduate, because some of them are very > open ended. Also, please note that Airavata may have some of these features > already, I am mainly categorizing so we will have a focused effort in > testing, re-writing or new implementations. > > Airavata high level feature list: > > Phase 1: Construct, Execute and monitor workflows from pre-deployed web > services. For workflow monitoring, check out the work we did in OODT in case you're interested. We recently built some JAX-RS-based services for health monitoring and pedigree tracing: OODT-139 [1]. > The workflow enactment engine will be the inherent Airavata Workflow > Interpreter. Register command line applications as web services, construct > and execute workflows with these application services. The applications may > run locally, on Grid enabled resources or by ssh'ing to a remote resource. > The client to test this phase workflows can be Airavata Workflow Client > (XBaya) running as a desktop application. Very cool! Is this code somewhere? I wonder if it would make sense to wrap the Airavata workflow engine with the OODT workflow engine interface. Then OODT system users could benefit from the monitoring and tool support apps and execute pre-built workflows on the Airavata engine. I'd love to take a look at the Airvata WE, so a pointer would be great. > > Phase 2: Execute all of phase 1 workflows on Apache ODE engine by generating > and deploying BPEL. Develop and deploy gadget interfaces to Apache Rave > container to support application registration, workflow submission and > monitoring components. Support applications running on virtual machine images > to be deployed to Amazon EC2, EUCALYPTUS and similar > infrastructure-as-a-service cloud deployments. Very cool. Do you think it's worth engaging the folks in the Hadoop community? They've done a done of work to execute tasks on EC2. Also it might be good to check out Apache Whirr [2], which is trying to abstract job execution and resource management for different cloud stacks. > > Phase 3: Expand the compute resources to Elastic Map Reduce and Hadoop based > executions. Focus on the data and metadata catalog integration like Apache > OODT. Sure. I'd just like to point out though that OODT isn't just data and metadata catalog integration. In fact, it's got most of the workflow stuff (along with resource management, task wrapping and execution and integration with Ganglia, and Torque/etc.) you've talked about above. Check out these components: http://oodt.apache.org/components/maven/workflow/ http://oodt.apache.org/components/maven/resource/ Also, this paper is worth checking out: http://sunset.usc.edu/~mattmann/pubs/SMCIT09.pdf > > I will stop here, to allow us to discuss the same. Once we narrow down on the > high level phase 1 goals, I will start a detailed discussion on where the > code is now and the steps to get to goal1. For the phase 1 goal of monitoring, I can definitely help out with JAX-RS type services like the ones from OODT-139 [1]. HTH, Chris [1] http://issues.apache.org/jira/browse/OODT-139 [2] http://incubator.apache.org/whirr/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
