[
https://issues.apache.org/jira/browse/AIRAVATA-357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248051#comment-13248051
]
Suresh Marru commented on AIRAVATA-357:
---------------------------------------
Thank you for quickly grabbing attention to this task. Yes, what you suggest on
moving part of the job is very appealing and will have large community uptake.
This is so useful for check pointable scientific applications. Airavata
framework should consider providing a capability to plug in application logic
which evaluates the long running execution, and trigger job migration
requesting increased resources or also request a change in data partitioning or
user inputs. Any study on understanding the associated over heads will be a big
plus.
As for the logic to when to move to cloud, its a complex beast, as you suggest,
you can do heuristics or CBR or other adaptive learning techniques, but really
the challenge comes in effectively mapping the current problem at hand with
previous data. The challenge being there may be semantic differences among
these executions which needs elaborative metadata and provenance description.
But glad to see you thinking on these aspects.
As for level of integration, all these suggestions are arguable and there are
pros and cons of each. I personally favor lose coupling and not having any
dependency on critical path of basic airavata functionality, so I second your
later approach as well. You can also make the cloud provider API's themselves
can be integrated into GFac. GFac chain of responsibility based architecture
make this integration seamless. If you are planning a GSoc project based on
this task, we can help elaborate in more detail. But in short, these
capabilities will require changes to XBaya, GFac, and possibly WS-Messenger
(adding publishers).
> Provide cloud bursting like capabilities to Airavata computational workflows
> integrating with Apache Whirr
> ----------------------------------------------------------------------------------------------------------
>
> Key: AIRAVATA-357
> URL: https://issues.apache.org/jira/browse/AIRAVATA-357
> Project: Airavata
> Issue Type: New Feature
> Components: GFac, Workflow Interpreter, XBaya
> Reporter: Suresh Marru
> Labels: gsoc2012, mentor
> Fix For: WISHLIST
>
>
> Apache Airavata provides capabilities to construct execute and monitor
> computational workflows with built in providers to execute applications on
> compute intensive resources. The users of Airavata constantly have the need
> to execute workflows which have tasks with a hybrid combination of
> computational grids and computational clouds. More over, some applications
> can be executed on multiple types of resources. The selection will depend on
> the distribution of data to be processed, the tolerable latency on shared
> batch queued grid clusters, and on the nature and size of the problem to be
> solved and data analysis tasks. Especially for applications which reduce the
> data as they proceed in the graph, they better fit for local map-reduce
> executions residing on hadoop based file systems.
> This idea has two implementation paths:
> Firstly extend airavata workflow providers to integrate with cloud based run
> times. Integrating with higher level API's like Apache Whirr provides a great
> value addition to this project to encompass multiple cloud services.
> Secondly, extending Airavata enactment called Workflow Interpreter so a
> component in the workflow is capable of initially running on local clusters
> deploying hadoop file systems and if the execution is saturating the local
> resources, the executions can scale out to commercial on-demand clouds or
> leverage high speed low latency interconnects on computational grids. This
> scaling technique is often referred to as Cloud Bursting. This project will
> not only enable this capability to Airavata workflow.
> User community & Impact of the software: Airavata is primarily targeted to
> build science gateways using computational resources from various
> disciplines. The initial targeted set of gateways include projects supporting
> research and education in chemistry, life sciences, biophysics, environmental
> sciences, geosciences astronomy and nuclear physics. The goal of airavata is
> to enhance productivity of these gateways to utilize cyberinfrastructure of
> resources (e.g., local lab resources, the Extreme Science and Engineering
> Discovery Environment (XSEDE), the Open Science Grid (OSG), University
> Clusters, Academic and Commercial Computational Clouds like FutureGrid &
> Amazon EC2). By using open community based software components and services
> like Airavata, gateways will be able to focus on providing additional
> scientific capabilities and to expanding the number of supported users. The
> capabilities of these gateways will offer clear benefits to society.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
